The Tor Browser / file comparison
comparison: build/docs/test_manifests.rst
build/docs/test_manifests.rst
- branch
- TOR_BUG_9701
- changeset 8
- 97036ab72558
equal
deleted
inserted
replaced
| -1:000000000000 | 0:fc731dbbfc18 |
|---|---|
| 1 .. _test_manifests: | |
| 2 | |
| 3 ============== | |
| 4 Test Manifests | |
| 5 ============== | |
| 6 | |
| 7 Many test suites have their test metadata defined in files called | |
| 8 **test manifests**. | |
| 9 | |
| 10 Test manifests are divided into two flavors: :ref:`manifest_destiny_manifests` | |
| 11 and :ref:`reftest_manifests`. | |
| 12 | |
| 13 Naming Convention | |
| 14 ================= | |
| 15 | |
| 16 The build system does not enforce file naming for test manifest files. | |
| 17 However, the following convention is used. | |
| 18 | |
| 19 mochitest.ini | |
| 20 For the *plain* flavor of mochitests. | |
| 21 | |
| 22 chrome.ini | |
| 23 For the *chrome* flavor of mochitests. | |
| 24 | |
| 25 browser.ini | |
| 26 For the *browser chrome* flavor of mochitests. | |
| 27 | |
| 28 a11y.ini | |
| 29 For the *a11y* flavor of mochitests. | |
| 30 | |
| 31 xpcshell.ini | |
| 32 For *xpcshell* tests. | |
| 33 | |
| 34 webapprt.ini | |
| 35 For the *chrome* flavor of webapp runtime mochitests. | |
| 36 | |
| 37 .. _manifest_destiny_manifests: | |
| 38 | |
| 39 Manifest Destiny Manifests | |
| 40 ========================== | |
| 41 | |
| 42 Manifest destiny manifests are essentially ini files that conform to a basic | |
| 43 set of assumptions. | |
| 44 | |
| 45 The `reference documentation <http://mozbase.readthedocs.org/en/latest/manifestdestiny.html>`_ | |
| 46 for manifest destiny manifests describes the basic format of test manifests. | |
| 47 | |
| 48 In summary, manifests are ini files with section names describing test files:: | |
| 49 | |
| 50 [test_foo.js] | |
| 51 [test_bar.js] | |
| 52 | |
| 53 Keys under sections can hold metadata about each test:: | |
| 54 | |
| 55 [test_foo.js] | |
| 56 skip-if = os == "win" | |
| 57 [test_foo.js] | |
| 58 skip-if = os == "linux" && debug | |
| 59 [test_baz.js] | |
| 60 fail-if = os == "mac" || os == "android" | |
| 61 | |
| 62 There is a special **DEFAULT** section whose keys/metadata apply to all | |
| 63 sections/tests:: | |
| 64 | |
| 65 [DEFAULT] | |
| 66 property = value | |
| 67 | |
| 68 [test_foo.js] | |
| 69 | |
| 70 In the above example, **test_foo.js** inherits the metadata **property = value** | |
| 71 from the **DEFAULT** section. | |
| 72 | |
| 73 Recognized Metadata | |
| 74 ------------------- | |
| 75 | |
| 76 Test manifests can define some common keys/metadata to influence behavior. | |
| 77 Those keys are as follows: | |
| 78 | |
| 79 head | |
| 80 List of files that will be executed before the test file. (Used in | |
| 81 xpcshell tests.) | |
| 82 | |
| 83 tail | |
| 84 List of files that will be executed after the test file. (Used in | |
| 85 xpcshell tests.) | |
| 86 | |
| 87 support-files | |
| 88 List of additional files required to run tests. This is typically | |
| 89 defined in the **DEFAULT** section. | |
| 90 | |
| 91 Unlike other file lists, *support-files* supports a globbing mechanism | |
| 92 to facilitate pulling in many files with minimal typing. This globbing | |
| 93 mechanism is activated if an entry in this value contains a ``*`` | |
| 94 character. A single ``*`` will wildcard match all files in a directory. | |
| 95 A double ``**`` will descend into child directories. For example, | |
| 96 ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where | |
| 97 ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. | |
| 98 | |
| 99 Support files starting with ``/`` are placed in a root directory, rather | |
| 100 than a location determined by the manifest location. For mochitests, | |
| 101 this allows for the placement of files at the server root. The source | |
| 102 file is selected from the base name (e.g., ``foo`` for ``/path/foo``). | |
| 103 Files starting with ``/`` cannot be selected using globbing. | |
| 104 | |
| 105 generated-files | |
| 106 List of files that are generated as part of the build and don't exist in | |
| 107 the source tree. | |
| 108 | |
| 109 The build system assumes that each manifest file, test file, and file | |
| 110 listed in **head**, **tail**, and **support-files** is static and | |
| 111 provided by the source tree (and not automatically generated as part | |
| 112 of the build). This variable tells the build system not to make this | |
| 113 assumption. | |
| 114 | |
| 115 This variable will likely go away sometime once all generated files are | |
| 116 accounted for in the build config. | |
| 117 | |
| 118 If a generated file is not listed in this key, a clobber build will | |
| 119 likely fail. | |
| 120 | |
| 121 dupe-manifest | |
| 122 Record that this manifest duplicates another manifest. | |
| 123 | |
| 124 The common scenario is two manifest files will include a shared | |
| 125 manifest file via the ``[include:file]`` special section. The build | |
| 126 system enforces that each test file is only provided by a single | |
| 127 manifest. Having this key present bypasses that check. | |
| 128 | |
| 129 The value of this key is ignored. | |
| 130 | |
| 131 run-if | |
| 132 Run this test only if the specified condition is true. | |
| 133 See :ref:`manifest_filter_language`. | |
| 134 | |
| 135 skip-if | |
| 136 Skip this test if the specified condition is true. | |
| 137 See :ref:`manifest_filter_language`. | |
| 138 | |
| 139 fail-if | |
| 140 Expect test failure if the specified condition is true. | |
| 141 See :ref:`manifest_filter_language`. | |
| 142 | |
| 143 run-sequentially | |
| 144 If present, the test should not be run in parallel with other tests. | |
| 145 | |
| 146 Some test harnesses support parallel test execution on separate processes | |
| 147 and/or threads (behavior varies by test harness). If this key is present, | |
| 148 the test harness should not attempt to run this test in parallel with any | |
| 149 other test. | |
| 150 | |
| 151 By convention, the value of this key is a string describing why the test | |
| 152 can't be run in parallel. | |
| 153 | |
| 154 .. _manifest_filter_language: | |
| 155 | |
| 156 Manifest Filter Language | |
| 157 ------------------------ | |
| 158 | |
| 159 Some manifest keys accept a special filter syntax as their values. These | |
| 160 values are essentially boolean expressions that are evaluated at test | |
| 161 execution time. | |
| 162 | |
| 163 The expressions can reference a well-defined set of variables, such as | |
| 164 ``os`` and ``debug``. These variables are populated from the | |
| 165 ``mozinfo.json`` file. For the full list of available variables, see | |
| 166 the :ref:`mozinfo documentation <mozinfo_attributes>`. | |
| 167 | |
| 168 See | |
| 169 `the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestdestiny/manifestparser/manifestparser.py>`_ for the full documentation of the | |
| 170 expression syntax until it is documented here. | |
| 171 | |
| 172 .. todo:: | |
| 173 | |
| 174 Document manifest filter language. | |
| 175 | |
| 176 .. _manifest_file_installation: | |
| 177 | |
| 178 File Installation | |
| 179 ----------------- | |
| 180 | |
| 181 Files referenced by manifests are automatically installed into the object | |
| 182 directory into paths defined in | |
| 183 :py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. | |
| 184 | |
| 185 Relative paths resolving to parent directory (e.g. | |
| 186 ``support-files = ../foo.txt`` have special behavior. | |
| 187 | |
| 188 For ``support-files``, the file will be installed to the default destination | |
| 189 for that manifest. Only the file's base name is used to construct the final | |
| 190 path: directories are irrelevant. Files starting with ``/`` are an exception, | |
| 191 these are installed relative to the root of the destination; the base name is | |
| 192 instead used to select the file.. | |
| 193 | |
| 194 For all other entry types, the file installation is skipped. | |
| 195 | |
| 196 .. _reftest_manifests: | |
| 197 | |
| 198 Reftest Manifests | |
| 199 ================= | |
| 200 | |
| 201 See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_. |
