build/docs/test_manifests.rst@97036ab72558 (annotated)
build/docs/test_manifests.rst
Tue, 06 Jan 2015 21:39:09 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Tue, 06 Jan 2015 21:39:09 +0100
- branch
- TOR_BUG_9701
- changeset 8
- 97036ab72558
- permissions
- -rw-r--r--
Conditionally force memory storage according to privacy.thirdparty.isolate;
This solves Tor bug #9701, complying with disk avoidance documented in
https://www.torproject.org/projects/torbrowser/design/#disk-avoidance.
| michael@0 | 1 | .. _test_manifests: |
| michael@0 | 2 | |
| michael@0 | 3 | ============== |
| michael@0 | 4 | Test Manifests |
| michael@0 | 5 | ============== |
| michael@0 | 6 | |
| michael@0 | 7 | Many test suites have their test metadata defined in files called |
| michael@0 | 8 | **test manifests**. |
| michael@0 | 9 | |
| michael@0 | 10 | Test manifests are divided into two flavors: :ref:`manifest_destiny_manifests` |
| michael@0 | 11 | and :ref:`reftest_manifests`. |
| michael@0 | 12 | |
| michael@0 | 13 | Naming Convention |
| michael@0 | 14 | ================= |
| michael@0 | 15 | |
| michael@0 | 16 | The build system does not enforce file naming for test manifest files. |
| michael@0 | 17 | However, the following convention is used. |
| michael@0 | 18 | |
| michael@0 | 19 | mochitest.ini |
| michael@0 | 20 | For the *plain* flavor of mochitests. |
| michael@0 | 21 | |
| michael@0 | 22 | chrome.ini |
| michael@0 | 23 | For the *chrome* flavor of mochitests. |
| michael@0 | 24 | |
| michael@0 | 25 | browser.ini |
| michael@0 | 26 | For the *browser chrome* flavor of mochitests. |
| michael@0 | 27 | |
| michael@0 | 28 | a11y.ini |
| michael@0 | 29 | For the *a11y* flavor of mochitests. |
| michael@0 | 30 | |
| michael@0 | 31 | xpcshell.ini |
| michael@0 | 32 | For *xpcshell* tests. |
| michael@0 | 33 | |
| michael@0 | 34 | webapprt.ini |
| michael@0 | 35 | For the *chrome* flavor of webapp runtime mochitests. |
| michael@0 | 36 | |
| michael@0 | 37 | .. _manifest_destiny_manifests: |
| michael@0 | 38 | |
| michael@0 | 39 | Manifest Destiny Manifests |
| michael@0 | 40 | ========================== |
| michael@0 | 41 | |
| michael@0 | 42 | Manifest destiny manifests are essentially ini files that conform to a basic |
| michael@0 | 43 | set of assumptions. |
| michael@0 | 44 | |
| michael@0 | 45 | The `reference documentation <http://mozbase.readthedocs.org/en/latest/manifestdestiny.html>`_ |
| michael@0 | 46 | for manifest destiny manifests describes the basic format of test manifests. |
| michael@0 | 47 | |
| michael@0 | 48 | In summary, manifests are ini files with section names describing test files:: |
| michael@0 | 49 | |
| michael@0 | 50 | [test_foo.js] |
| michael@0 | 51 | [test_bar.js] |
| michael@0 | 52 | |
| michael@0 | 53 | Keys under sections can hold metadata about each test:: |
| michael@0 | 54 | |
| michael@0 | 55 | [test_foo.js] |
| michael@0 | 56 | skip-if = os == "win" |
| michael@0 | 57 | [test_foo.js] |
| michael@0 | 58 | skip-if = os == "linux" && debug |
| michael@0 | 59 | [test_baz.js] |
| michael@0 | 60 | fail-if = os == "mac" || os == "android" |
| michael@0 | 61 | |
| michael@0 | 62 | There is a special **DEFAULT** section whose keys/metadata apply to all |
| michael@0 | 63 | sections/tests:: |
| michael@0 | 64 | |
| michael@0 | 65 | [DEFAULT] |
| michael@0 | 66 | property = value |
| michael@0 | 67 | |
| michael@0 | 68 | [test_foo.js] |
| michael@0 | 69 | |
| michael@0 | 70 | In the above example, **test_foo.js** inherits the metadata **property = value** |
| michael@0 | 71 | from the **DEFAULT** section. |
| michael@0 | 72 | |
| michael@0 | 73 | Recognized Metadata |
| michael@0 | 74 | ------------------- |
| michael@0 | 75 | |
| michael@0 | 76 | Test manifests can define some common keys/metadata to influence behavior. |
| michael@0 | 77 | Those keys are as follows: |
| michael@0 | 78 | |
| michael@0 | 79 | head |
| michael@0 | 80 | List of files that will be executed before the test file. (Used in |
| michael@0 | 81 | xpcshell tests.) |
| michael@0 | 82 | |
| michael@0 | 83 | tail |
| michael@0 | 84 | List of files that will be executed after the test file. (Used in |
| michael@0 | 85 | xpcshell tests.) |
| michael@0 | 86 | |
| michael@0 | 87 | support-files |
| michael@0 | 88 | List of additional files required to run tests. This is typically |
| michael@0 | 89 | defined in the **DEFAULT** section. |
| michael@0 | 90 | |
| michael@0 | 91 | Unlike other file lists, *support-files* supports a globbing mechanism |
| michael@0 | 92 | to facilitate pulling in many files with minimal typing. This globbing |
| michael@0 | 93 | mechanism is activated if an entry in this value contains a ``*`` |
| michael@0 | 94 | character. A single ``*`` will wildcard match all files in a directory. |
| michael@0 | 95 | A double ``**`` will descend into child directories. For example, |
| michael@0 | 96 | ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where |
| michael@0 | 97 | ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. |
| michael@0 | 98 | |
| michael@0 | 99 | Support files starting with ``/`` are placed in a root directory, rather |
| michael@0 | 100 | than a location determined by the manifest location. For mochitests, |
| michael@0 | 101 | this allows for the placement of files at the server root. The source |
| michael@0 | 102 | file is selected from the base name (e.g., ``foo`` for ``/path/foo``). |
| michael@0 | 103 | Files starting with ``/`` cannot be selected using globbing. |
| michael@0 | 104 | |
| michael@0 | 105 | generated-files |
| michael@0 | 106 | List of files that are generated as part of the build and don't exist in |
| michael@0 | 107 | the source tree. |
| michael@0 | 108 | |
| michael@0 | 109 | The build system assumes that each manifest file, test file, and file |
| michael@0 | 110 | listed in **head**, **tail**, and **support-files** is static and |
| michael@0 | 111 | provided by the source tree (and not automatically generated as part |
| michael@0 | 112 | of the build). This variable tells the build system not to make this |
| michael@0 | 113 | assumption. |
| michael@0 | 114 | |
| michael@0 | 115 | This variable will likely go away sometime once all generated files are |
| michael@0 | 116 | accounted for in the build config. |
| michael@0 | 117 | |
| michael@0 | 118 | If a generated file is not listed in this key, a clobber build will |
| michael@0 | 119 | likely fail. |
| michael@0 | 120 | |
| michael@0 | 121 | dupe-manifest |
| michael@0 | 122 | Record that this manifest duplicates another manifest. |
| michael@0 | 123 | |
| michael@0 | 124 | The common scenario is two manifest files will include a shared |
| michael@0 | 125 | manifest file via the ``[include:file]`` special section. The build |
| michael@0 | 126 | system enforces that each test file is only provided by a single |
| michael@0 | 127 | manifest. Having this key present bypasses that check. |
| michael@0 | 128 | |
| michael@0 | 129 | The value of this key is ignored. |
| michael@0 | 130 | |
| michael@0 | 131 | run-if |
| michael@0 | 132 | Run this test only if the specified condition is true. |
| michael@0 | 133 | See :ref:`manifest_filter_language`. |
| michael@0 | 134 | |
| michael@0 | 135 | skip-if |
| michael@0 | 136 | Skip this test if the specified condition is true. |
| michael@0 | 137 | See :ref:`manifest_filter_language`. |
| michael@0 | 138 | |
| michael@0 | 139 | fail-if |
| michael@0 | 140 | Expect test failure if the specified condition is true. |
| michael@0 | 141 | See :ref:`manifest_filter_language`. |
| michael@0 | 142 | |
| michael@0 | 143 | run-sequentially |
| michael@0 | 144 | If present, the test should not be run in parallel with other tests. |
| michael@0 | 145 | |
| michael@0 | 146 | Some test harnesses support parallel test execution on separate processes |
| michael@0 | 147 | and/or threads (behavior varies by test harness). If this key is present, |
| michael@0 | 148 | the test harness should not attempt to run this test in parallel with any |
| michael@0 | 149 | other test. |
| michael@0 | 150 | |
| michael@0 | 151 | By convention, the value of this key is a string describing why the test |
| michael@0 | 152 | can't be run in parallel. |
| michael@0 | 153 | |
| michael@0 | 154 | .. _manifest_filter_language: |
| michael@0 | 155 | |
| michael@0 | 156 | Manifest Filter Language |
| michael@0 | 157 | ------------------------ |
| michael@0 | 158 | |
| michael@0 | 159 | Some manifest keys accept a special filter syntax as their values. These |
| michael@0 | 160 | values are essentially boolean expressions that are evaluated at test |
| michael@0 | 161 | execution time. |
| michael@0 | 162 | |
| michael@0 | 163 | The expressions can reference a well-defined set of variables, such as |
| michael@0 | 164 | ``os`` and ``debug``. These variables are populated from the |
| michael@0 | 165 | ``mozinfo.json`` file. For the full list of available variables, see |
| michael@0 | 166 | the :ref:`mozinfo documentation <mozinfo_attributes>`. |
| michael@0 | 167 | |
| michael@0 | 168 | See |
| michael@0 | 169 | `the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestdestiny/manifestparser/manifestparser.py>`_ for the full documentation of the |
| michael@0 | 170 | expression syntax until it is documented here. |
| michael@0 | 171 | |
| michael@0 | 172 | .. todo:: |
| michael@0 | 173 | |
| michael@0 | 174 | Document manifest filter language. |
| michael@0 | 175 | |
| michael@0 | 176 | .. _manifest_file_installation: |
| michael@0 | 177 | |
| michael@0 | 178 | File Installation |
| michael@0 | 179 | ----------------- |
| michael@0 | 180 | |
| michael@0 | 181 | Files referenced by manifests are automatically installed into the object |
| michael@0 | 182 | directory into paths defined in |
| michael@0 | 183 | :py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. |
| michael@0 | 184 | |
| michael@0 | 185 | Relative paths resolving to parent directory (e.g. |
| michael@0 | 186 | ``support-files = ../foo.txt`` have special behavior. |
| michael@0 | 187 | |
| michael@0 | 188 | For ``support-files``, the file will be installed to the default destination |
| michael@0 | 189 | for that manifest. Only the file's base name is used to construct the final |
| michael@0 | 190 | path: directories are irrelevant. Files starting with ``/`` are an exception, |
| michael@0 | 191 | these are installed relative to the root of the destination; the base name is |
| michael@0 | 192 | instead used to select the file.. |
| michael@0 | 193 | |
| michael@0 | 194 | For all other entry types, the file installation is skipped. |
| michael@0 | 195 | |
| michael@0 | 196 | .. _reftest_manifests: |
| michael@0 | 197 | |
| michael@0 | 198 | Reftest Manifests |
| michael@0 | 199 | ================= |
| michael@0 | 200 | |
| michael@0 | 201 | See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_. |
