1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/build/docs/test_manifests.rst Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,201 @@
1.4 +.. _test_manifests:
1.5 +
1.6 +==============
1.7 +Test Manifests
1.8 +==============
1.9 +
1.10 +Many test suites have their test metadata defined in files called
1.11 +**test manifests**.
1.12 +
1.13 +Test manifests are divided into two flavors: :ref:`manifest_destiny_manifests`
1.14 +and :ref:`reftest_manifests`.
1.15 +
1.16 +Naming Convention
1.17 +=================
1.18 +
1.19 +The build system does not enforce file naming for test manifest files.
1.20 +However, the following convention is used.
1.21 +
1.22 +mochitest.ini
1.23 + For the *plain* flavor of mochitests.
1.24 +
1.25 +chrome.ini
1.26 + For the *chrome* flavor of mochitests.
1.27 +
1.28 +browser.ini
1.29 + For the *browser chrome* flavor of mochitests.
1.30 +
1.31 +a11y.ini
1.32 + For the *a11y* flavor of mochitests.
1.33 +
1.34 +xpcshell.ini
1.35 + For *xpcshell* tests.
1.36 +
1.37 +webapprt.ini
1.38 + For the *chrome* flavor of webapp runtime mochitests.
1.39 +
1.40 +.. _manifest_destiny_manifests:
1.41 +
1.42 +Manifest Destiny Manifests
1.43 +==========================
1.44 +
1.45 +Manifest destiny manifests are essentially ini files that conform to a basic
1.46 +set of assumptions.
1.47 +
1.48 +The `reference documentation <http://mozbase.readthedocs.org/en/latest/manifestdestiny.html>`_
1.49 +for manifest destiny manifests describes the basic format of test manifests.
1.50 +
1.51 +In summary, manifests are ini files with section names describing test files::
1.52 +
1.53 + [test_foo.js]
1.54 + [test_bar.js]
1.55 +
1.56 +Keys under sections can hold metadata about each test::
1.57 +
1.58 + [test_foo.js]
1.59 + skip-if = os == "win"
1.60 + [test_foo.js]
1.61 + skip-if = os == "linux" && debug
1.62 + [test_baz.js]
1.63 + fail-if = os == "mac" || os == "android"
1.64 +
1.65 +There is a special **DEFAULT** section whose keys/metadata apply to all
1.66 +sections/tests::
1.67 +
1.68 + [DEFAULT]
1.69 + property = value
1.70 +
1.71 + [test_foo.js]
1.72 +
1.73 +In the above example, **test_foo.js** inherits the metadata **property = value**
1.74 +from the **DEFAULT** section.
1.75 +
1.76 +Recognized Metadata
1.77 +-------------------
1.78 +
1.79 +Test manifests can define some common keys/metadata to influence behavior.
1.80 +Those keys are as follows:
1.81 +
1.82 +head
1.83 + List of files that will be executed before the test file. (Used in
1.84 + xpcshell tests.)
1.85 +
1.86 +tail
1.87 + List of files that will be executed after the test file. (Used in
1.88 + xpcshell tests.)
1.89 +
1.90 +support-files
1.91 + List of additional files required to run tests. This is typically
1.92 + defined in the **DEFAULT** section.
1.93 +
1.94 + Unlike other file lists, *support-files* supports a globbing mechanism
1.95 + to facilitate pulling in many files with minimal typing. This globbing
1.96 + mechanism is activated if an entry in this value contains a ``*``
1.97 + character. A single ``*`` will wildcard match all files in a directory.
1.98 + A double ``**`` will descend into child directories. For example,
1.99 + ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where
1.100 + ``data/**`` will match ``data/foo`` and ``data/subdir/bar``.
1.101 +
1.102 + Support files starting with ``/`` are placed in a root directory, rather
1.103 + than a location determined by the manifest location. For mochitests,
1.104 + this allows for the placement of files at the server root. The source
1.105 + file is selected from the base name (e.g., ``foo`` for ``/path/foo``).
1.106 + Files starting with ``/`` cannot be selected using globbing.
1.107 +
1.108 +generated-files
1.109 + List of files that are generated as part of the build and don't exist in
1.110 + the source tree.
1.111 +
1.112 + The build system assumes that each manifest file, test file, and file
1.113 + listed in **head**, **tail**, and **support-files** is static and
1.114 + provided by the source tree (and not automatically generated as part
1.115 + of the build). This variable tells the build system not to make this
1.116 + assumption.
1.117 +
1.118 + This variable will likely go away sometime once all generated files are
1.119 + accounted for in the build config.
1.120 +
1.121 + If a generated file is not listed in this key, a clobber build will
1.122 + likely fail.
1.123 +
1.124 +dupe-manifest
1.125 + Record that this manifest duplicates another manifest.
1.126 +
1.127 + The common scenario is two manifest files will include a shared
1.128 + manifest file via the ``[include:file]`` special section. The build
1.129 + system enforces that each test file is only provided by a single
1.130 + manifest. Having this key present bypasses that check.
1.131 +
1.132 + The value of this key is ignored.
1.133 +
1.134 +run-if
1.135 + Run this test only if the specified condition is true.
1.136 + See :ref:`manifest_filter_language`.
1.137 +
1.138 +skip-if
1.139 + Skip this test if the specified condition is true.
1.140 + See :ref:`manifest_filter_language`.
1.141 +
1.142 +fail-if
1.143 + Expect test failure if the specified condition is true.
1.144 + See :ref:`manifest_filter_language`.
1.145 +
1.146 +run-sequentially
1.147 + If present, the test should not be run in parallel with other tests.
1.148 +
1.149 + Some test harnesses support parallel test execution on separate processes
1.150 + and/or threads (behavior varies by test harness). If this key is present,
1.151 + the test harness should not attempt to run this test in parallel with any
1.152 + other test.
1.153 +
1.154 + By convention, the value of this key is a string describing why the test
1.155 + can't be run in parallel.
1.156 +
1.157 +.. _manifest_filter_language:
1.158 +
1.159 +Manifest Filter Language
1.160 +------------------------
1.161 +
1.162 +Some manifest keys accept a special filter syntax as their values. These
1.163 +values are essentially boolean expressions that are evaluated at test
1.164 +execution time.
1.165 +
1.166 +The expressions can reference a well-defined set of variables, such as
1.167 +``os`` and ``debug``. These variables are populated from the
1.168 +``mozinfo.json`` file. For the full list of available variables, see
1.169 +the :ref:`mozinfo documentation <mozinfo_attributes>`.
1.170 +
1.171 +See
1.172 +`the source <https://hg.mozilla.org/mozilla-central/file/default/testing/mozbase/manifestdestiny/manifestparser/manifestparser.py>`_ for the full documentation of the
1.173 +expression syntax until it is documented here.
1.174 +
1.175 +.. todo::
1.176 +
1.177 + Document manifest filter language.
1.178 +
1.179 +.. _manifest_file_installation:
1.180 +
1.181 +File Installation
1.182 +-----------------
1.183 +
1.184 +Files referenced by manifests are automatically installed into the object
1.185 +directory into paths defined in
1.186 +:py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`.
1.187 +
1.188 +Relative paths resolving to parent directory (e.g.
1.189 +``support-files = ../foo.txt`` have special behavior.
1.190 +
1.191 +For ``support-files``, the file will be installed to the default destination
1.192 +for that manifest. Only the file's base name is used to construct the final
1.193 +path: directories are irrelevant. Files starting with ``/`` are an exception,
1.194 +these are installed relative to the root of the destination; the base name is
1.195 +instead used to select the file..
1.196 +
1.197 +For all other entry types, the file installation is skipped.
1.198 +
1.199 +.. _reftest_manifests:
1.200 +
1.201 +Reftest Manifests
1.202 +=================
1.203 +
1.204 +See `MDN <https://developer.mozilla.org/en-US/docs/Creating_reftest-based_unit_tests>`_.
