michael@0: .. _test_manifests: michael@0: michael@0: ============== michael@0: Test Manifests michael@0: ============== michael@0: michael@0: Many test suites have their test metadata defined in files called michael@0: **test manifests**. michael@0: michael@0: Test manifests are divided into two flavors: :ref:`manifest_destiny_manifests` michael@0: and :ref:`reftest_manifests`. michael@0: michael@0: Naming Convention michael@0: ================= michael@0: michael@0: The build system does not enforce file naming for test manifest files. michael@0: However, the following convention is used. michael@0: michael@0: mochitest.ini michael@0: For the *plain* flavor of mochitests. michael@0: michael@0: chrome.ini michael@0: For the *chrome* flavor of mochitests. michael@0: michael@0: browser.ini michael@0: For the *browser chrome* flavor of mochitests. michael@0: michael@0: a11y.ini michael@0: For the *a11y* flavor of mochitests. michael@0: michael@0: xpcshell.ini michael@0: For *xpcshell* tests. michael@0: michael@0: webapprt.ini michael@0: For the *chrome* flavor of webapp runtime mochitests. michael@0: michael@0: .. _manifest_destiny_manifests: michael@0: michael@0: Manifest Destiny Manifests michael@0: ========================== michael@0: michael@0: Manifest destiny manifests are essentially ini files that conform to a basic michael@0: set of assumptions. michael@0: michael@0: The `reference documentation `_ michael@0: for manifest destiny manifests describes the basic format of test manifests. michael@0: michael@0: In summary, manifests are ini files with section names describing test files:: michael@0: michael@0: [test_foo.js] michael@0: [test_bar.js] michael@0: michael@0: Keys under sections can hold metadata about each test:: michael@0: michael@0: [test_foo.js] michael@0: skip-if = os == "win" michael@0: [test_foo.js] michael@0: skip-if = os == "linux" && debug michael@0: [test_baz.js] michael@0: fail-if = os == "mac" || os == "android" michael@0: michael@0: There is a special **DEFAULT** section whose keys/metadata apply to all michael@0: sections/tests:: michael@0: michael@0: [DEFAULT] michael@0: property = value michael@0: michael@0: [test_foo.js] michael@0: michael@0: In the above example, **test_foo.js** inherits the metadata **property = value** michael@0: from the **DEFAULT** section. michael@0: michael@0: Recognized Metadata michael@0: ------------------- michael@0: michael@0: Test manifests can define some common keys/metadata to influence behavior. michael@0: Those keys are as follows: michael@0: michael@0: head michael@0: List of files that will be executed before the test file. (Used in michael@0: xpcshell tests.) michael@0: michael@0: tail michael@0: List of files that will be executed after the test file. (Used in michael@0: xpcshell tests.) michael@0: michael@0: support-files michael@0: List of additional files required to run tests. This is typically michael@0: defined in the **DEFAULT** section. michael@0: michael@0: Unlike other file lists, *support-files* supports a globbing mechanism michael@0: to facilitate pulling in many files with minimal typing. This globbing michael@0: mechanism is activated if an entry in this value contains a ``*`` michael@0: character. A single ``*`` will wildcard match all files in a directory. michael@0: A double ``**`` will descend into child directories. For example, michael@0: ``data/*`` will match ``data/foo`` but not ``data/subdir/bar`` where michael@0: ``data/**`` will match ``data/foo`` and ``data/subdir/bar``. michael@0: michael@0: Support files starting with ``/`` are placed in a root directory, rather michael@0: than a location determined by the manifest location. For mochitests, michael@0: this allows for the placement of files at the server root. The source michael@0: file is selected from the base name (e.g., ``foo`` for ``/path/foo``). michael@0: Files starting with ``/`` cannot be selected using globbing. michael@0: michael@0: generated-files michael@0: List of files that are generated as part of the build and don't exist in michael@0: the source tree. michael@0: michael@0: The build system assumes that each manifest file, test file, and file michael@0: listed in **head**, **tail**, and **support-files** is static and michael@0: provided by the source tree (and not automatically generated as part michael@0: of the build). This variable tells the build system not to make this michael@0: assumption. michael@0: michael@0: This variable will likely go away sometime once all generated files are michael@0: accounted for in the build config. michael@0: michael@0: If a generated file is not listed in this key, a clobber build will michael@0: likely fail. michael@0: michael@0: dupe-manifest michael@0: Record that this manifest duplicates another manifest. michael@0: michael@0: The common scenario is two manifest files will include a shared michael@0: manifest file via the ``[include:file]`` special section. The build michael@0: system enforces that each test file is only provided by a single michael@0: manifest. Having this key present bypasses that check. michael@0: michael@0: The value of this key is ignored. michael@0: michael@0: run-if michael@0: Run this test only if the specified condition is true. michael@0: See :ref:`manifest_filter_language`. michael@0: michael@0: skip-if michael@0: Skip this test if the specified condition is true. michael@0: See :ref:`manifest_filter_language`. michael@0: michael@0: fail-if michael@0: Expect test failure if the specified condition is true. michael@0: See :ref:`manifest_filter_language`. michael@0: michael@0: run-sequentially michael@0: If present, the test should not be run in parallel with other tests. michael@0: michael@0: Some test harnesses support parallel test execution on separate processes michael@0: and/or threads (behavior varies by test harness). If this key is present, michael@0: the test harness should not attempt to run this test in parallel with any michael@0: other test. michael@0: michael@0: By convention, the value of this key is a string describing why the test michael@0: can't be run in parallel. michael@0: michael@0: .. _manifest_filter_language: michael@0: michael@0: Manifest Filter Language michael@0: ------------------------ michael@0: michael@0: Some manifest keys accept a special filter syntax as their values. These michael@0: values are essentially boolean expressions that are evaluated at test michael@0: execution time. michael@0: michael@0: The expressions can reference a well-defined set of variables, such as michael@0: ``os`` and ``debug``. These variables are populated from the michael@0: ``mozinfo.json`` file. For the full list of available variables, see michael@0: the :ref:`mozinfo documentation `. michael@0: michael@0: See michael@0: `the source `_ for the full documentation of the michael@0: expression syntax until it is documented here. michael@0: michael@0: .. todo:: michael@0: michael@0: Document manifest filter language. michael@0: michael@0: .. _manifest_file_installation: michael@0: michael@0: File Installation michael@0: ----------------- michael@0: michael@0: Files referenced by manifests are automatically installed into the object michael@0: directory into paths defined in michael@0: :py:func:`mozbuild.frontend.emitter.TreeMetadataEmitter._process_test_manifest`. michael@0: michael@0: Relative paths resolving to parent directory (e.g. michael@0: ``support-files = ../foo.txt`` have special behavior. michael@0: michael@0: For ``support-files``, the file will be installed to the default destination michael@0: for that manifest. Only the file's base name is used to construct the final michael@0: path: directories are irrelevant. Files starting with ``/`` are an exception, michael@0: these are installed relative to the root of the destination; the base name is michael@0: instead used to select the file.. michael@0: michael@0: For all other entry types, the file installation is skipped. michael@0: michael@0: .. _reftest_manifests: michael@0: michael@0: Reftest Manifests michael@0: ================= michael@0: michael@0: See `MDN `_.