michael@0: Managing lists of tests
michael@0: =======================
michael@0: 
michael@0: We don't always want to run all tests, all the time. Sometimes a test
michael@0: may be broken, in other cases we only want to run a test on a specific
michael@0: platform or build of Mozilla. To handle these cases (and more), we
michael@0: created a python library to create and use test "manifests", which
michael@0: codify this information.
michael@0: 
michael@0: :mod:`manifestdestiny` --- Create and manage test manifests
michael@0: -----------------------------------------------------------
michael@0: 
michael@0: manifestdestiny lets you easily create and use test manifests, to
michael@0: control which tests are run under what circumstances.
michael@0: 
michael@0: What ManifestDestiny gives you:
michael@0: 
michael@0: * manifests are ordered lists of tests
michael@0: * tests may have an arbitrary number of key, value pairs
michael@0: * the parser returns an ordered list of test data structures, which
michael@0:   are just dicts with some keys.  For example, a test with no
michael@0:   user-specified metadata looks like this:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [{'expected': 'pass',
michael@0:       'path': '/home/mozilla/mozmill/src/ManifestDestiny/manifestdestiny/tests/testToolbar/testBackForwardButtons.js',
michael@0:       'relpath': 'testToolbar/testBackForwardButtons.js',
michael@0:       'name': 'testBackForwardButtons.js',
michael@0:       'here': '/home/mozilla/mozmill/src/ManifestDestiny/manifestdestiny/tests',
michael@0:       'manifest': '/home/mozilla/mozmill/src/ManifestDestiny/manifestdestiny/tests/manifest.ini',}]
michael@0: 
michael@0: The keys displayed here (path, relpath, name, here, and manifest) are
michael@0: reserved keys for ManifestDestiny and any consuming APIs.  You can add
michael@0: additional key, value metadata to each test.
michael@0: 
michael@0: Why have test manifests?
michael@0: ````````````````````````
michael@0: 
michael@0: It is desirable to have a unified format for test manifests for testing
michael@0: [mozilla-central](http://hg.mozilla.org/mozilla-central), etc.
michael@0: 
michael@0: * It is desirable to be able to selectively enable or disable tests based on platform or other conditions. This should be easy to do. Currently, since many of the harnesses just crawl directories, there is no effective way of disabling a test except for removal from mozilla-central
michael@0: * It is desriable to do this in a universal way so that enabling and disabling tests as well as other tasks are easily accessible to a wider audience than just those intimately familiar with the specific test framework.
michael@0: * It is desirable to have other metadata on top of the test. For instance, let's say a test is marked as skipped. It would be nice to give the reason why.
michael@0: 
michael@0: 
michael@0: Most Mozilla test harnesses work by crawling a directory structure.
michael@0: While this is straight-forward, manifests offer several practical
michael@0: advantages:
michael@0: 
michael@0: * ability to turn a test off easily: if a test is broken on m-c
michael@0:   currently, the only way to turn it off, generally speaking, is just
michael@0:   removing the test.  Often this is undesirable, as if the test should
michael@0:   be dismissed because other people want to land and it can't be
michael@0:   investigated in real time (is it a failure? is the test bad? is no
michael@0:   one around that knows the test?), then backing out a test is at best
michael@0:   problematic.  With a manifest, a test may be disabled without
michael@0:   removing it from the tree and a bug filed with the appropriate
michael@0:   reason:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:      [test_broken.js]
michael@0:      disabled = https://bugzilla.mozilla.org/show_bug.cgi?id=123456
michael@0: 
michael@0: * ability to run different (subsets of) tests on different
michael@0:   platforms. Traditionally, we've done a bit of magic or had the test
michael@0:   know what platform it would or would not run on. With manifests, you
michael@0:   can mark what platforms a test will or will not run on and change
michael@0:   these without changing the test.
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:      [test_works_on_windows_only.js]
michael@0:      run-if = os == 'win'
michael@0: 
michael@0: * ability to markup tests with metadata. We have a large, complicated,
michael@0:   and always changing infrastructure.  key, value metadata may be used
michael@0:   as an annotation to a test and appropriately curated and mined.  For
michael@0:   instance, we could mark certain tests as randomorange with a bug
michael@0:   number, if it were desirable.
michael@0: 
michael@0: * ability to have sane and well-defined test-runs. You can keep
michael@0:   different manifests for different test runs and ``[include:]``
michael@0:   (sub)manifests as appropriate to your needs.
michael@0: 
michael@0: Manifest Format
michael@0: ````````
michael@0: 
michael@0: Manifests are .ini file with the section names denoting the path
michael@0: relative to the manifest:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [foo.js]
michael@0:     [bar.js]
michael@0:     [fleem.js]
michael@0: 
michael@0: The sections are read in order. In addition, tests may include
michael@0: arbitrary key, value metadata to be used by the harness.  You may also
michael@0: have a `[DEFAULT]` section that will give key, value pairs that will
michael@0: be inherited by each test unless overridden:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [DEFAULT]
michael@0:     type = restart
michael@0: 
michael@0:     [lilies.js]
michael@0:     color = white
michael@0: 
michael@0:     [daffodils.js]
michael@0:     color = yellow
michael@0:     type = other
michael@0:     # override type from DEFAULT
michael@0: 
michael@0:     [roses.js]
michael@0:     color = red
michael@0: 
michael@0: You can also include other manifests:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [include:subdir/anothermanifest.ini]
michael@0: 
michael@0: Manifests are included relative to the directory of the manifest with
michael@0: the `[include:]` directive unless they are absolute paths.
michael@0: 
michael@0: By default you can use both '#' and ';' as comment characters. Comments
michael@0: must start on a new line, inline comments are not supported.
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [roses.js]
michael@0:     # a valid comment
michael@0:     ; another valid comment
michael@0:     color = red # not a valid comment
michael@0: 
michael@0: In the example above, the 'color' property will have the value 'red #
michael@0: not a valid comment'.
michael@0: 
michael@0: Manifest Conditional Expressions
michael@0: ````````````````````````````````
michael@0: The conditional expressions used in manifests are parsed using the *ExpressionParser* class.
michael@0: 
michael@0: .. autoclass:: manifestparser.ExpressionParser
michael@0: 
michael@0: Consumers of this module are expected to pass in a value dictionary
michael@0: for evaluating conditional expressions. A common pattern is to pass
michael@0: the dictionary from the :mod:`mozinfo` module.
michael@0: 
michael@0: Data
michael@0: ````
michael@0: 
michael@0: Manifest Destiny gives tests as a list of dictionaries (in python
michael@0: terms).
michael@0: 
michael@0: * path: full path to the test
michael@0: * relpath: relative path starting from the root manifest location
michael@0: * name: file name of the test
michael@0: * here: the parent directory of the manifest
michael@0: * manifest: the path to the manifest containing the test
michael@0: 
michael@0: This data corresponds to a one-line manifest:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [testToolbar/testBackForwardButtons.js]
michael@0: 
michael@0: If additional key, values were specified, they would be in this dict
michael@0: as well.
michael@0: 
michael@0: Outside of the reserved keys, the remaining key, values
michael@0: are up to convention to use.  There is a (currently very minimal)
michael@0: generic integration layer in ManifestDestiny for use of all harnesses,
michael@0: `manifestparser.TestManifest`.
michael@0: For instance, if the 'disabled' key is present, you can get the set of
michael@0: tests without disabled (various other queries are doable as well).
michael@0: 
michael@0: Since the system is convention-based, the harnesses may do whatever
michael@0: they want with the data.  They may ignore it completely, they may use
michael@0: the provided integration layer, or they may provide their own
michael@0: integration layer.  This should allow whatever sort of logic is
michael@0: desired.  For instance, if in yourtestharness you wanted to run only on
michael@0: mondays for a certain class of tests:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     tests = []
michael@0:     for test in manifests.tests:
michael@0:         if 'runOnDay' in test:
michael@0:            if calendar.day_name[calendar.weekday(*datetime.datetime.now().timetuple()[:3])].lower() == test['runOnDay'].lower():
michael@0:                tests.append(test)
michael@0:         else:
michael@0:            tests.append(test)
michael@0: 
michael@0: To recap:
michael@0: * the manifests allow you to specify test data
michael@0: * the parser gives you this data
michael@0: * you can use it however you want or process it further as you need
michael@0: 
michael@0: Tests are denoted by sections in an .ini file (see
michael@0: http://hg.mozilla.org/automation/ManifestDestiny/file/tip/manifestdestiny/tests/mozmill-example.ini).
michael@0: 
michael@0: Additional manifest files may be included with an `[include:]` directive:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [include:path-to-additional-file.manifest]
michael@0: 
michael@0: The path to included files is relative to the current manifest.
michael@0: 
michael@0: The `[DEFAULT]` section contains variables that all tests inherit from.
michael@0: 
michael@0: Included files will inherit the top-level variables but may override
michael@0: in their own `[DEFAULT]` section.
michael@0: 
michael@0: ManifestDestiny Architecture
michael@0: ````````````````````````````
michael@0: 
michael@0: There is a two- or three-layered approach to the ManifestDestiny
michael@0: architecture, depending on your needs:
michael@0: 
michael@0: 1. ManifestParser: this is a generic parser for .ini manifests that
michael@0: facilitates the `[include:]` logic and the inheritence of
michael@0: metadata. Despite the internal variable being called `self.tests`
michael@0: (an oversight), this layer has nothing in particular to do with tests.
michael@0: 
michael@0: 2. TestManifest: this is a harness-agnostic integration layer that is
michael@0: test-specific. TestManifest faciliates `skip-if` and `run-if` logic.
michael@0: 
michael@0: 3. Optionally, a harness will have an integration layer than inherits
michael@0: from TestManifest if more harness-specific customization is desired at
michael@0: the manifest level.
michael@0: 
michael@0: See the source code at https://github.com/mozilla/mozbase/tree/master/manifestdestiny
michael@0: and
michael@0: https://github.com/mozilla/mozbase/blob/master/manifestdestiny/manifestparser.py
michael@0: in particular.
michael@0: 
michael@0: Using Manifests
michael@0: ```````````````
michael@0: 
michael@0: A test harness will normally call `TestManifest.active_tests`:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     def active_tests(self, exists=True, disabled=True, **tags):
michael@0: 
michael@0: The manifests are passed to the `__init__` or `read` methods with
michael@0: appropriate arguments.  `active_tests` then allows you to select the
michael@0: tests you want:
michael@0: 
michael@0: - exists : return only existing tests
michael@0: - disabled : whether to return disabled tests; if not these will be
michael@0:   filtered out; if True (the default), the `disabled` key of a
michael@0:   test's metadata will be present and will be set to the reason that a
michael@0:   test is disabled
michael@0: - tags : keys and values to filter on (e.g. `os='linux'`)
michael@0: 
michael@0: `active_tests` looks for tests with `skip-if`
michael@0: `run-if`.  If the condition is or is not fulfilled,
michael@0: respectively, the test is marked as disabled.  For instance, if you
michael@0: pass `**dict(os='linux')` as `**tags`, if a test contains a line
michael@0: `skip-if = os == 'linux'` this test will be disabled, or
michael@0: `run-if = os = 'win'` in which case the test will also be disabled.  It
michael@0: is up to the harness to pass in tags appropriate to its usage.
michael@0: 
michael@0: Creating Manifests
michael@0: ``````````````````
michael@0: 
michael@0: ManifestDestiny comes with a console script, `manifestparser create`, that
michael@0: may be used to create a seed manifest structure from a directory of
michael@0: files.  Run `manifestparser help create` for usage information.
michael@0: 
michael@0: Copying Manifests
michael@0: `````````````````
michael@0: 
michael@0: To copy tests and manifests from a source:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser [options] copy from_manifest to_directory -tag1 -tag2 `key1=value1 key2=value2 ...
michael@0: 
michael@0: Updating Tests
michael@0: ``````````````
michael@0: 
michael@0: To update the tests associated with with a manifest from a source
michael@0: directory:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser [options] update manifest from_directory -tag1 -tag2 `key1=value1 `key2=value2 ...
michael@0: 
michael@0: Usage example
michael@0: `````````````
michael@0: 
michael@0: Here is an example of how to create manifests for a directory tree and
michael@0: update the tests listed in the manifests from an external source.
michael@0: 
michael@0: Creating Manifests
michael@0: ``````````````````
michael@0: 
michael@0: Let's say you want to make a series of manifests for a given directory structure containing `.js` test files:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     testing/mozmill/tests/firefox/
michael@0:     testing/mozmill/tests/firefox/testAwesomeBar/
michael@0:     testing/mozmill/tests/firefox/testPreferences/
michael@0:     testing/mozmill/tests/firefox/testPrivateBrowsing/
michael@0:     testing/mozmill/tests/firefox/testSessionStore/
michael@0:     testing/mozmill/tests/firefox/testTechnicalTools/
michael@0:     testing/mozmill/tests/firefox/testToolbar/
michael@0:     testing/mozmill/tests/firefox/restartTests
michael@0: 
michael@0: You can use `manifestparser create` to do this:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     $ manifestparser help create
michael@0:     Usage: manifestparser.py [options] create directory <directory> <...>
michael@0: 
michael@0:          create a manifest from a list of directories
michael@0: 
michael@0:     Options:
michael@0:       -p PATTERN, `pattern=PATTERN
michael@0:                             glob pattern for files
michael@0:       -i IGNORE, `ignore=IGNORE
michael@0:                             directories to ignore
michael@0:       -w IN_PLACE, --in-place=IN_PLACE
michael@0:                             Write .ini files in place; filename to write to
michael@0: 
michael@0: We only want `.js` files and we want to skip the `restartTests` directory.
michael@0: We also want to write a manifest per directory, so I use the `--in-place`
michael@0: option to write the manifests:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser create . -i restartTests -p '*.js' -w manifest.ini
michael@0: 
michael@0: This creates a manifest.ini per directory that we care about with the JS test files:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     testing/mozmill/tests/firefox/manifest.ini
michael@0:     testing/mozmill/tests/firefox/testAwesomeBar/manifest.ini
michael@0:     testing/mozmill/tests/firefox/testPreferences/manifest.ini
michael@0:     testing/mozmill/tests/firefox/testPrivateBrowsing/manifest.ini
michael@0:     testing/mozmill/tests/firefox/testSessionStore/manifest.ini
michael@0:     testing/mozmill/tests/firefox/testTechnicalTools/manifest.ini
michael@0:     testing/mozmill/tests/firefox/testToolbar/manifest.ini
michael@0: 
michael@0: The top-level `manifest.ini` merely has `[include:]` references to the sub manifests:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     [include:testAwesomeBar/manifest.ini]
michael@0:     [include:testPreferences/manifest.ini]
michael@0:     [include:testPrivateBrowsing/manifest.ini]
michael@0:     [include:testSessionStore/manifest.ini]
michael@0:     [include:testTechnicalTools/manifest.ini]
michael@0:     [include:testToolbar/manifest.ini]
michael@0: 
michael@0: Each sub-level manifest contains the (`.js`) test files relative to it.
michael@0: 
michael@0: Updating the tests from manifests
michael@0: `````````````````````````````````
michael@0: 
michael@0: You may need to update tests as given in manifests from a different source directory.
michael@0: `manifestparser update` was made for just this purpose:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     Usage: manifestparser [options] update manifest directory -tag1 -tag2 `key1=value1 --key2=value2 ...
michael@0: 
michael@0:         update the tests as listed in a manifest from a directory
michael@0: 
michael@0: To update from a directory of tests in `~/mozmill/src/mozmill-tests/firefox/` run:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser update manifest.ini ~/mozmill/src/mozmill-tests/firefox/
michael@0: 
michael@0: Tests
michael@0: `````
michael@0: 
michael@0: ManifestDestiny includes a suite of tests:
michael@0: 
michael@0: https://github.com/mozilla/mozbase/tree/master/manifestdestiny/tests
michael@0: 
michael@0: `test_manifest.txt` is a doctest that may be helpful in figuring out
michael@0: how to use the API.  Tests are run via `python test.py`.
michael@0: 
michael@0: Bugs
michael@0: ````
michael@0: 
michael@0: Please file any bugs or feature requests at
michael@0: 
michael@0: https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=ManifestParser
michael@0: 
michael@0: Or contact jhammel @mozilla.org or in #ateam on irc.mozilla.org
michael@0: 
michael@0: CLI
michael@0: ```
michael@0: 
michael@0: Run `manifestparser help` for usage information.
michael@0: 
michael@0: To create a manifest from a set of directories:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser [options] create directory <directory> <...> [create-options]
michael@0: 
michael@0: To output a manifest of tests:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser [options] write manifest <manifest> <...> -tag1 -tag2 --key1=value1 --key2=value2 ...
michael@0: 
michael@0: To copy tests and manifests from a source:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser [options] copy from_manifest to_manifest -tag1 -tag2 `key1=value1 key2=value2 ...
michael@0: 
michael@0: To update the tests associated with with a manifest from a source
michael@0: directory:
michael@0: 
michael@0: .. code-block:: text
michael@0: 
michael@0:     manifestparser [options] update manifest from_directory -tag1 -tag2 --key1=value1 --key2=value2 ...
michael@0: 
michael@0: Design Considerations
michael@0: `````````````````````
michael@0: 
michael@0: Contrary to some opinion, manifestparser.py and the associated .ini
michael@0: format were not magically plucked from the sky but were descended upon
michael@0: through several design considerations.
michael@0: 
michael@0: * test manifests should be ordered.  While python 2.6 and greater has
michael@0:   a ConfigParser that can use an ordered dictionary, it is a
michael@0:   requirement that we support python 2.4 for the build + testing
michael@0:   environment.  To that end, a `read_ini` function was implemented
michael@0:   in manifestparser.py that should be the equivalent of the .ini
michael@0:   dialect used by ConfigParser.
michael@0: 
michael@0: * the manifest format should be easily human readable/writable.  While
michael@0:   there was initially some thought of using JSON, there was pushback
michael@0:   that JSON was not easily editable.  An ideal manifest format would
michael@0:   degenerate to a line-separated list of files.  While .ini format
michael@0:   requires an additional `[]` per line, and while there have been
michael@0:   complaints about this, hopefully this is good enough.
michael@0: 
michael@0: * python does not have an in-built YAML parser.  Since it was
michael@0:   undesirable for manifestparser.py to have any dependencies, YAML was
michael@0:   dismissed as a format.
michael@0: 
michael@0: * we could have used a proprietary format but decided against it.
michael@0:   Everyone knows .ini and there are good tools to deal with it.
michael@0:   However, since read_ini is the only function that transforms a
michael@0:   manifest to a list of key, value pairs, while the implications for
michael@0:   changing the format impacts downstream code, doing so should be
michael@0:   programmatically simple.
michael@0: 
michael@0: * there should be a single file that may easily be
michael@0:   transported. Traditionally, test harnesses have lived in
michael@0:   mozilla-central. This is less true these days and it is increasingly
michael@0:   likely that more tests will not live in mozilla-central going
michael@0:   forward.  So `manifestparser.py` should be highly consumable. To
michael@0:   this end, it is a single file, as appropriate to mozilla-central,
michael@0:   which is also a working python package deployed to PyPI for easy
michael@0:   installation.
michael@0: 
michael@0: Historical Reference
michael@0: ````````````````````
michael@0: 
michael@0: Date-ordered list of links about how manifests came to be where they are today::
michael@0: 
michael@0: * https://wiki.mozilla.org/Auto-tools/Projects/UniversalManifest
michael@0: * http://alice.nodelman.net/blog/post/2010/05/
michael@0: * http://alice.nodelman.net/blog/post/universal-manifest-for-unit-tests-a-proposal/
michael@0: * https://elvis314.wordpress.com/2010/07/05/improving-personal-hygiene-by-adjusting-mochitests/
michael@0: * https://elvis314.wordpress.com/2010/07/27/types-of-data-we-care-about-in-a-manifest/
michael@0: * https://bugzilla.mozilla.org/show_bug.cgi?id=585106
michael@0: * http://elvis314.wordpress.com/2011/05/20/converting-xpcshell-from-listing-directories-to-a-manifest/
michael@0: * https://bugzilla.mozilla.org/show_bug.cgi?id=616999
michael@0: * https://developer.mozilla.org/en/Writing_xpcshell-based_unit_tests#Adding_your_tests_to_the_xpcshell_manifest