The Tor Browser / file comparison
comparison: build/docs/mozbuild-files.rst
build/docs/mozbuild-files.rst
- branch
- TOR_BUG_9701
- changeset 15
- b8a032363ba2
equal
deleted
inserted
replaced
| -1:000000000000 | 0:f6234dc2cf74 |
|---|---|
| 1 .. _mozbuild-files: | |
| 2 | |
| 3 =============== | |
| 4 moz.build Files | |
| 5 =============== | |
| 6 | |
| 7 ``moz.build`` files are the mechanism by which tree metadata (notably | |
| 8 the build configuration) is defined. | |
| 9 | |
| 10 Directories in the tree contain ``moz.build`` files which declare | |
| 11 functionality for their respective part of the tree. This includes | |
| 12 things such as the list of C++ files to compile, where to find tests, | |
| 13 etc. | |
| 14 | |
| 15 ``moz.build`` files are actually Python scripts. However, their | |
| 16 execution is governed by special rules. This is explained below. | |
| 17 | |
| 18 moz.build Python Sandbox | |
| 19 ======================== | |
| 20 | |
| 21 As mentioned above, ``moz.build`` files are Python scripts. However, | |
| 22 they are executed in a special Python *sandbox* that significantly | |
| 23 changes and limits the execution environment. The environment is so | |
| 24 different, it's doubtful most ``moz.build`` files would execute without | |
| 25 error if executed by a vanilla Python interpreter (e.g. ``python | |
| 26 moz.build``. | |
| 27 | |
| 28 The following properties make execution of ``moz.build`` files special: | |
| 29 | |
| 30 1. The execution environment exposes a limited subset of Python. | |
| 31 2. There is a special set of global symbols and an enforced naming | |
| 32 convention of symbols. | |
| 33 | |
| 34 The limited subset of Python is actually an extremely limited subset. | |
| 35 Only a few symbols from ``__builtins__`` are exposed. These include | |
| 36 ``True``, ``False``, and ``None``. Global functions like ``import``, | |
| 37 ``print``, and ``open`` aren't available. Without these, ``moz.build`` | |
| 38 files can do very little. *This is by design*. | |
| 39 | |
| 40 The execution sandbox treats all ``UPPERCASE`` variables specially. Any | |
| 41 ``UPPERCASE`` variable must be known to the sandbox before the script | |
| 42 executes. Any attempt to read or write to an unknown ``UPPERCASE`` | |
| 43 variable will result in an exception being raised. Furthermore, the | |
| 44 types of all ``UPPERCASE`` variables is strictly enforced. Attempts to | |
| 45 assign an incompatible type to an ``UPPERCASE`` variable will result in | |
| 46 an exception being raised. | |
| 47 | |
| 48 The strictness of behavior with ``UPPERCASE`` variables is a very | |
| 49 intentional design decision. By ensuring strict behavior, any operation | |
| 50 involving an ``UPPERCASE`` variable is guaranteed to have well-defined | |
| 51 side-effects. Previously, when the build configuration was defined in | |
| 52 ``Makefiles``, assignments to variables that did nothing would go | |
| 53 unnoticed. ``moz.build`` files fix this problem by eliminating the | |
| 54 potential for false promises. | |
| 55 | |
| 56 In the sandbox, all ``UPPERCASE`` variables are globals and all | |
| 57 non-``UPPERCASE`` variables are locals. After a ``moz.build`` file has | |
| 58 completed execution, only the globals are used to retrieve state. | |
| 59 | |
| 60 The set of variables and functions available to the Python sandbox is | |
| 61 defined by the :py:mod:`mozbuild.frontend.sandbox_symbols` module. The | |
| 62 data structures in this module are consumed by the | |
| 63 :py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct | |
| 64 the sandbox. There are tests to ensure that the set of symbols exposed | |
| 65 to an empty sandbox are all defined in the ``sandbox_symbols`` module. | |
| 66 This module also contains documentation for each symbol, so nothing can | |
| 67 sneak into the sandbox without being explicitly defined and documented. | |
| 68 | |
| 69 Reading and Traversing moz.build Files | |
| 70 ====================================== | |
| 71 | |
| 72 The process responsible for reading ``moz.build`` files simply starts at | |
| 73 a root ``moz.build`` file, processes it, emits the globals namespace to | |
| 74 a consumer, and then proceeds to process additional referenced | |
| 75 ``moz.build`` files from the original file. The consumer then examines | |
| 76 the globals/``UPPERCASE`` variables set as part of execution and then | |
| 77 converts the data therein to Python class instances. | |
| 78 | |
| 79 The executed Python sandbox is essentially represented as a dictionary | |
| 80 of all the special ``UPPERCASE`` variables populated during its | |
| 81 execution. | |
| 82 | |
| 83 The code for reading ``moz.build`` files lives in | |
| 84 :py:mod:`mozbuild.frontend.reader`. The evaluated Python sandboxes are | |
| 85 passed into :py:mod:`mozbuild.frontend.emitter`, which converts them to | |
| 86 classes defined in :py:mod:`mozbuild.frontend.data`. Each class in this | |
| 87 module define a domain-specific component of tree metdata. e.g. there | |
| 88 will be separate classes that represent a JavaScript file vs a compiled | |
| 89 C++ file or test manifests. This means downstream consumers of this data | |
| 90 can filter on class types to only consume what they are interested in. | |
| 91 | |
| 92 There is no well-defined mapping between ``moz.build`` file instances | |
| 93 and the number of :py:mod:`mozbuild.frontend.data` classes derived from | |
| 94 each. Depending on the content of the ``moz.build`` file, there may be 1 | |
| 95 object derived or 100. | |
| 96 | |
| 97 The purpose of the ``emitter`` layer between low-level sandbox execution | |
| 98 and metadata representation is to facilitate a unified normalization and | |
| 99 verification step. There are multiple downstream consumers of the | |
| 100 ``moz.build``-derived data and many will perform the same actions. This | |
| 101 logic can be complicated, so we a component dedicated to it. | |
| 102 | |
| 103 Other Notes | |
| 104 =========== | |
| 105 | |
| 106 :py:class:`mozbuild.frontend.reader.BuildReader`` and | |
| 107 :py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a | |
| 108 stream-based API courtesy of generators. When you hook them up properly, | |
| 109 the :py:mod:`mozbuild.frontend.data` classes are emitted before all | |
| 110 ``moz.build`` files have been read. This means that downstream errors | |
| 111 are raised soon after sandbox execution. | |
| 112 | |
| 113 Lots of the code for evaluating Python sandboxes is applicable to | |
| 114 non-Mozilla systems. In theory, it could be extracted into a standalone | |
| 115 and generic package. However, until there is a need, there will | |
| 116 likely be some tightly coupled bits. |
