build/docs/mozbuild-files.rst@97036ab72558 (annotated)
build/docs/mozbuild-files.rst
Tue, 06 Jan 2015 21:39:09 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Tue, 06 Jan 2015 21:39:09 +0100
- branch
- TOR_BUG_9701
- changeset 8
- 97036ab72558
- permissions
- -rw-r--r--
Conditionally force memory storage according to privacy.thirdparty.isolate;
This solves Tor bug #9701, complying with disk avoidance documented in
https://www.torproject.org/projects/torbrowser/design/#disk-avoidance.
| michael@0 | 1 | .. _mozbuild-files: |
| michael@0 | 2 | |
| michael@0 | 3 | =============== |
| michael@0 | 4 | moz.build Files |
| michael@0 | 5 | =============== |
| michael@0 | 6 | |
| michael@0 | 7 | ``moz.build`` files are the mechanism by which tree metadata (notably |
| michael@0 | 8 | the build configuration) is defined. |
| michael@0 | 9 | |
| michael@0 | 10 | Directories in the tree contain ``moz.build`` files which declare |
| michael@0 | 11 | functionality for their respective part of the tree. This includes |
| michael@0 | 12 | things such as the list of C++ files to compile, where to find tests, |
| michael@0 | 13 | etc. |
| michael@0 | 14 | |
| michael@0 | 15 | ``moz.build`` files are actually Python scripts. However, their |
| michael@0 | 16 | execution is governed by special rules. This is explained below. |
| michael@0 | 17 | |
| michael@0 | 18 | moz.build Python Sandbox |
| michael@0 | 19 | ======================== |
| michael@0 | 20 | |
| michael@0 | 21 | As mentioned above, ``moz.build`` files are Python scripts. However, |
| michael@0 | 22 | they are executed in a special Python *sandbox* that significantly |
| michael@0 | 23 | changes and limits the execution environment. The environment is so |
| michael@0 | 24 | different, it's doubtful most ``moz.build`` files would execute without |
| michael@0 | 25 | error if executed by a vanilla Python interpreter (e.g. ``python |
| michael@0 | 26 | moz.build``. |
| michael@0 | 27 | |
| michael@0 | 28 | The following properties make execution of ``moz.build`` files special: |
| michael@0 | 29 | |
| michael@0 | 30 | 1. The execution environment exposes a limited subset of Python. |
| michael@0 | 31 | 2. There is a special set of global symbols and an enforced naming |
| michael@0 | 32 | convention of symbols. |
| michael@0 | 33 | |
| michael@0 | 34 | The limited subset of Python is actually an extremely limited subset. |
| michael@0 | 35 | Only a few symbols from ``__builtins__`` are exposed. These include |
| michael@0 | 36 | ``True``, ``False``, and ``None``. Global functions like ``import``, |
| michael@0 | 37 | ``print``, and ``open`` aren't available. Without these, ``moz.build`` |
| michael@0 | 38 | files can do very little. *This is by design*. |
| michael@0 | 39 | |
| michael@0 | 40 | The execution sandbox treats all ``UPPERCASE`` variables specially. Any |
| michael@0 | 41 | ``UPPERCASE`` variable must be known to the sandbox before the script |
| michael@0 | 42 | executes. Any attempt to read or write to an unknown ``UPPERCASE`` |
| michael@0 | 43 | variable will result in an exception being raised. Furthermore, the |
| michael@0 | 44 | types of all ``UPPERCASE`` variables is strictly enforced. Attempts to |
| michael@0 | 45 | assign an incompatible type to an ``UPPERCASE`` variable will result in |
| michael@0 | 46 | an exception being raised. |
| michael@0 | 47 | |
| michael@0 | 48 | The strictness of behavior with ``UPPERCASE`` variables is a very |
| michael@0 | 49 | intentional design decision. By ensuring strict behavior, any operation |
| michael@0 | 50 | involving an ``UPPERCASE`` variable is guaranteed to have well-defined |
| michael@0 | 51 | side-effects. Previously, when the build configuration was defined in |
| michael@0 | 52 | ``Makefiles``, assignments to variables that did nothing would go |
| michael@0 | 53 | unnoticed. ``moz.build`` files fix this problem by eliminating the |
| michael@0 | 54 | potential for false promises. |
| michael@0 | 55 | |
| michael@0 | 56 | In the sandbox, all ``UPPERCASE`` variables are globals and all |
| michael@0 | 57 | non-``UPPERCASE`` variables are locals. After a ``moz.build`` file has |
| michael@0 | 58 | completed execution, only the globals are used to retrieve state. |
| michael@0 | 59 | |
| michael@0 | 60 | The set of variables and functions available to the Python sandbox is |
| michael@0 | 61 | defined by the :py:mod:`mozbuild.frontend.sandbox_symbols` module. The |
| michael@0 | 62 | data structures in this module are consumed by the |
| michael@0 | 63 | :py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct |
| michael@0 | 64 | the sandbox. There are tests to ensure that the set of symbols exposed |
| michael@0 | 65 | to an empty sandbox are all defined in the ``sandbox_symbols`` module. |
| michael@0 | 66 | This module also contains documentation for each symbol, so nothing can |
| michael@0 | 67 | sneak into the sandbox without being explicitly defined and documented. |
| michael@0 | 68 | |
| michael@0 | 69 | Reading and Traversing moz.build Files |
| michael@0 | 70 | ====================================== |
| michael@0 | 71 | |
| michael@0 | 72 | The process responsible for reading ``moz.build`` files simply starts at |
| michael@0 | 73 | a root ``moz.build`` file, processes it, emits the globals namespace to |
| michael@0 | 74 | a consumer, and then proceeds to process additional referenced |
| michael@0 | 75 | ``moz.build`` files from the original file. The consumer then examines |
| michael@0 | 76 | the globals/``UPPERCASE`` variables set as part of execution and then |
| michael@0 | 77 | converts the data therein to Python class instances. |
| michael@0 | 78 | |
| michael@0 | 79 | The executed Python sandbox is essentially represented as a dictionary |
| michael@0 | 80 | of all the special ``UPPERCASE`` variables populated during its |
| michael@0 | 81 | execution. |
| michael@0 | 82 | |
| michael@0 | 83 | The code for reading ``moz.build`` files lives in |
| michael@0 | 84 | :py:mod:`mozbuild.frontend.reader`. The evaluated Python sandboxes are |
| michael@0 | 85 | passed into :py:mod:`mozbuild.frontend.emitter`, which converts them to |
| michael@0 | 86 | classes defined in :py:mod:`mozbuild.frontend.data`. Each class in this |
| michael@0 | 87 | module define a domain-specific component of tree metdata. e.g. there |
| michael@0 | 88 | will be separate classes that represent a JavaScript file vs a compiled |
| michael@0 | 89 | C++ file or test manifests. This means downstream consumers of this data |
| michael@0 | 90 | can filter on class types to only consume what they are interested in. |
| michael@0 | 91 | |
| michael@0 | 92 | There is no well-defined mapping between ``moz.build`` file instances |
| michael@0 | 93 | and the number of :py:mod:`mozbuild.frontend.data` classes derived from |
| michael@0 | 94 | each. Depending on the content of the ``moz.build`` file, there may be 1 |
| michael@0 | 95 | object derived or 100. |
| michael@0 | 96 | |
| michael@0 | 97 | The purpose of the ``emitter`` layer between low-level sandbox execution |
| michael@0 | 98 | and metadata representation is to facilitate a unified normalization and |
| michael@0 | 99 | verification step. There are multiple downstream consumers of the |
| michael@0 | 100 | ``moz.build``-derived data and many will perform the same actions. This |
| michael@0 | 101 | logic can be complicated, so we a component dedicated to it. |
| michael@0 | 102 | |
| michael@0 | 103 | Other Notes |
| michael@0 | 104 | =========== |
| michael@0 | 105 | |
| michael@0 | 106 | :py:class:`mozbuild.frontend.reader.BuildReader`` and |
| michael@0 | 107 | :py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a |
| michael@0 | 108 | stream-based API courtesy of generators. When you hook them up properly, |
| michael@0 | 109 | the :py:mod:`mozbuild.frontend.data` classes are emitted before all |
| michael@0 | 110 | ``moz.build`` files have been read. This means that downstream errors |
| michael@0 | 111 | are raised soon after sandbox execution. |
| michael@0 | 112 | |
| michael@0 | 113 | Lots of the code for evaluating Python sandboxes is applicable to |
| michael@0 | 114 | non-Mozilla systems. In theory, it could be extracted into a standalone |
| michael@0 | 115 | and generic package. However, until there is a need, there will |
| michael@0 | 116 | likely be some tightly coupled bits. |
