michael@0: .. _mozbuild-files: michael@0: michael@0: =============== michael@0: moz.build Files michael@0: =============== michael@0: michael@0: ``moz.build`` files are the mechanism by which tree metadata (notably michael@0: the build configuration) is defined. michael@0: michael@0: Directories in the tree contain ``moz.build`` files which declare michael@0: functionality for their respective part of the tree. This includes michael@0: things such as the list of C++ files to compile, where to find tests, michael@0: etc. michael@0: michael@0: ``moz.build`` files are actually Python scripts. However, their michael@0: execution is governed by special rules. This is explained below. michael@0: michael@0: moz.build Python Sandbox michael@0: ======================== michael@0: michael@0: As mentioned above, ``moz.build`` files are Python scripts. However, michael@0: they are executed in a special Python *sandbox* that significantly michael@0: changes and limits the execution environment. The environment is so michael@0: different, it's doubtful most ``moz.build`` files would execute without michael@0: error if executed by a vanilla Python interpreter (e.g. ``python michael@0: moz.build``. michael@0: michael@0: The following properties make execution of ``moz.build`` files special: michael@0: michael@0: 1. The execution environment exposes a limited subset of Python. michael@0: 2. There is a special set of global symbols and an enforced naming michael@0: convention of symbols. michael@0: michael@0: The limited subset of Python is actually an extremely limited subset. michael@0: Only a few symbols from ``__builtins__`` are exposed. These include michael@0: ``True``, ``False``, and ``None``. Global functions like ``import``, michael@0: ``print``, and ``open`` aren't available. Without these, ``moz.build`` michael@0: files can do very little. *This is by design*. michael@0: michael@0: The execution sandbox treats all ``UPPERCASE`` variables specially. Any michael@0: ``UPPERCASE`` variable must be known to the sandbox before the script michael@0: executes. Any attempt to read or write to an unknown ``UPPERCASE`` michael@0: variable will result in an exception being raised. Furthermore, the michael@0: types of all ``UPPERCASE`` variables is strictly enforced. Attempts to michael@0: assign an incompatible type to an ``UPPERCASE`` variable will result in michael@0: an exception being raised. michael@0: michael@0: The strictness of behavior with ``UPPERCASE`` variables is a very michael@0: intentional design decision. By ensuring strict behavior, any operation michael@0: involving an ``UPPERCASE`` variable is guaranteed to have well-defined michael@0: side-effects. Previously, when the build configuration was defined in michael@0: ``Makefiles``, assignments to variables that did nothing would go michael@0: unnoticed. ``moz.build`` files fix this problem by eliminating the michael@0: potential for false promises. michael@0: michael@0: In the sandbox, all ``UPPERCASE`` variables are globals and all michael@0: non-``UPPERCASE`` variables are locals. After a ``moz.build`` file has michael@0: completed execution, only the globals are used to retrieve state. michael@0: michael@0: The set of variables and functions available to the Python sandbox is michael@0: defined by the :py:mod:`mozbuild.frontend.sandbox_symbols` module. The michael@0: data structures in this module are consumed by the michael@0: :py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct michael@0: the sandbox. There are tests to ensure that the set of symbols exposed michael@0: to an empty sandbox are all defined in the ``sandbox_symbols`` module. michael@0: This module also contains documentation for each symbol, so nothing can michael@0: sneak into the sandbox without being explicitly defined and documented. michael@0: michael@0: Reading and Traversing moz.build Files michael@0: ====================================== michael@0: michael@0: The process responsible for reading ``moz.build`` files simply starts at michael@0: a root ``moz.build`` file, processes it, emits the globals namespace to michael@0: a consumer, and then proceeds to process additional referenced michael@0: ``moz.build`` files from the original file. The consumer then examines michael@0: the globals/``UPPERCASE`` variables set as part of execution and then michael@0: converts the data therein to Python class instances. michael@0: michael@0: The executed Python sandbox is essentially represented as a dictionary michael@0: of all the special ``UPPERCASE`` variables populated during its michael@0: execution. michael@0: michael@0: The code for reading ``moz.build`` files lives in michael@0: :py:mod:`mozbuild.frontend.reader`. The evaluated Python sandboxes are michael@0: passed into :py:mod:`mozbuild.frontend.emitter`, which converts them to michael@0: classes defined in :py:mod:`mozbuild.frontend.data`. Each class in this michael@0: module define a domain-specific component of tree metdata. e.g. there michael@0: will be separate classes that represent a JavaScript file vs a compiled michael@0: C++ file or test manifests. This means downstream consumers of this data michael@0: can filter on class types to only consume what they are interested in. michael@0: michael@0: There is no well-defined mapping between ``moz.build`` file instances michael@0: and the number of :py:mod:`mozbuild.frontend.data` classes derived from michael@0: each. Depending on the content of the ``moz.build`` file, there may be 1 michael@0: object derived or 100. michael@0: michael@0: The purpose of the ``emitter`` layer between low-level sandbox execution michael@0: and metadata representation is to facilitate a unified normalization and michael@0: verification step. There are multiple downstream consumers of the michael@0: ``moz.build``-derived data and many will perform the same actions. This michael@0: logic can be complicated, so we a component dedicated to it. michael@0: michael@0: Other Notes michael@0: =========== michael@0: michael@0: :py:class:`mozbuild.frontend.reader.BuildReader`` and michael@0: :py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a michael@0: stream-based API courtesy of generators. When you hook them up properly, michael@0: the :py:mod:`mozbuild.frontend.data` classes are emitted before all michael@0: ``moz.build`` files have been read. This means that downstream errors michael@0: are raised soon after sandbox execution. michael@0: michael@0: Lots of the code for evaluating Python sandboxes is applicable to michael@0: non-Mozilla systems. In theory, it could be extracted into a standalone michael@0: and generic package. However, until there is a need, there will michael@0: likely be some tightly coupled bits.