diff -r 000000000000 -r 6474c204b198 build/docs/mozbuild-files.rst --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/build/docs/mozbuild-files.rst Wed Dec 31 06:09:35 2014 +0100 @@ -0,0 +1,116 @@ +.. _mozbuild-files: + +=============== +moz.build Files +=============== + +``moz.build`` files are the mechanism by which tree metadata (notably +the build configuration) is defined. + +Directories in the tree contain ``moz.build`` files which declare +functionality for their respective part of the tree. This includes +things such as the list of C++ files to compile, where to find tests, +etc. + +``moz.build`` files are actually Python scripts. However, their +execution is governed by special rules. This is explained below. + +moz.build Python Sandbox +======================== + +As mentioned above, ``moz.build`` files are Python scripts. However, +they are executed in a special Python *sandbox* that significantly +changes and limits the execution environment. The environment is so +different, it's doubtful most ``moz.build`` files would execute without +error if executed by a vanilla Python interpreter (e.g. ``python +moz.build``. + +The following properties make execution of ``moz.build`` files special: + +1. The execution environment exposes a limited subset of Python. +2. There is a special set of global symbols and an enforced naming + convention of symbols. + +The limited subset of Python is actually an extremely limited subset. +Only a few symbols from ``__builtins__`` are exposed. These include +``True``, ``False``, and ``None``. Global functions like ``import``, +``print``, and ``open`` aren't available. Without these, ``moz.build`` +files can do very little. *This is by design*. + +The execution sandbox treats all ``UPPERCASE`` variables specially. Any +``UPPERCASE`` variable must be known to the sandbox before the script +executes. Any attempt to read or write to an unknown ``UPPERCASE`` +variable will result in an exception being raised. Furthermore, the +types of all ``UPPERCASE`` variables is strictly enforced. Attempts to +assign an incompatible type to an ``UPPERCASE`` variable will result in +an exception being raised. + +The strictness of behavior with ``UPPERCASE`` variables is a very +intentional design decision. By ensuring strict behavior, any operation +involving an ``UPPERCASE`` variable is guaranteed to have well-defined +side-effects. Previously, when the build configuration was defined in +``Makefiles``, assignments to variables that did nothing would go +unnoticed. ``moz.build`` files fix this problem by eliminating the +potential for false promises. + +In the sandbox, all ``UPPERCASE`` variables are globals and all +non-``UPPERCASE`` variables are locals. After a ``moz.build`` file has +completed execution, only the globals are used to retrieve state. + +The set of variables and functions available to the Python sandbox is +defined by the :py:mod:`mozbuild.frontend.sandbox_symbols` module. The +data structures in this module are consumed by the +:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct +the sandbox. There are tests to ensure that the set of symbols exposed +to an empty sandbox are all defined in the ``sandbox_symbols`` module. +This module also contains documentation for each symbol, so nothing can +sneak into the sandbox without being explicitly defined and documented. + +Reading and Traversing moz.build Files +====================================== + +The process responsible for reading ``moz.build`` files simply starts at +a root ``moz.build`` file, processes it, emits the globals namespace to +a consumer, and then proceeds to process additional referenced +``moz.build`` files from the original file. The consumer then examines +the globals/``UPPERCASE`` variables set as part of execution and then +converts the data therein to Python class instances. + +The executed Python sandbox is essentially represented as a dictionary +of all the special ``UPPERCASE`` variables populated during its +execution. + +The code for reading ``moz.build`` files lives in +:py:mod:`mozbuild.frontend.reader`. The evaluated Python sandboxes are +passed into :py:mod:`mozbuild.frontend.emitter`, which converts them to +classes defined in :py:mod:`mozbuild.frontend.data`. Each class in this +module define a domain-specific component of tree metdata. e.g. there +will be separate classes that represent a JavaScript file vs a compiled +C++ file or test manifests. This means downstream consumers of this data +can filter on class types to only consume what they are interested in. + +There is no well-defined mapping between ``moz.build`` file instances +and the number of :py:mod:`mozbuild.frontend.data` classes derived from +each. Depending on the content of the ``moz.build`` file, there may be 1 +object derived or 100. + +The purpose of the ``emitter`` layer between low-level sandbox execution +and metadata representation is to facilitate a unified normalization and +verification step. There are multiple downstream consumers of the +``moz.build``-derived data and many will perform the same actions. This +logic can be complicated, so we a component dedicated to it. + +Other Notes +=========== + +:py:class:`mozbuild.frontend.reader.BuildReader`` and +:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a +stream-based API courtesy of generators. When you hook them up properly, +the :py:mod:`mozbuild.frontend.data` classes are emitted before all +``moz.build`` files have been read. This means that downstream errors +are raised soon after sandbox execution. + +Lots of the code for evaluating Python sandboxes is applicable to +non-Mozilla systems. In theory, it could be extracted into a standalone +and generic package. However, until there is a need, there will +likely be some tightly coupled bits.