1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/build/docs/mozbuild-files.rst Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,116 @@
1.4 +.. _mozbuild-files:
1.5 +
1.6 +===============
1.7 +moz.build Files
1.8 +===============
1.9 +
1.10 +``moz.build`` files are the mechanism by which tree metadata (notably
1.11 +the build configuration) is defined.
1.12 +
1.13 +Directories in the tree contain ``moz.build`` files which declare
1.14 +functionality for their respective part of the tree. This includes
1.15 +things such as the list of C++ files to compile, where to find tests,
1.16 +etc.
1.17 +
1.18 +``moz.build`` files are actually Python scripts. However, their
1.19 +execution is governed by special rules. This is explained below.
1.20 +
1.21 +moz.build Python Sandbox
1.22 +========================
1.23 +
1.24 +As mentioned above, ``moz.build`` files are Python scripts. However,
1.25 +they are executed in a special Python *sandbox* that significantly
1.26 +changes and limits the execution environment. The environment is so
1.27 +different, it's doubtful most ``moz.build`` files would execute without
1.28 +error if executed by a vanilla Python interpreter (e.g. ``python
1.29 +moz.build``.
1.30 +
1.31 +The following properties make execution of ``moz.build`` files special:
1.32 +
1.33 +1. The execution environment exposes a limited subset of Python.
1.34 +2. There is a special set of global symbols and an enforced naming
1.35 + convention of symbols.
1.36 +
1.37 +The limited subset of Python is actually an extremely limited subset.
1.38 +Only a few symbols from ``__builtins__`` are exposed. These include
1.39 +``True``, ``False``, and ``None``. Global functions like ``import``,
1.40 +``print``, and ``open`` aren't available. Without these, ``moz.build``
1.41 +files can do very little. *This is by design*.
1.42 +
1.43 +The execution sandbox treats all ``UPPERCASE`` variables specially. Any
1.44 +``UPPERCASE`` variable must be known to the sandbox before the script
1.45 +executes. Any attempt to read or write to an unknown ``UPPERCASE``
1.46 +variable will result in an exception being raised. Furthermore, the
1.47 +types of all ``UPPERCASE`` variables is strictly enforced. Attempts to
1.48 +assign an incompatible type to an ``UPPERCASE`` variable will result in
1.49 +an exception being raised.
1.50 +
1.51 +The strictness of behavior with ``UPPERCASE`` variables is a very
1.52 +intentional design decision. By ensuring strict behavior, any operation
1.53 +involving an ``UPPERCASE`` variable is guaranteed to have well-defined
1.54 +side-effects. Previously, when the build configuration was defined in
1.55 +``Makefiles``, assignments to variables that did nothing would go
1.56 +unnoticed. ``moz.build`` files fix this problem by eliminating the
1.57 +potential for false promises.
1.58 +
1.59 +In the sandbox, all ``UPPERCASE`` variables are globals and all
1.60 +non-``UPPERCASE`` variables are locals. After a ``moz.build`` file has
1.61 +completed execution, only the globals are used to retrieve state.
1.62 +
1.63 +The set of variables and functions available to the Python sandbox is
1.64 +defined by the :py:mod:`mozbuild.frontend.sandbox_symbols` module. The
1.65 +data structures in this module are consumed by the
1.66 +:py:class:`mozbuild.frontend.reader.MozbuildSandbox` class to construct
1.67 +the sandbox. There are tests to ensure that the set of symbols exposed
1.68 +to an empty sandbox are all defined in the ``sandbox_symbols`` module.
1.69 +This module also contains documentation for each symbol, so nothing can
1.70 +sneak into the sandbox without being explicitly defined and documented.
1.71 +
1.72 +Reading and Traversing moz.build Files
1.73 +======================================
1.74 +
1.75 +The process responsible for reading ``moz.build`` files simply starts at
1.76 +a root ``moz.build`` file, processes it, emits the globals namespace to
1.77 +a consumer, and then proceeds to process additional referenced
1.78 +``moz.build`` files from the original file. The consumer then examines
1.79 +the globals/``UPPERCASE`` variables set as part of execution and then
1.80 +converts the data therein to Python class instances.
1.81 +
1.82 +The executed Python sandbox is essentially represented as a dictionary
1.83 +of all the special ``UPPERCASE`` variables populated during its
1.84 +execution.
1.85 +
1.86 +The code for reading ``moz.build`` files lives in
1.87 +:py:mod:`mozbuild.frontend.reader`. The evaluated Python sandboxes are
1.88 +passed into :py:mod:`mozbuild.frontend.emitter`, which converts them to
1.89 +classes defined in :py:mod:`mozbuild.frontend.data`. Each class in this
1.90 +module define a domain-specific component of tree metdata. e.g. there
1.91 +will be separate classes that represent a JavaScript file vs a compiled
1.92 +C++ file or test manifests. This means downstream consumers of this data
1.93 +can filter on class types to only consume what they are interested in.
1.94 +
1.95 +There is no well-defined mapping between ``moz.build`` file instances
1.96 +and the number of :py:mod:`mozbuild.frontend.data` classes derived from
1.97 +each. Depending on the content of the ``moz.build`` file, there may be 1
1.98 +object derived or 100.
1.99 +
1.100 +The purpose of the ``emitter`` layer between low-level sandbox execution
1.101 +and metadata representation is to facilitate a unified normalization and
1.102 +verification step. There are multiple downstream consumers of the
1.103 +``moz.build``-derived data and many will perform the same actions. This
1.104 +logic can be complicated, so we a component dedicated to it.
1.105 +
1.106 +Other Notes
1.107 +===========
1.108 +
1.109 +:py:class:`mozbuild.frontend.reader.BuildReader`` and
1.110 +:py:class:`mozbuild.frontend.reader.TreeMetadataEmitter`` have a
1.111 +stream-based API courtesy of generators. When you hook them up properly,
1.112 +the :py:mod:`mozbuild.frontend.data` classes are emitted before all
1.113 +``moz.build`` files have been read. This means that downstream errors
1.114 +are raised soon after sandbox execution.
1.115 +
1.116 +Lots of the code for evaluating Python sandboxes is applicable to
1.117 +non-Mozilla systems. In theory, it could be extracted into a standalone
1.118 +and generic package. However, until there is a need, there will
1.119 +likely be some tightly coupled bits.
