michael@0: .. _python: michael@0: michael@0: =========================== michael@0: Python and the Build System michael@0: =========================== michael@0: michael@0: The Python programming language is used significantly in the build michael@0: system. If we need to write code for the build system or for a tool michael@0: related to the build system, Python is typically the first choice. michael@0: michael@0: Python Requirements michael@0: =================== michael@0: michael@0: The tree requires Python 2.7.3 or greater but not Python 3 to build. michael@0: All Python packages not in the Python distribution are included in the michael@0: source tree. So all you should need is a vanilla Python install and you michael@0: should be good to go. michael@0: michael@0: Only CPython (the Python distribution available from www.python.org) is michael@0: supported. michael@0: michael@0: We require Python 2.7.3 (and not say 2.7.2) to build because Python michael@0: 2.7.3 contains numerous bug fixes, especially around the area of Unicode michael@0: handling. These bug fixes are extremely annoying and have to be worked michael@0: around. The build maintainers were tired of doing this, so the minimum michael@0: version requirement was upped (bug 870420). michael@0: michael@0: We intend to eventually support Python 3. This will come by way of dual michael@0: 2.7/3.x compatibility because a single flag day conversion to 3.x will michael@0: be too cumbersome given the amount of Python that would need converted. michael@0: We will not know which 3.x minor release we are targeting until this michael@0: effort is underway. This is tracked in bug 636155. michael@0: michael@0: Compiled Python Packages michael@0: ======================== michael@0: michael@0: There are some features of the build that rely on compiled Python packages michael@0: (packages containing C source). These features are currently all michael@0: optional because not every system contains the Python development michael@0: headers required to build these extensions. michael@0: michael@0: We recommend you have the Python development headers installed (``mach michael@0: bootstrap`` should do this for you) so you can take advantage of these michael@0: features. michael@0: michael@0: Issues with OS X System Python michael@0: ============================== michael@0: michael@0: The Python that ships with OS X has historically been littered with michael@0: subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't michael@0: ship with Python 2.7.3 (10.8 ships with 2.7.2). michael@0: michael@0: OS X 10.8 and below users will be required to install a new Python michael@0: distribution. This may not be necessary for OS X 10.9+. However, we michael@0: still recommend installing a separate Python because of the history with michael@0: OS X's system Python issues. michael@0: michael@0: We recommend installing Python through Homebrew or MacPorts. If you run michael@0: ``mach bootstrap``, this should be done for you. michael@0: michael@0: Virtualenvs michael@0: =========== michael@0: michael@0: The build system relies heavily on michael@0: `virtualenvs `_. Virtualenvs are michael@0: standalone and isolated Python environments. The problem a virtualenv michael@0: solves is that of dependencies across multiple Python components. If two michael@0: components on a system relied on different versions of a package, there michael@0: could be a conflict. Instead of managing multiple versions of a package michael@0: simultaneously, Python and virtualenvs take the route that it is easier michael@0: to just keep them separate so there is no potential for conflicts. michael@0: michael@0: Very early in the build process, a virtualenv is created inside the michael@0: :term:`object directory`. The virtualenv is configured such that it can michael@0: find all the Python packages in the source tree. The code for this lives michael@0: in :py:mod:`mozbuild.virtualenv`. michael@0: michael@0: Deficiencies michael@0: ------------ michael@0: michael@0: There are numerous deficiencies with the way virtualenvs are handled in michael@0: the build system. michael@0: michael@0: * mach reinvents the virtualenv. michael@0: michael@0: There is code in ``build/mach_bootstrap.py`` that configures ``sys.path`` michael@0: much the same way the virtualenv does. There are various bugs tracking michael@0: this. However, no clear solution has yet been devised. It's not a huge michael@0: problem and thus not a huge priority. michael@0: michael@0: * They aren't preserved across copies and packaging. michael@0: michael@0: If you attempt to copy an entire tree from one machine to another or michael@0: from one directory to another, chances are the virtualenv will fall michael@0: apart. It would be nice if we could preserve it somehow. Instead of michael@0: actually solving portable virtualenvs, all we really need to solve is michael@0: encapsulating the logic for populating the virtualenv along with all michael@0: dependent files in the appropriate place. michael@0: michael@0: * .pyc files written to source directory. michael@0: michael@0: We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file michael@0: is a special file that contains a list of paths. Python will take the michael@0: set of listed paths encountered in ``.pth`` files and add them to michael@0: ``sys.path``. michael@0: michael@0: When Python compiles a ``.py`` file to bytecode, it writes out a michael@0: ``.pyc`` file so it doesn't have to perform this compilation again. michael@0: It puts these ``.pyc`` files alongside the ``.pyc`` file. Python michael@0: provides very little control for determing where these ``.pyc`` files michael@0: go, even in Python 3 (which offers customer importers). michael@0: michael@0: With ``.pth`` files pointing back to directories in the source tree michael@0: and not the object directory, ``.pyc`` files are created in the source michael@0: tree. This is bad because when Python imports a module, it first looks michael@0: for a ``.pyc`` file before the ``.py`` file. If there is a ``.pyc`` michael@0: file but no ``.py`` file, it will happily import the module. This michael@0: wreaks havoc during file moves, refactoring, etc. michael@0: michael@0: There are various proposals for fixing this. See bug 795995. michael@0: michael@0: Installing Python Manually michael@0: ========================== michael@0: michael@0: We highly recommend you use your system's package manager or a michael@0: well-supported 3rd party package manager to install Python for you. If michael@0: these are not available to you, we recommend the following tools for michael@0: installing Python: michael@0: michael@0: * `buildout.python `_ michael@0: * `pyenv `_ michael@0: * An official installer from http://www.python.org. michael@0: michael@0: If all else fails, consider compiling Python from source manually. But this michael@0: should be viewed as the least desirable option. michael@0: michael@0: Common Issues with Python michael@0: ========================= michael@0: michael@0: Upgrading your Python distribution breaks the virtualenv michael@0: -------------------------------------------------------- michael@0: michael@0: If you upgrade the Python distribution (e.g. install Python 2.7.5 michael@0: from 2.7.3, chances are parts of the virtualenv will break. michael@0: This commonly manifests as a cryptic ``Cannot import XXX`` exception. michael@0: More often than not, the module being imported contains binary/compiled michael@0: components. michael@0: michael@0: If you upgrade or reinstall your Python distribution, we recommend michael@0: clobbering your build. michael@0: michael@0: Packages installed at the system level conflict with build system's michael@0: ------------------------------------------------------------------- michael@0: michael@0: It is common for people to install Python packages using ``sudo`` (e.g. michael@0: ``sudo pip install psutil``) or with the system's package manager michael@0: (e.g. ``apt-get install python-mysql``. michael@0: michael@0: A problem with this is that packages installed at the system level may michael@0: conflict with the package provided by the source tree. As of bug 907902 michael@0: and changeset f18eae7c3b27 (September 16, 2013), this should no longer michael@0: be an issue since the virtualenv created as part of the build doesn't michael@0: add the system's ``site-packages`` directory to ``sys.path``. However, michael@0: poorly installed packages may still find a way to creep into the mix and michael@0: interfere with our virtualenv. michael@0: michael@0: As a general principle, we recommend against using your system's package michael@0: manager or using ``sudo`` to install Python packages. Instead, create michael@0: virtualenvs and isolated Python environments for all of your Python michael@0: projects. michael@0: michael@0: Python on $PATH is not appropriate michael@0: ---------------------------------- michael@0: michael@0: Tools like ``mach`` will look for Python by performing ``/usr/bin/env michael@0: python`` or equivalent. Please be sure the appropriate Python 2.7.3+ michael@0: path is on $PATH. On OS X, this likely means you'll need to modify your michael@0: shell's init script to put something ahead of ``/usr/bin``.