1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/build/docs/python.rst Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,178 @@
1.4 +.. _python:
1.5 +
1.6 +===========================
1.7 +Python and the Build System
1.8 +===========================
1.9 +
1.10 +The Python programming language is used significantly in the build
1.11 +system. If we need to write code for the build system or for a tool
1.12 +related to the build system, Python is typically the first choice.
1.13 +
1.14 +Python Requirements
1.15 +===================
1.16 +
1.17 +The tree requires Python 2.7.3 or greater but not Python 3 to build.
1.18 +All Python packages not in the Python distribution are included in the
1.19 +source tree. So all you should need is a vanilla Python install and you
1.20 +should be good to go.
1.21 +
1.22 +Only CPython (the Python distribution available from www.python.org) is
1.23 +supported.
1.24 +
1.25 +We require Python 2.7.3 (and not say 2.7.2) to build because Python
1.26 +2.7.3 contains numerous bug fixes, especially around the area of Unicode
1.27 +handling. These bug fixes are extremely annoying and have to be worked
1.28 +around. The build maintainers were tired of doing this, so the minimum
1.29 +version requirement was upped (bug 870420).
1.30 +
1.31 +We intend to eventually support Python 3. This will come by way of dual
1.32 +2.7/3.x compatibility because a single flag day conversion to 3.x will
1.33 +be too cumbersome given the amount of Python that would need converted.
1.34 +We will not know which 3.x minor release we are targeting until this
1.35 +effort is underway. This is tracked in bug 636155.
1.36 +
1.37 +Compiled Python Packages
1.38 +========================
1.39 +
1.40 +There are some features of the build that rely on compiled Python packages
1.41 +(packages containing C source). These features are currently all
1.42 +optional because not every system contains the Python development
1.43 +headers required to build these extensions.
1.44 +
1.45 +We recommend you have the Python development headers installed (``mach
1.46 +bootstrap`` should do this for you) so you can take advantage of these
1.47 +features.
1.48 +
1.49 +Issues with OS X System Python
1.50 +==============================
1.51 +
1.52 +The Python that ships with OS X has historically been littered with
1.53 +subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't
1.54 +ship with Python 2.7.3 (10.8 ships with 2.7.2).
1.55 +
1.56 +OS X 10.8 and below users will be required to install a new Python
1.57 +distribution. This may not be necessary for OS X 10.9+. However, we
1.58 +still recommend installing a separate Python because of the history with
1.59 +OS X's system Python issues.
1.60 +
1.61 +We recommend installing Python through Homebrew or MacPorts. If you run
1.62 +``mach bootstrap``, this should be done for you.
1.63 +
1.64 +Virtualenvs
1.65 +===========
1.66 +
1.67 +The build system relies heavily on
1.68 +`virtualenvs <http://www.virtualenv.org/en/latest/>`_. Virtualenvs are
1.69 +standalone and isolated Python environments. The problem a virtualenv
1.70 +solves is that of dependencies across multiple Python components. If two
1.71 +components on a system relied on different versions of a package, there
1.72 +could be a conflict. Instead of managing multiple versions of a package
1.73 +simultaneously, Python and virtualenvs take the route that it is easier
1.74 +to just keep them separate so there is no potential for conflicts.
1.75 +
1.76 +Very early in the build process, a virtualenv is created inside the
1.77 +:term:`object directory`. The virtualenv is configured such that it can
1.78 +find all the Python packages in the source tree. The code for this lives
1.79 +in :py:mod:`mozbuild.virtualenv`.
1.80 +
1.81 +Deficiencies
1.82 +------------
1.83 +
1.84 +There are numerous deficiencies with the way virtualenvs are handled in
1.85 +the build system.
1.86 +
1.87 +* mach reinvents the virtualenv.
1.88 +
1.89 + There is code in ``build/mach_bootstrap.py`` that configures ``sys.path``
1.90 + much the same way the virtualenv does. There are various bugs tracking
1.91 + this. However, no clear solution has yet been devised. It's not a huge
1.92 + problem and thus not a huge priority.
1.93 +
1.94 +* They aren't preserved across copies and packaging.
1.95 +
1.96 + If you attempt to copy an entire tree from one machine to another or
1.97 + from one directory to another, chances are the virtualenv will fall
1.98 + apart. It would be nice if we could preserve it somehow. Instead of
1.99 + actually solving portable virtualenvs, all we really need to solve is
1.100 + encapsulating the logic for populating the virtualenv along with all
1.101 + dependent files in the appropriate place.
1.102 +
1.103 +* .pyc files written to source directory.
1.104 +
1.105 + We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file
1.106 + is a special file that contains a list of paths. Python will take the
1.107 + set of listed paths encountered in ``.pth`` files and add them to
1.108 + ``sys.path``.
1.109 +
1.110 + When Python compiles a ``.py`` file to bytecode, it writes out a
1.111 + ``.pyc`` file so it doesn't have to perform this compilation again.
1.112 + It puts these ``.pyc`` files alongside the ``.pyc`` file. Python
1.113 + provides very little control for determing where these ``.pyc`` files
1.114 + go, even in Python 3 (which offers customer importers).
1.115 +
1.116 + With ``.pth`` files pointing back to directories in the source tree
1.117 + and not the object directory, ``.pyc`` files are created in the source
1.118 + tree. This is bad because when Python imports a module, it first looks
1.119 + for a ``.pyc`` file before the ``.py`` file. If there is a ``.pyc``
1.120 + file but no ``.py`` file, it will happily import the module. This
1.121 + wreaks havoc during file moves, refactoring, etc.
1.122 +
1.123 + There are various proposals for fixing this. See bug 795995.
1.124 +
1.125 +Installing Python Manually
1.126 +==========================
1.127 +
1.128 +We highly recommend you use your system's package manager or a
1.129 +well-supported 3rd party package manager to install Python for you. If
1.130 +these are not available to you, we recommend the following tools for
1.131 +installing Python:
1.132 +
1.133 +* `buildout.python <https://github.com/collective/buildout.python>`_
1.134 +* `pyenv <https://github.com/yyuu/pyenv>`_
1.135 +* An official installer from http://www.python.org.
1.136 +
1.137 +If all else fails, consider compiling Python from source manually. But this
1.138 +should be viewed as the least desirable option.
1.139 +
1.140 +Common Issues with Python
1.141 +=========================
1.142 +
1.143 +Upgrading your Python distribution breaks the virtualenv
1.144 +--------------------------------------------------------
1.145 +
1.146 +If you upgrade the Python distribution (e.g. install Python 2.7.5
1.147 +from 2.7.3, chances are parts of the virtualenv will break.
1.148 +This commonly manifests as a cryptic ``Cannot import XXX`` exception.
1.149 +More often than not, the module being imported contains binary/compiled
1.150 +components.
1.151 +
1.152 +If you upgrade or reinstall your Python distribution, we recommend
1.153 +clobbering your build.
1.154 +
1.155 +Packages installed at the system level conflict with build system's
1.156 +-------------------------------------------------------------------
1.157 +
1.158 +It is common for people to install Python packages using ``sudo`` (e.g.
1.159 +``sudo pip install psutil``) or with the system's package manager
1.160 +(e.g. ``apt-get install python-mysql``.
1.161 +
1.162 +A problem with this is that packages installed at the system level may
1.163 +conflict with the package provided by the source tree. As of bug 907902
1.164 +and changeset f18eae7c3b27 (September 16, 2013), this should no longer
1.165 +be an issue since the virtualenv created as part of the build doesn't
1.166 +add the system's ``site-packages`` directory to ``sys.path``. However,
1.167 +poorly installed packages may still find a way to creep into the mix and
1.168 +interfere with our virtualenv.
1.169 +
1.170 +As a general principle, we recommend against using your system's package
1.171 +manager or using ``sudo`` to install Python packages. Instead, create
1.172 +virtualenvs and isolated Python environments for all of your Python
1.173 +projects.
1.174 +
1.175 +Python on $PATH is not appropriate
1.176 +----------------------------------
1.177 +
1.178 +Tools like ``mach`` will look for Python by performing ``/usr/bin/env
1.179 +python`` or equivalent. Please be sure the appropriate Python 2.7.3+
1.180 +path is on $PATH. On OS X, this likely means you'll need to modify your
1.181 +shell's init script to put something ahead of ``/usr/bin``.
