The Tor Browser / file comparison
comparison: build/docs/python.rst
build/docs/python.rst
- changeset 0
- 6474c204b198
equal
deleted
inserted
replaced
| -1:000000000000 | 0:68e5a8c667e7 |
|---|---|
| 1 .. _python: | |
| 2 | |
| 3 =========================== | |
| 4 Python and the Build System | |
| 5 =========================== | |
| 6 | |
| 7 The Python programming language is used significantly in the build | |
| 8 system. If we need to write code for the build system or for a tool | |
| 9 related to the build system, Python is typically the first choice. | |
| 10 | |
| 11 Python Requirements | |
| 12 =================== | |
| 13 | |
| 14 The tree requires Python 2.7.3 or greater but not Python 3 to build. | |
| 15 All Python packages not in the Python distribution are included in the | |
| 16 source tree. So all you should need is a vanilla Python install and you | |
| 17 should be good to go. | |
| 18 | |
| 19 Only CPython (the Python distribution available from www.python.org) is | |
| 20 supported. | |
| 21 | |
| 22 We require Python 2.7.3 (and not say 2.7.2) to build because Python | |
| 23 2.7.3 contains numerous bug fixes, especially around the area of Unicode | |
| 24 handling. These bug fixes are extremely annoying and have to be worked | |
| 25 around. The build maintainers were tired of doing this, so the minimum | |
| 26 version requirement was upped (bug 870420). | |
| 27 | |
| 28 We intend to eventually support Python 3. This will come by way of dual | |
| 29 2.7/3.x compatibility because a single flag day conversion to 3.x will | |
| 30 be too cumbersome given the amount of Python that would need converted. | |
| 31 We will not know which 3.x minor release we are targeting until this | |
| 32 effort is underway. This is tracked in bug 636155. | |
| 33 | |
| 34 Compiled Python Packages | |
| 35 ======================== | |
| 36 | |
| 37 There are some features of the build that rely on compiled Python packages | |
| 38 (packages containing C source). These features are currently all | |
| 39 optional because not every system contains the Python development | |
| 40 headers required to build these extensions. | |
| 41 | |
| 42 We recommend you have the Python development headers installed (``mach | |
| 43 bootstrap`` should do this for you) so you can take advantage of these | |
| 44 features. | |
| 45 | |
| 46 Issues with OS X System Python | |
| 47 ============================== | |
| 48 | |
| 49 The Python that ships with OS X has historically been littered with | |
| 50 subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't | |
| 51 ship with Python 2.7.3 (10.8 ships with 2.7.2). | |
| 52 | |
| 53 OS X 10.8 and below users will be required to install a new Python | |
| 54 distribution. This may not be necessary for OS X 10.9+. However, we | |
| 55 still recommend installing a separate Python because of the history with | |
| 56 OS X's system Python issues. | |
| 57 | |
| 58 We recommend installing Python through Homebrew or MacPorts. If you run | |
| 59 ``mach bootstrap``, this should be done for you. | |
| 60 | |
| 61 Virtualenvs | |
| 62 =========== | |
| 63 | |
| 64 The build system relies heavily on | |
| 65 `virtualenvs <http://www.virtualenv.org/en/latest/>`_. Virtualenvs are | |
| 66 standalone and isolated Python environments. The problem a virtualenv | |
| 67 solves is that of dependencies across multiple Python components. If two | |
| 68 components on a system relied on different versions of a package, there | |
| 69 could be a conflict. Instead of managing multiple versions of a package | |
| 70 simultaneously, Python and virtualenvs take the route that it is easier | |
| 71 to just keep them separate so there is no potential for conflicts. | |
| 72 | |
| 73 Very early in the build process, a virtualenv is created inside the | |
| 74 :term:`object directory`. The virtualenv is configured such that it can | |
| 75 find all the Python packages in the source tree. The code for this lives | |
| 76 in :py:mod:`mozbuild.virtualenv`. | |
| 77 | |
| 78 Deficiencies | |
| 79 ------------ | |
| 80 | |
| 81 There are numerous deficiencies with the way virtualenvs are handled in | |
| 82 the build system. | |
| 83 | |
| 84 * mach reinvents the virtualenv. | |
| 85 | |
| 86 There is code in ``build/mach_bootstrap.py`` that configures ``sys.path`` | |
| 87 much the same way the virtualenv does. There are various bugs tracking | |
| 88 this. However, no clear solution has yet been devised. It's not a huge | |
| 89 problem and thus not a huge priority. | |
| 90 | |
| 91 * They aren't preserved across copies and packaging. | |
| 92 | |
| 93 If you attempt to copy an entire tree from one machine to another or | |
| 94 from one directory to another, chances are the virtualenv will fall | |
| 95 apart. It would be nice if we could preserve it somehow. Instead of | |
| 96 actually solving portable virtualenvs, all we really need to solve is | |
| 97 encapsulating the logic for populating the virtualenv along with all | |
| 98 dependent files in the appropriate place. | |
| 99 | |
| 100 * .pyc files written to source directory. | |
| 101 | |
| 102 We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file | |
| 103 is a special file that contains a list of paths. Python will take the | |
| 104 set of listed paths encountered in ``.pth`` files and add them to | |
| 105 ``sys.path``. | |
| 106 | |
| 107 When Python compiles a ``.py`` file to bytecode, it writes out a | |
| 108 ``.pyc`` file so it doesn't have to perform this compilation again. | |
| 109 It puts these ``.pyc`` files alongside the ``.pyc`` file. Python | |
| 110 provides very little control for determing where these ``.pyc`` files | |
| 111 go, even in Python 3 (which offers customer importers). | |
| 112 | |
| 113 With ``.pth`` files pointing back to directories in the source tree | |
| 114 and not the object directory, ``.pyc`` files are created in the source | |
| 115 tree. This is bad because when Python imports a module, it first looks | |
| 116 for a ``.pyc`` file before the ``.py`` file. If there is a ``.pyc`` | |
| 117 file but no ``.py`` file, it will happily import the module. This | |
| 118 wreaks havoc during file moves, refactoring, etc. | |
| 119 | |
| 120 There are various proposals for fixing this. See bug 795995. | |
| 121 | |
| 122 Installing Python Manually | |
| 123 ========================== | |
| 124 | |
| 125 We highly recommend you use your system's package manager or a | |
| 126 well-supported 3rd party package manager to install Python for you. If | |
| 127 these are not available to you, we recommend the following tools for | |
| 128 installing Python: | |
| 129 | |
| 130 * `buildout.python <https://github.com/collective/buildout.python>`_ | |
| 131 * `pyenv <https://github.com/yyuu/pyenv>`_ | |
| 132 * An official installer from http://www.python.org. | |
| 133 | |
| 134 If all else fails, consider compiling Python from source manually. But this | |
| 135 should be viewed as the least desirable option. | |
| 136 | |
| 137 Common Issues with Python | |
| 138 ========================= | |
| 139 | |
| 140 Upgrading your Python distribution breaks the virtualenv | |
| 141 -------------------------------------------------------- | |
| 142 | |
| 143 If you upgrade the Python distribution (e.g. install Python 2.7.5 | |
| 144 from 2.7.3, chances are parts of the virtualenv will break. | |
| 145 This commonly manifests as a cryptic ``Cannot import XXX`` exception. | |
| 146 More often than not, the module being imported contains binary/compiled | |
| 147 components. | |
| 148 | |
| 149 If you upgrade or reinstall your Python distribution, we recommend | |
| 150 clobbering your build. | |
| 151 | |
| 152 Packages installed at the system level conflict with build system's | |
| 153 ------------------------------------------------------------------- | |
| 154 | |
| 155 It is common for people to install Python packages using ``sudo`` (e.g. | |
| 156 ``sudo pip install psutil``) or with the system's package manager | |
| 157 (e.g. ``apt-get install python-mysql``. | |
| 158 | |
| 159 A problem with this is that packages installed at the system level may | |
| 160 conflict with the package provided by the source tree. As of bug 907902 | |
| 161 and changeset f18eae7c3b27 (September 16, 2013), this should no longer | |
| 162 be an issue since the virtualenv created as part of the build doesn't | |
| 163 add the system's ``site-packages`` directory to ``sys.path``. However, | |
| 164 poorly installed packages may still find a way to creep into the mix and | |
| 165 interfere with our virtualenv. | |
| 166 | |
| 167 As a general principle, we recommend against using your system's package | |
| 168 manager or using ``sudo`` to install Python packages. Instead, create | |
| 169 virtualenvs and isolated Python environments for all of your Python | |
| 170 projects. | |
| 171 | |
| 172 Python on $PATH is not appropriate | |
| 173 ---------------------------------- | |
| 174 | |
| 175 Tools like ``mach`` will look for Python by performing ``/usr/bin/env | |
| 176 python`` or equivalent. Please be sure the appropriate Python 2.7.3+ | |
| 177 path is on $PATH. On OS X, this likely means you'll need to modify your | |
| 178 shell's init script to put something ahead of ``/usr/bin``. |
