build/docs/python.rst@97036ab72558 (annotated)
build/docs/python.rst
Tue, 06 Jan 2015 21:39:09 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Tue, 06 Jan 2015 21:39:09 +0100
- branch
- TOR_BUG_9701
- changeset 8
- 97036ab72558
- permissions
- -rw-r--r--
Conditionally force memory storage according to privacy.thirdparty.isolate;
This solves Tor bug #9701, complying with disk avoidance documented in
https://www.torproject.org/projects/torbrowser/design/#disk-avoidance.
| michael@0 | 1 | .. _python: |
| michael@0 | 2 | |
| michael@0 | 3 | =========================== |
| michael@0 | 4 | Python and the Build System |
| michael@0 | 5 | =========================== |
| michael@0 | 6 | |
| michael@0 | 7 | The Python programming language is used significantly in the build |
| michael@0 | 8 | system. If we need to write code for the build system or for a tool |
| michael@0 | 9 | related to the build system, Python is typically the first choice. |
| michael@0 | 10 | |
| michael@0 | 11 | Python Requirements |
| michael@0 | 12 | =================== |
| michael@0 | 13 | |
| michael@0 | 14 | The tree requires Python 2.7.3 or greater but not Python 3 to build. |
| michael@0 | 15 | All Python packages not in the Python distribution are included in the |
| michael@0 | 16 | source tree. So all you should need is a vanilla Python install and you |
| michael@0 | 17 | should be good to go. |
| michael@0 | 18 | |
| michael@0 | 19 | Only CPython (the Python distribution available from www.python.org) is |
| michael@0 | 20 | supported. |
| michael@0 | 21 | |
| michael@0 | 22 | We require Python 2.7.3 (and not say 2.7.2) to build because Python |
| michael@0 | 23 | 2.7.3 contains numerous bug fixes, especially around the area of Unicode |
| michael@0 | 24 | handling. These bug fixes are extremely annoying and have to be worked |
| michael@0 | 25 | around. The build maintainers were tired of doing this, so the minimum |
| michael@0 | 26 | version requirement was upped (bug 870420). |
| michael@0 | 27 | |
| michael@0 | 28 | We intend to eventually support Python 3. This will come by way of dual |
| michael@0 | 29 | 2.7/3.x compatibility because a single flag day conversion to 3.x will |
| michael@0 | 30 | be too cumbersome given the amount of Python that would need converted. |
| michael@0 | 31 | We will not know which 3.x minor release we are targeting until this |
| michael@0 | 32 | effort is underway. This is tracked in bug 636155. |
| michael@0 | 33 | |
| michael@0 | 34 | Compiled Python Packages |
| michael@0 | 35 | ======================== |
| michael@0 | 36 | |
| michael@0 | 37 | There are some features of the build that rely on compiled Python packages |
| michael@0 | 38 | (packages containing C source). These features are currently all |
| michael@0 | 39 | optional because not every system contains the Python development |
| michael@0 | 40 | headers required to build these extensions. |
| michael@0 | 41 | |
| michael@0 | 42 | We recommend you have the Python development headers installed (``mach |
| michael@0 | 43 | bootstrap`` should do this for you) so you can take advantage of these |
| michael@0 | 44 | features. |
| michael@0 | 45 | |
| michael@0 | 46 | Issues with OS X System Python |
| michael@0 | 47 | ============================== |
| michael@0 | 48 | |
| michael@0 | 49 | The Python that ships with OS X has historically been littered with |
| michael@0 | 50 | subtle bugs and suboptimalities. Furthermore, OS X up through 10.8 don't |
| michael@0 | 51 | ship with Python 2.7.3 (10.8 ships with 2.7.2). |
| michael@0 | 52 | |
| michael@0 | 53 | OS X 10.8 and below users will be required to install a new Python |
| michael@0 | 54 | distribution. This may not be necessary for OS X 10.9+. However, we |
| michael@0 | 55 | still recommend installing a separate Python because of the history with |
| michael@0 | 56 | OS X's system Python issues. |
| michael@0 | 57 | |
| michael@0 | 58 | We recommend installing Python through Homebrew or MacPorts. If you run |
| michael@0 | 59 | ``mach bootstrap``, this should be done for you. |
| michael@0 | 60 | |
| michael@0 | 61 | Virtualenvs |
| michael@0 | 62 | =========== |
| michael@0 | 63 | |
| michael@0 | 64 | The build system relies heavily on |
| michael@0 | 65 | `virtualenvs <http://www.virtualenv.org/en/latest/>`_. Virtualenvs are |
| michael@0 | 66 | standalone and isolated Python environments. The problem a virtualenv |
| michael@0 | 67 | solves is that of dependencies across multiple Python components. If two |
| michael@0 | 68 | components on a system relied on different versions of a package, there |
| michael@0 | 69 | could be a conflict. Instead of managing multiple versions of a package |
| michael@0 | 70 | simultaneously, Python and virtualenvs take the route that it is easier |
| michael@0 | 71 | to just keep them separate so there is no potential for conflicts. |
| michael@0 | 72 | |
| michael@0 | 73 | Very early in the build process, a virtualenv is created inside the |
| michael@0 | 74 | :term:`object directory`. The virtualenv is configured such that it can |
| michael@0 | 75 | find all the Python packages in the source tree. The code for this lives |
| michael@0 | 76 | in :py:mod:`mozbuild.virtualenv`. |
| michael@0 | 77 | |
| michael@0 | 78 | Deficiencies |
| michael@0 | 79 | ------------ |
| michael@0 | 80 | |
| michael@0 | 81 | There are numerous deficiencies with the way virtualenvs are handled in |
| michael@0 | 82 | the build system. |
| michael@0 | 83 | |
| michael@0 | 84 | * mach reinvents the virtualenv. |
| michael@0 | 85 | |
| michael@0 | 86 | There is code in ``build/mach_bootstrap.py`` that configures ``sys.path`` |
| michael@0 | 87 | much the same way the virtualenv does. There are various bugs tracking |
| michael@0 | 88 | this. However, no clear solution has yet been devised. It's not a huge |
| michael@0 | 89 | problem and thus not a huge priority. |
| michael@0 | 90 | |
| michael@0 | 91 | * They aren't preserved across copies and packaging. |
| michael@0 | 92 | |
| michael@0 | 93 | If you attempt to copy an entire tree from one machine to another or |
| michael@0 | 94 | from one directory to another, chances are the virtualenv will fall |
| michael@0 | 95 | apart. It would be nice if we could preserve it somehow. Instead of |
| michael@0 | 96 | actually solving portable virtualenvs, all we really need to solve is |
| michael@0 | 97 | encapsulating the logic for populating the virtualenv along with all |
| michael@0 | 98 | dependent files in the appropriate place. |
| michael@0 | 99 | |
| michael@0 | 100 | * .pyc files written to source directory. |
| michael@0 | 101 | |
| michael@0 | 102 | We rely heavily on ``.pth`` files in our virtualenv. A ``.pth`` file |
| michael@0 | 103 | is a special file that contains a list of paths. Python will take the |
| michael@0 | 104 | set of listed paths encountered in ``.pth`` files and add them to |
| michael@0 | 105 | ``sys.path``. |
| michael@0 | 106 | |
| michael@0 | 107 | When Python compiles a ``.py`` file to bytecode, it writes out a |
| michael@0 | 108 | ``.pyc`` file so it doesn't have to perform this compilation again. |
| michael@0 | 109 | It puts these ``.pyc`` files alongside the ``.pyc`` file. Python |
| michael@0 | 110 | provides very little control for determing where these ``.pyc`` files |
| michael@0 | 111 | go, even in Python 3 (which offers customer importers). |
| michael@0 | 112 | |
| michael@0 | 113 | With ``.pth`` files pointing back to directories in the source tree |
| michael@0 | 114 | and not the object directory, ``.pyc`` files are created in the source |
| michael@0 | 115 | tree. This is bad because when Python imports a module, it first looks |
| michael@0 | 116 | for a ``.pyc`` file before the ``.py`` file. If there is a ``.pyc`` |
| michael@0 | 117 | file but no ``.py`` file, it will happily import the module. This |
| michael@0 | 118 | wreaks havoc during file moves, refactoring, etc. |
| michael@0 | 119 | |
| michael@0 | 120 | There are various proposals for fixing this. See bug 795995. |
| michael@0 | 121 | |
| michael@0 | 122 | Installing Python Manually |
| michael@0 | 123 | ========================== |
| michael@0 | 124 | |
| michael@0 | 125 | We highly recommend you use your system's package manager or a |
| michael@0 | 126 | well-supported 3rd party package manager to install Python for you. If |
| michael@0 | 127 | these are not available to you, we recommend the following tools for |
| michael@0 | 128 | installing Python: |
| michael@0 | 129 | |
| michael@0 | 130 | * `buildout.python <https://github.com/collective/buildout.python>`_ |
| michael@0 | 131 | * `pyenv <https://github.com/yyuu/pyenv>`_ |
| michael@0 | 132 | * An official installer from http://www.python.org. |
| michael@0 | 133 | |
| michael@0 | 134 | If all else fails, consider compiling Python from source manually. But this |
| michael@0 | 135 | should be viewed as the least desirable option. |
| michael@0 | 136 | |
| michael@0 | 137 | Common Issues with Python |
| michael@0 | 138 | ========================= |
| michael@0 | 139 | |
| michael@0 | 140 | Upgrading your Python distribution breaks the virtualenv |
| michael@0 | 141 | -------------------------------------------------------- |
| michael@0 | 142 | |
| michael@0 | 143 | If you upgrade the Python distribution (e.g. install Python 2.7.5 |
| michael@0 | 144 | from 2.7.3, chances are parts of the virtualenv will break. |
| michael@0 | 145 | This commonly manifests as a cryptic ``Cannot import XXX`` exception. |
| michael@0 | 146 | More often than not, the module being imported contains binary/compiled |
| michael@0 | 147 | components. |
| michael@0 | 148 | |
| michael@0 | 149 | If you upgrade or reinstall your Python distribution, we recommend |
| michael@0 | 150 | clobbering your build. |
| michael@0 | 151 | |
| michael@0 | 152 | Packages installed at the system level conflict with build system's |
| michael@0 | 153 | ------------------------------------------------------------------- |
| michael@0 | 154 | |
| michael@0 | 155 | It is common for people to install Python packages using ``sudo`` (e.g. |
| michael@0 | 156 | ``sudo pip install psutil``) or with the system's package manager |
| michael@0 | 157 | (e.g. ``apt-get install python-mysql``. |
| michael@0 | 158 | |
| michael@0 | 159 | A problem with this is that packages installed at the system level may |
| michael@0 | 160 | conflict with the package provided by the source tree. As of bug 907902 |
| michael@0 | 161 | and changeset f18eae7c3b27 (September 16, 2013), this should no longer |
| michael@0 | 162 | be an issue since the virtualenv created as part of the build doesn't |
| michael@0 | 163 | add the system's ``site-packages`` directory to ``sys.path``. However, |
| michael@0 | 164 | poorly installed packages may still find a way to creep into the mix and |
| michael@0 | 165 | interfere with our virtualenv. |
| michael@0 | 166 | |
| michael@0 | 167 | As a general principle, we recommend against using your system's package |
| michael@0 | 168 | manager or using ``sudo`` to install Python packages. Instead, create |
| michael@0 | 169 | virtualenvs and isolated Python environments for all of your Python |
| michael@0 | 170 | projects. |
| michael@0 | 171 | |
| michael@0 | 172 | Python on $PATH is not appropriate |
| michael@0 | 173 | ---------------------------------- |
| michael@0 | 174 | |
| michael@0 | 175 | Tools like ``mach`` will look for Python by performing ``/usr/bin/env |
| michael@0 | 176 | python`` or equivalent. Please be sure the appropriate Python 2.7.3+ |
| michael@0 | 177 | path is on $PATH. On OS X, this likely means you'll need to modify your |
| michael@0 | 178 | shell's init script to put something ahead of ``/usr/bin``. |
