build/docs/slow.rst@6474c204b198 (annotated)
build/docs/slow.rst
Wed, 31 Dec 2014 06:09:35 +0100
- author
- Michael Schloh von Bennewitz <michael@schloh.com>
- date
- Wed, 31 Dec 2014 06:09:35 +0100
- changeset 0
- 6474c204b198
- permissions
- -rw-r--r--
Cloned upstream origin tor-browser at tor-browser-31.3.0esr-4.5-1-build1
revision ID fc1c9ff7c1b2defdbc039f12214767608f46423f for hacking purpose.
| michael@0 | 1 | .. _slow: |
| michael@0 | 2 | |
| michael@0 | 3 | ============================ |
| michael@0 | 4 | Why the Build System is Slow |
| michael@0 | 5 | ============================ |
| michael@0 | 6 | |
| michael@0 | 7 | A common complaint about the build system is that it's slow. There are |
| michael@0 | 8 | many reasons contributing to its slowness. We will attempt to document |
| michael@0 | 9 | them here. |
| michael@0 | 10 | |
| michael@0 | 11 | First, it is important to distinguish between a :term:`clobber build` |
| michael@0 | 12 | and an :term:`incremental build`. The reasons for why each are slow can |
| michael@0 | 13 | be different. |
| michael@0 | 14 | |
| michael@0 | 15 | The build does a lot of work |
| michael@0 | 16 | ============================ |
| michael@0 | 17 | |
| michael@0 | 18 | It may not be obvious, but the main reason the build system is slow is |
| michael@0 | 19 | because it does a lot of work! The source tree consists of a few |
| michael@0 | 20 | thousand C++ files. On a modern machine, we spend over 120 minutes of CPU |
| michael@0 | 21 | core time compiling files! So, if you are looking for the root cause of |
| michael@0 | 22 | slow clobber builds, look at the sheer volume of C++ files in the tree. |
| michael@0 | 23 | |
| michael@0 | 24 | You don't have enough CPU cores and MHz |
| michael@0 | 25 | ======================================= |
| michael@0 | 26 | |
| michael@0 | 27 | The build should be CPU bound. If the build system maintainers are |
| michael@0 | 28 | optimizing the build system perfectly, every CPU core in your machine |
| michael@0 | 29 | should be 100% saturated during a build. While this isn't currently the |
| michael@0 | 30 | case (keep reading below), generally speaking, the more CPU cores you |
| michael@0 | 31 | have in your machine and the more total MHz in your machine, the better. |
| michael@0 | 32 | |
| michael@0 | 33 | **We highly recommend building with no fewer than 4 physical CPU |
| michael@0 | 34 | cores.** Please note the *physical* in this sentence. Hyperthreaded |
| michael@0 | 35 | cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical |
| michael@0 | 36 | for example) only yield at most a 1.25x speedup per core. |
| michael@0 | 37 | |
| michael@0 | 38 | We also recommend using the most modern CPU model possible. Haswell |
| michael@0 | 39 | chips deliver much more performance per CPU cycle than say Sandy Bridge |
| michael@0 | 40 | CPUs. |
| michael@0 | 41 | |
| michael@0 | 42 | This cause impacts both clobber and incremental builds. |
| michael@0 | 43 | |
| michael@0 | 44 | You are building with a slow I/O layer |
| michael@0 | 45 | ====================================== |
| michael@0 | 46 | |
| michael@0 | 47 | The build system can be I/O bound if your I/O layer is slow. Linking |
| michael@0 | 48 | libxul on some platforms and build architectures can perform gigabytes |
| michael@0 | 49 | of I/O. |
| michael@0 | 50 | |
| michael@0 | 51 | To minimize the impact of slow I/O on build performance, **we highly |
| michael@0 | 52 | recommend building with an SSD.** Power users with enough memory may opt |
| michael@0 | 53 | to build from a RAM disk. Mechanical disks should be avoided if at all |
| michael@0 | 54 | possible. |
| michael@0 | 55 | |
| michael@0 | 56 | Some may dispute the importance of an SSD on build times. It is true |
| michael@0 | 57 | that the beneficial impact of an SSD can be mitigated if your system has |
| michael@0 | 58 | lots of memory and the build files stay in the page cache. However, |
| michael@0 | 59 | operating system memory management is complicated. You don't really have |
| michael@0 | 60 | control over what or when something is evicted from the page cache. |
| michael@0 | 61 | Therefore, unless your machine is a dedicated build machine or you have |
| michael@0 | 62 | more memory than is needed by everything running on your machine, |
| michael@0 | 63 | chances are you'll run into page cache eviction and you I/O layer will |
| michael@0 | 64 | impact build performance. That being said, an SSD certainly doesn't |
| michael@0 | 65 | hurt build times. And, anyone who has used a machine with an SSD will |
| michael@0 | 66 | tell you how great of an investment it is for performance all around the |
| michael@0 | 67 | operating system. On top of that, some automated tests are I/O bound |
| michael@0 | 68 | (like those touching SQLite databases), so an SSD will make tests |
| michael@0 | 69 | faster. |
| michael@0 | 70 | |
| michael@0 | 71 | This cause impacts both clobber and incremental builds. |
| michael@0 | 72 | |
| michael@0 | 73 | You don't have enough memory |
| michael@0 | 74 | ============================ |
| michael@0 | 75 | |
| michael@0 | 76 | The build system allocates a lot of memory, especially when building |
| michael@0 | 77 | many things in parallel. If you don't have enough free system memory, |
| michael@0 | 78 | the build will cause swap activity, slowing down your system and the |
| michael@0 | 79 | build. Even if you never get to the point of swapping, the build system |
| michael@0 | 80 | performs a lot of I/O and having all accessed files in memory and the |
| michael@0 | 81 | page cache can significantly reduce the influence of the I/O layer on |
| michael@0 | 82 | the build system. |
| michael@0 | 83 | |
| michael@0 | 84 | **We recommend building with no less than 8 GB of system memory.** As |
| michael@0 | 85 | always, the more memory you have, the better. For a bare bones machine |
| michael@0 | 86 | doing nothing more than building the source tree, anything more than 16 |
| michael@0 | 87 | GB is likely entering the point of diminishing returns. |
| michael@0 | 88 | |
| michael@0 | 89 | This cause impacts both clobber and incremental builds. |
| michael@0 | 90 | |
| michael@0 | 91 | You are building with pymake |
| michael@0 | 92 | ============================ |
| michael@0 | 93 | |
| michael@0 | 94 | Pymake is slower than GNU make. One reason is Python is generally slower |
| michael@0 | 95 | than C. The build system maintainers are consistently looking at |
| michael@0 | 96 | optimizing pymake. However, it is death by a thousand cuts. |
| michael@0 | 97 | |
| michael@0 | 98 | This cause impacts both clobber and incremental builds. |
| michael@0 | 99 | |
| michael@0 | 100 | You are building on Windows |
| michael@0 | 101 | =========================== |
| michael@0 | 102 | |
| michael@0 | 103 | Builds on Windows are slow for a few reasons. First, Windows builds use |
| michael@0 | 104 | pymake, not GNU make (because of compatibility issues with GNU make). |
| michael@0 | 105 | But, there are other sources of slowness. |
| michael@0 | 106 | |
| michael@0 | 107 | New processes on Windows are about a magnitude slower to spawn than on |
| michael@0 | 108 | UNIX-y systems such as Linux. This is because Windows has optimized new |
| michael@0 | 109 | threads while the \*NIX platforms typically optimize new processes. |
| michael@0 | 110 | Anyway, the build system spawns thousands of new processes during a |
| michael@0 | 111 | build. Parts of the build that rely on rapid spawning of new processes |
| michael@0 | 112 | are slow on Windows as a result. This is most pronounced when running |
| michael@0 | 113 | *configure*. The configure file is a giant shell script and shell |
| michael@0 | 114 | scripts rely heavily on new processes. This is why configure on Windows |
| michael@0 | 115 | can run over a minute slower on Windows. |
| michael@0 | 116 | |
| michael@0 | 117 | Another reason Windows builds are slower is because Windows lacks proper |
| michael@0 | 118 | symlink support. On systems that support symlinks, we can generate a |
| michael@0 | 119 | file into a staging area then symlink it into the final directory very |
| michael@0 | 120 | quickly. On Windows, we have to perform a full file copy. This incurs |
| michael@0 | 121 | much more I/O. And if done poorly, can muck with file modification |
| michael@0 | 122 | times, messing up build dependencies. As of the summer of 2013, the |
| michael@0 | 123 | impact of symlinks is being mitigated through the use |
| michael@0 | 124 | of an :term:`install manifest`. |
| michael@0 | 125 | |
| michael@0 | 126 | These issues impact both clobber and incremental builds. |
| michael@0 | 127 | |
| michael@0 | 128 | Recursive make traversal is slow |
| michael@0 | 129 | ================================ |
| michael@0 | 130 | |
| michael@0 | 131 | The build system has traditionally been built by employing recursive |
| michael@0 | 132 | make. Recursive make involves make iterating through directories / make |
| michael@0 | 133 | files sequentially and executing each in turn. This is inefficient for |
| michael@0 | 134 | directories containing few targets/tasks because make could be *starved* |
| michael@0 | 135 | for work when processing these directories. Any time make is starved, |
| michael@0 | 136 | the build isn't using all available CPU cycles and the build is slower |
| michael@0 | 137 | as a result. |
| michael@0 | 138 | |
| michael@0 | 139 | Work has started in bug 907365 to fix this issue by changing the way |
| michael@0 | 140 | make traverses all the make files. |
| michael@0 | 141 | |
| michael@0 | 142 | The impact of slow recursive make traversal is mostly felt on |
| michael@0 | 143 | incremental builds. Traditionally, most of the wall time during a |
| michael@0 | 144 | no-op build is spent in make traversal. |
| michael@0 | 145 | |
| michael@0 | 146 | make is inefficient |
| michael@0 | 147 | =================== |
| michael@0 | 148 | |
| michael@0 | 149 | Compared to modern build backends like Tup or Ninja, make is slow and |
| michael@0 | 150 | inefficient. We can only make make so fast. At some point, we'll hit a |
| michael@0 | 151 | performance plateau and will need to use a different tool to make builds |
| michael@0 | 152 | faster. |
| michael@0 | 153 | |
| michael@0 | 154 | Please note that clobber and incremental builds are different. A clobber |
| michael@0 | 155 | build with make will likely be as fast as a clobber build with e.g. Tup. |
| michael@0 | 156 | However, Tup should vastly outperform make when it comes to incremental |
| michael@0 | 157 | builds. Therefore, this issue is mostly seen when performing incremental |
| michael@0 | 158 | builds. |
| michael@0 | 159 | |
| michael@0 | 160 | C++ header dependency hell |
| michael@0 | 161 | ========================== |
| michael@0 | 162 | |
| michael@0 | 163 | Modifying a *.h* file can have significant impact on the build system. |
| michael@0 | 164 | If you modify a *.h* that is used by 1000 C++ files, all of those 1000 |
| michael@0 | 165 | C++ files will be recompiled. |
| michael@0 | 166 | |
| michael@0 | 167 | Our code base has traditionally been sloppy managing the impact of |
| michael@0 | 168 | changed headers on build performance. Bug 785103 tracks improving the |
| michael@0 | 169 | situation. |
| michael@0 | 170 | |
| michael@0 | 171 | This issue mostly impacts the times of an :term:`incremental build`. |
| michael@0 | 172 | |
| michael@0 | 173 | A search/indexing service on your machine is running |
| michael@0 | 174 | ==================================================== |
| michael@0 | 175 | |
| michael@0 | 176 | Many operating systems have a background service that automatically |
| michael@0 | 177 | indexes filesystem content to make searching faster. On Windows, you |
| michael@0 | 178 | have the Windows Search Service. On OS X, you have Finder. |
| michael@0 | 179 | |
| michael@0 | 180 | These background services sometimes take a keen interest in the files |
| michael@0 | 181 | being produced as part of the build. Since the build system produces |
| michael@0 | 182 | hundreds of megabytes or even a few gigabytes of file data, you can |
| michael@0 | 183 | imagine how much work this is to index! If this work is being performed |
| michael@0 | 184 | while the build is running, your build will be slower. |
| michael@0 | 185 | |
| michael@0 | 186 | OS X's Finder is notorious for indexing when the build is running. And, |
| michael@0 | 187 | it has a tendency to suck up a whole CPU core. This can make builds |
| michael@0 | 188 | several minutes slower. If you build with ``mach`` and have the optional |
| michael@0 | 189 | ``psutil`` package built (it requires Python development headers - see |
| michael@0 | 190 | :ref:`python` for more) and Finder is running during a build, mach will |
| michael@0 | 191 | print a warning at the end of the build, complete with instructions on |
| michael@0 | 192 | how to fix it. |
