The Tor Browser / file comparison

comparison: build/docs/slow.rst

build/docs/slow.rst

changeset 0
6474c204b198
equal deleted inserted replaced
-1:000000000000 0:e4ddd34c6ca1
1 .. _slow:
2
3 ============================
4 Why the Build System is Slow
5 ============================
6
7 A common complaint about the build system is that it's slow. There are
8 many reasons contributing to its slowness. We will attempt to document
9 them here.
10
11 First, it is important to distinguish between a :term:`clobber build`
12 and an :term:`incremental build`. The reasons for why each are slow can
13 be different.
14
15 The build does a lot of work
16 ============================
17
18 It may not be obvious, but the main reason the build system is slow is
19 because it does a lot of work! The source tree consists of a few
20 thousand C++ files. On a modern machine, we spend over 120 minutes of CPU
21 core time compiling files! So, if you are looking for the root cause of
22 slow clobber builds, look at the sheer volume of C++ files in the tree.
23
24 You don't have enough CPU cores and MHz
25 =======================================
26
27 The build should be CPU bound. If the build system maintainers are
28 optimizing the build system perfectly, every CPU core in your machine
29 should be 100% saturated during a build. While this isn't currently the
30 case (keep reading below), generally speaking, the more CPU cores you
31 have in your machine and the more total MHz in your machine, the better.
32
33 **We highly recommend building with no fewer than 4 physical CPU
34 cores.** Please note the *physical* in this sentence. Hyperthreaded
35 cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical
36 for example) only yield at most a 1.25x speedup per core.
37
38 We also recommend using the most modern CPU model possible. Haswell
39 chips deliver much more performance per CPU cycle than say Sandy Bridge
40 CPUs.
41
42 This cause impacts both clobber and incremental builds.
43
44 You are building with a slow I/O layer
45 ======================================
46
47 The build system can be I/O bound if your I/O layer is slow. Linking
48 libxul on some platforms and build architectures can perform gigabytes
49 of I/O.
50
51 To minimize the impact of slow I/O on build performance, **we highly
52 recommend building with an SSD.** Power users with enough memory may opt
53 to build from a RAM disk. Mechanical disks should be avoided if at all
54 possible.
55
56 Some may dispute the importance of an SSD on build times. It is true
57 that the beneficial impact of an SSD can be mitigated if your system has
58 lots of memory and the build files stay in the page cache. However,
59 operating system memory management is complicated. You don't really have
60 control over what or when something is evicted from the page cache.
61 Therefore, unless your machine is a dedicated build machine or you have
62 more memory than is needed by everything running on your machine,
63 chances are you'll run into page cache eviction and you I/O layer will
64 impact build performance. That being said, an SSD certainly doesn't
65 hurt build times. And, anyone who has used a machine with an SSD will
66 tell you how great of an investment it is for performance all around the
67 operating system. On top of that, some automated tests are I/O bound
68 (like those touching SQLite databases), so an SSD will make tests
69 faster.
70
71 This cause impacts both clobber and incremental builds.
72
73 You don't have enough memory
74 ============================
75
76 The build system allocates a lot of memory, especially when building
77 many things in parallel. If you don't have enough free system memory,
78 the build will cause swap activity, slowing down your system and the
79 build. Even if you never get to the point of swapping, the build system
80 performs a lot of I/O and having all accessed files in memory and the
81 page cache can significantly reduce the influence of the I/O layer on
82 the build system.
83
84 **We recommend building with no less than 8 GB of system memory.** As
85 always, the more memory you have, the better. For a bare bones machine
86 doing nothing more than building the source tree, anything more than 16
87 GB is likely entering the point of diminishing returns.
88
89 This cause impacts both clobber and incremental builds.
90
91 You are building with pymake
92 ============================
93
94 Pymake is slower than GNU make. One reason is Python is generally slower
95 than C. The build system maintainers are consistently looking at
96 optimizing pymake. However, it is death by a thousand cuts.
97
98 This cause impacts both clobber and incremental builds.
99
100 You are building on Windows
101 ===========================
102
103 Builds on Windows are slow for a few reasons. First, Windows builds use
104 pymake, not GNU make (because of compatibility issues with GNU make).
105 But, there are other sources of slowness.
106
107 New processes on Windows are about a magnitude slower to spawn than on
108 UNIX-y systems such as Linux. This is because Windows has optimized new
109 threads while the \*NIX platforms typically optimize new processes.
110 Anyway, the build system spawns thousands of new processes during a
111 build. Parts of the build that rely on rapid spawning of new processes
112 are slow on Windows as a result. This is most pronounced when running
113 *configure*. The configure file is a giant shell script and shell
114 scripts rely heavily on new processes. This is why configure on Windows
115 can run over a minute slower on Windows.
116
117 Another reason Windows builds are slower is because Windows lacks proper
118 symlink support. On systems that support symlinks, we can generate a
119 file into a staging area then symlink it into the final directory very
120 quickly. On Windows, we have to perform a full file copy. This incurs
121 much more I/O. And if done poorly, can muck with file modification
122 times, messing up build dependencies. As of the summer of 2013, the
123 impact of symlinks is being mitigated through the use
124 of an :term:`install manifest`.
125
126 These issues impact both clobber and incremental builds.
127
128 Recursive make traversal is slow
129 ================================
130
131 The build system has traditionally been built by employing recursive
132 make. Recursive make involves make iterating through directories / make
133 files sequentially and executing each in turn. This is inefficient for
134 directories containing few targets/tasks because make could be *starved*
135 for work when processing these directories. Any time make is starved,
136 the build isn't using all available CPU cycles and the build is slower
137 as a result.
138
139 Work has started in bug 907365 to fix this issue by changing the way
140 make traverses all the make files.
141
142 The impact of slow recursive make traversal is mostly felt on
143 incremental builds. Traditionally, most of the wall time during a
144 no-op build is spent in make traversal.
145
146 make is inefficient
147 ===================
148
149 Compared to modern build backends like Tup or Ninja, make is slow and
150 inefficient. We can only make make so fast. At some point, we'll hit a
151 performance plateau and will need to use a different tool to make builds
152 faster.
153
154 Please note that clobber and incremental builds are different. A clobber
155 build with make will likely be as fast as a clobber build with e.g. Tup.
156 However, Tup should vastly outperform make when it comes to incremental
157 builds. Therefore, this issue is mostly seen when performing incremental
158 builds.
159
160 C++ header dependency hell
161 ==========================
162
163 Modifying a *.h* file can have significant impact on the build system.
164 If you modify a *.h* that is used by 1000 C++ files, all of those 1000
165 C++ files will be recompiled.
166
167 Our code base has traditionally been sloppy managing the impact of
168 changed headers on build performance. Bug 785103 tracks improving the
169 situation.
170
171 This issue mostly impacts the times of an :term:`incremental build`.
172
173 A search/indexing service on your machine is running
174 ====================================================
175
176 Many operating systems have a background service that automatically
177 indexes filesystem content to make searching faster. On Windows, you
178 have the Windows Search Service. On OS X, you have Finder.
179
180 These background services sometimes take a keen interest in the files
181 being produced as part of the build. Since the build system produces
182 hundreds of megabytes or even a few gigabytes of file data, you can
183 imagine how much work this is to index! If this work is being performed
184 while the build is running, your build will be slower.
185
186 OS X's Finder is notorious for indexing when the build is running. And,
187 it has a tendency to suck up a whole CPU core. This can make builds
188 several minutes slower. If you build with ``mach`` and have the optional
189 ``psutil`` package built (it requires Python development headers - see
190 :ref:`python` for more) and Finder is running during a build, mach will
191 print a warning at the end of the build, complete with instructions on
192 how to fix it.

Mercurial Repository: The Tor Browser

Europalab SCM Repository Server

mercurial