The Tor Browser / file revision

 .. _slow:

 ============================

 Why the Build System is Slow

 ============================

 A common complaint about the build system is that it's slow. There are

 many reasons contributing to its slowness. We will attempt to document

 them here.

 First, it is important to distinguish between a :term:`clobber build`

 and an :term:`incremental build`. The reasons for why each are slow can

 be different.

 The build does a lot of work

 ============================

 It may not be obvious, but the main reason the build system is slow is

 because it does a lot of work! The source tree consists of a few

 thousand C++ files. On a modern machine, we spend over 120 minutes of CPU

 core time compiling files! So, if you are looking for the root cause of

 slow clobber builds, look at the sheer volume of C++ files in the tree.

 You don't have enough CPU cores and MHz

 =======================================

 The build should be CPU bound. If the build system maintainers are

 optimizing the build system perfectly, every CPU core in your machine

 should be 100% saturated during a build. While this isn't currently the

 case (keep reading below), generally speaking, the more CPU cores you

 have in your machine and the more total MHz in your machine, the better.

 **We highly recommend building with no fewer than 4 physical CPU

 cores.** Please note the *physical* in this sentence. Hyperthreaded

 cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical

 for example) only yield at most a 1.25x speedup per core.

 We also recommend using the most modern CPU model possible. Haswell

 chips deliver much more performance per CPU cycle than say Sandy Bridge

 CPUs.

 This cause impacts both clobber and incremental builds.

 You are building with a slow I/O layer

 ======================================

 The build system can be I/O bound if your I/O layer is slow. Linking

 libxul on some platforms and build architectures can perform gigabytes

 of I/O.

 To minimize the impact of slow I/O on build performance, **we highly

 recommend building with an SSD.** Power users with enough memory may opt

 to build from a RAM disk. Mechanical disks should be avoided if at all

 possible.

 Some may dispute the importance of an SSD on build times. It is true

 that the beneficial impact of an SSD can be mitigated if your system has

 lots of memory and the build files stay in the page cache. However,

 operating system memory management is complicated. You don't really have

 control over what or when something is evicted from the page cache.

 Therefore, unless your machine is a dedicated build machine or you have

 more memory than is needed by everything running on your machine,

 chances are you'll run into page cache eviction and you I/O layer will

 impact build performance. That being said, an SSD certainly doesn't

 hurt build times. And, anyone who has used a machine with an SSD will

 tell you how great of an investment it is for performance all around the

 operating system. On top of that, some automated tests are I/O bound

 (like those touching SQLite databases), so an SSD will make tests

 faster.

 This cause impacts both clobber and incremental builds.

 You don't have enough memory

 ============================

 The build system allocates a lot of memory, especially when building

 many things in parallel. If you don't have enough free system memory,

 the build will cause swap activity, slowing down your system and the

 build. Even if you never get to the point of swapping, the build system

 performs a lot of I/O and having all accessed files in memory and the

 page cache can significantly reduce the influence of the I/O layer on

 the build system.

 **We recommend building with no less than 8 GB of system memory.** As

 always, the more memory you have, the better. For a bare bones machine

 doing nothing more than building the source tree, anything more than 16

 GB is likely entering the point of diminishing returns.

 This cause impacts both clobber and incremental builds.

 You are building with pymake

 ============================

 Pymake is slower than GNU make. One reason is Python is generally slower

 than C. The build system maintainers are consistently looking at

 optimizing pymake. However, it is death by a thousand cuts.

 This cause impacts both clobber and incremental builds.

 You are building on Windows

 ===========================

 Builds on Windows are slow for a few reasons. First, Windows builds use

 pymake, not GNU make (because of compatibility issues with GNU make).

 But, there are other sources of slowness.

 New processes on Windows are about a magnitude slower to spawn than on

 UNIX-y systems such as Linux. This is because Windows has optimized new

 threads while the \*NIX platforms typically optimize new processes.

 Anyway, the build system spawns thousands of new processes during a

 build. Parts of the build that rely on rapid spawning of new processes

 are slow on Windows as a result. This is most pronounced when running

 *configure*. The configure file is a giant shell script and shell

 scripts rely heavily on new processes. This is why configure on Windows

 can run over a minute slower on Windows.

 Another reason Windows builds are slower is because Windows lacks proper

 symlink support. On systems that support symlinks, we can generate a

 file into a staging area then symlink it into the final directory very

 quickly. On Windows, we have to perform a full file copy. This incurs

 much more I/O. And if done poorly, can muck with file modification

 times, messing up build dependencies. As of the summer of 2013, the

 impact of symlinks is being mitigated through the use

 of an :term:`install manifest`.

 These issues impact both clobber and incremental builds.

 Recursive make traversal is slow

 ================================

 The build system has traditionally been built by employing recursive

 make. Recursive make involves make iterating through directories / make

 files sequentially and executing each in turn. This is inefficient for

 directories containing few targets/tasks because make could be *starved*

 for work when processing these directories. Any time make is starved,

 the build isn't using all available CPU cycles and the build is slower

 as a result.

 Work has started in bug 907365 to fix this issue by changing the way

 make traverses all the make files.

 The impact of slow recursive make traversal is mostly felt on

 incremental builds. Traditionally, most of the wall time during a

 no-op build is spent in make traversal.

 make is inefficient

 ===================

 Compared to modern build backends like Tup or Ninja, make is slow and

 inefficient. We can only make make so fast. At some point, we'll hit a

 performance plateau and will need to use a different tool to make builds

 faster.

 Please note that clobber and incremental builds are different. A clobber

 build with make will likely be as fast as a clobber build with e.g. Tup.

 However, Tup should vastly outperform make when it comes to incremental

 builds. Therefore, this issue is mostly seen when performing incremental

 builds.

 C++ header dependency hell

 ==========================

 Modifying a *.h* file can have significant impact on the build system.

 If you modify a *.h* that is used by 1000 C++ files, all of those 1000

 C++ files will be recompiled.

 Our code base has traditionally been sloppy managing the impact of

 changed headers on build performance. Bug 785103 tracks improving the

 situation.

 This issue mostly impacts the times of an :term:`incremental build`.

 A search/indexing service on your machine is running

 ====================================================

 Many operating systems have a background service that automatically

 indexes filesystem content to make searching faster. On Windows, you

 have the Windows Search Service. On OS X, you have Finder.

 These background services sometimes take a keen interest in the files

 being produced as part of the build. Since the build system produces

 hundreds of megabytes or even a few gigabytes of file data, you can

 imagine how much work this is to index! If this work is being performed

 while the build is running, your build will be slower.

 OS X's Finder is notorious for indexing when the build is running. And,

 it has a tendency to suck up a whole CPU core. This can make builds

 several minutes slower. If you build with ``mach`` and have the optional

 ``psutil`` package built (it requires Python development headers - see

 :ref:`python` for more) and Finder is running during a build, mach will

 print a warning at the end of the build, complete with instructions on

 how to fix it.