1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/build/docs/slow.rst Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,192 @@
1.4 +.. _slow:
1.5 +
1.6 +============================
1.7 +Why the Build System is Slow
1.8 +============================
1.9 +
1.10 +A common complaint about the build system is that it's slow. There are
1.11 +many reasons contributing to its slowness. We will attempt to document
1.12 +them here.
1.13 +
1.14 +First, it is important to distinguish between a :term:`clobber build`
1.15 +and an :term:`incremental build`. The reasons for why each are slow can
1.16 +be different.
1.17 +
1.18 +The build does a lot of work
1.19 +============================
1.20 +
1.21 +It may not be obvious, but the main reason the build system is slow is
1.22 +because it does a lot of work! The source tree consists of a few
1.23 +thousand C++ files. On a modern machine, we spend over 120 minutes of CPU
1.24 +core time compiling files! So, if you are looking for the root cause of
1.25 +slow clobber builds, look at the sheer volume of C++ files in the tree.
1.26 +
1.27 +You don't have enough CPU cores and MHz
1.28 +=======================================
1.29 +
1.30 +The build should be CPU bound. If the build system maintainers are
1.31 +optimizing the build system perfectly, every CPU core in your machine
1.32 +should be 100% saturated during a build. While this isn't currently the
1.33 +case (keep reading below), generally speaking, the more CPU cores you
1.34 +have in your machine and the more total MHz in your machine, the better.
1.35 +
1.36 +**We highly recommend building with no fewer than 4 physical CPU
1.37 +cores.** Please note the *physical* in this sentence. Hyperthreaded
1.38 +cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical
1.39 +for example) only yield at most a 1.25x speedup per core.
1.40 +
1.41 +We also recommend using the most modern CPU model possible. Haswell
1.42 +chips deliver much more performance per CPU cycle than say Sandy Bridge
1.43 +CPUs.
1.44 +
1.45 +This cause impacts both clobber and incremental builds.
1.46 +
1.47 +You are building with a slow I/O layer
1.48 +======================================
1.49 +
1.50 +The build system can be I/O bound if your I/O layer is slow. Linking
1.51 +libxul on some platforms and build architectures can perform gigabytes
1.52 +of I/O.
1.53 +
1.54 +To minimize the impact of slow I/O on build performance, **we highly
1.55 +recommend building with an SSD.** Power users with enough memory may opt
1.56 +to build from a RAM disk. Mechanical disks should be avoided if at all
1.57 +possible.
1.58 +
1.59 +Some may dispute the importance of an SSD on build times. It is true
1.60 +that the beneficial impact of an SSD can be mitigated if your system has
1.61 +lots of memory and the build files stay in the page cache. However,
1.62 +operating system memory management is complicated. You don't really have
1.63 +control over what or when something is evicted from the page cache.
1.64 +Therefore, unless your machine is a dedicated build machine or you have
1.65 +more memory than is needed by everything running on your machine,
1.66 +chances are you'll run into page cache eviction and you I/O layer will
1.67 +impact build performance. That being said, an SSD certainly doesn't
1.68 +hurt build times. And, anyone who has used a machine with an SSD will
1.69 +tell you how great of an investment it is for performance all around the
1.70 +operating system. On top of that, some automated tests are I/O bound
1.71 +(like those touching SQLite databases), so an SSD will make tests
1.72 +faster.
1.73 +
1.74 +This cause impacts both clobber and incremental builds.
1.75 +
1.76 +You don't have enough memory
1.77 +============================
1.78 +
1.79 +The build system allocates a lot of memory, especially when building
1.80 +many things in parallel. If you don't have enough free system memory,
1.81 +the build will cause swap activity, slowing down your system and the
1.82 +build. Even if you never get to the point of swapping, the build system
1.83 +performs a lot of I/O and having all accessed files in memory and the
1.84 +page cache can significantly reduce the influence of the I/O layer on
1.85 +the build system.
1.86 +
1.87 +**We recommend building with no less than 8 GB of system memory.** As
1.88 +always, the more memory you have, the better. For a bare bones machine
1.89 +doing nothing more than building the source tree, anything more than 16
1.90 +GB is likely entering the point of diminishing returns.
1.91 +
1.92 +This cause impacts both clobber and incremental builds.
1.93 +
1.94 +You are building with pymake
1.95 +============================
1.96 +
1.97 +Pymake is slower than GNU make. One reason is Python is generally slower
1.98 +than C. The build system maintainers are consistently looking at
1.99 +optimizing pymake. However, it is death by a thousand cuts.
1.100 +
1.101 +This cause impacts both clobber and incremental builds.
1.102 +
1.103 +You are building on Windows
1.104 +===========================
1.105 +
1.106 +Builds on Windows are slow for a few reasons. First, Windows builds use
1.107 +pymake, not GNU make (because of compatibility issues with GNU make).
1.108 +But, there are other sources of slowness.
1.109 +
1.110 +New processes on Windows are about a magnitude slower to spawn than on
1.111 +UNIX-y systems such as Linux. This is because Windows has optimized new
1.112 +threads while the \*NIX platforms typically optimize new processes.
1.113 +Anyway, the build system spawns thousands of new processes during a
1.114 +build. Parts of the build that rely on rapid spawning of new processes
1.115 +are slow on Windows as a result. This is most pronounced when running
1.116 +*configure*. The configure file is a giant shell script and shell
1.117 +scripts rely heavily on new processes. This is why configure on Windows
1.118 +can run over a minute slower on Windows.
1.119 +
1.120 +Another reason Windows builds are slower is because Windows lacks proper
1.121 +symlink support. On systems that support symlinks, we can generate a
1.122 +file into a staging area then symlink it into the final directory very
1.123 +quickly. On Windows, we have to perform a full file copy. This incurs
1.124 +much more I/O. And if done poorly, can muck with file modification
1.125 +times, messing up build dependencies. As of the summer of 2013, the
1.126 +impact of symlinks is being mitigated through the use
1.127 +of an :term:`install manifest`.
1.128 +
1.129 +These issues impact both clobber and incremental builds.
1.130 +
1.131 +Recursive make traversal is slow
1.132 +================================
1.133 +
1.134 +The build system has traditionally been built by employing recursive
1.135 +make. Recursive make involves make iterating through directories / make
1.136 +files sequentially and executing each in turn. This is inefficient for
1.137 +directories containing few targets/tasks because make could be *starved*
1.138 +for work when processing these directories. Any time make is starved,
1.139 +the build isn't using all available CPU cycles and the build is slower
1.140 +as a result.
1.141 +
1.142 +Work has started in bug 907365 to fix this issue by changing the way
1.143 +make traverses all the make files.
1.144 +
1.145 +The impact of slow recursive make traversal is mostly felt on
1.146 +incremental builds. Traditionally, most of the wall time during a
1.147 +no-op build is spent in make traversal.
1.148 +
1.149 +make is inefficient
1.150 +===================
1.151 +
1.152 +Compared to modern build backends like Tup or Ninja, make is slow and
1.153 +inefficient. We can only make make so fast. At some point, we'll hit a
1.154 +performance plateau and will need to use a different tool to make builds
1.155 +faster.
1.156 +
1.157 +Please note that clobber and incremental builds are different. A clobber
1.158 +build with make will likely be as fast as a clobber build with e.g. Tup.
1.159 +However, Tup should vastly outperform make when it comes to incremental
1.160 +builds. Therefore, this issue is mostly seen when performing incremental
1.161 +builds.
1.162 +
1.163 +C++ header dependency hell
1.164 +==========================
1.165 +
1.166 +Modifying a *.h* file can have significant impact on the build system.
1.167 +If you modify a *.h* that is used by 1000 C++ files, all of those 1000
1.168 +C++ files will be recompiled.
1.169 +
1.170 +Our code base has traditionally been sloppy managing the impact of
1.171 +changed headers on build performance. Bug 785103 tracks improving the
1.172 +situation.
1.173 +
1.174 +This issue mostly impacts the times of an :term:`incremental build`.
1.175 +
1.176 +A search/indexing service on your machine is running
1.177 +====================================================
1.178 +
1.179 +Many operating systems have a background service that automatically
1.180 +indexes filesystem content to make searching faster. On Windows, you
1.181 +have the Windows Search Service. On OS X, you have Finder.
1.182 +
1.183 +These background services sometimes take a keen interest in the files
1.184 +being produced as part of the build. Since the build system produces
1.185 +hundreds of megabytes or even a few gigabytes of file data, you can
1.186 +imagine how much work this is to index! If this work is being performed
1.187 +while the build is running, your build will be slower.
1.188 +
1.189 +OS X's Finder is notorious for indexing when the build is running. And,
1.190 +it has a tendency to suck up a whole CPU core. This can make builds
1.191 +several minutes slower. If you build with ``mach`` and have the optional
1.192 +``psutil`` package built (it requires Python development headers - see
1.193 +:ref:`python` for more) and Finder is running during a build, mach will
1.194 +print a warning at the end of the build, complete with instructions on
1.195 +how to fix it.
