michael@0: .. _slow: michael@0: michael@0: ============================ michael@0: Why the Build System is Slow michael@0: ============================ michael@0: michael@0: A common complaint about the build system is that it's slow. There are michael@0: many reasons contributing to its slowness. We will attempt to document michael@0: them here. michael@0: michael@0: First, it is important to distinguish between a :term:`clobber build` michael@0: and an :term:`incremental build`. The reasons for why each are slow can michael@0: be different. michael@0: michael@0: The build does a lot of work michael@0: ============================ michael@0: michael@0: It may not be obvious, but the main reason the build system is slow is michael@0: because it does a lot of work! The source tree consists of a few michael@0: thousand C++ files. On a modern machine, we spend over 120 minutes of CPU michael@0: core time compiling files! So, if you are looking for the root cause of michael@0: slow clobber builds, look at the sheer volume of C++ files in the tree. michael@0: michael@0: You don't have enough CPU cores and MHz michael@0: ======================================= michael@0: michael@0: The build should be CPU bound. If the build system maintainers are michael@0: optimizing the build system perfectly, every CPU core in your machine michael@0: should be 100% saturated during a build. While this isn't currently the michael@0: case (keep reading below), generally speaking, the more CPU cores you michael@0: have in your machine and the more total MHz in your machine, the better. michael@0: michael@0: **We highly recommend building with no fewer than 4 physical CPU michael@0: cores.** Please note the *physical* in this sentence. Hyperthreaded michael@0: cores (an Intel Core i7 will report 8 CPU cores but only 4 are physical michael@0: for example) only yield at most a 1.25x speedup per core. michael@0: michael@0: We also recommend using the most modern CPU model possible. Haswell michael@0: chips deliver much more performance per CPU cycle than say Sandy Bridge michael@0: CPUs. michael@0: michael@0: This cause impacts both clobber and incremental builds. michael@0: michael@0: You are building with a slow I/O layer michael@0: ====================================== michael@0: michael@0: The build system can be I/O bound if your I/O layer is slow. Linking michael@0: libxul on some platforms and build architectures can perform gigabytes michael@0: of I/O. michael@0: michael@0: To minimize the impact of slow I/O on build performance, **we highly michael@0: recommend building with an SSD.** Power users with enough memory may opt michael@0: to build from a RAM disk. Mechanical disks should be avoided if at all michael@0: possible. michael@0: michael@0: Some may dispute the importance of an SSD on build times. It is true michael@0: that the beneficial impact of an SSD can be mitigated if your system has michael@0: lots of memory and the build files stay in the page cache. However, michael@0: operating system memory management is complicated. You don't really have michael@0: control over what or when something is evicted from the page cache. michael@0: Therefore, unless your machine is a dedicated build machine or you have michael@0: more memory than is needed by everything running on your machine, michael@0: chances are you'll run into page cache eviction and you I/O layer will michael@0: impact build performance. That being said, an SSD certainly doesn't michael@0: hurt build times. And, anyone who has used a machine with an SSD will michael@0: tell you how great of an investment it is for performance all around the michael@0: operating system. On top of that, some automated tests are I/O bound michael@0: (like those touching SQLite databases), so an SSD will make tests michael@0: faster. michael@0: michael@0: This cause impacts both clobber and incremental builds. michael@0: michael@0: You don't have enough memory michael@0: ============================ michael@0: michael@0: The build system allocates a lot of memory, especially when building michael@0: many things in parallel. If you don't have enough free system memory, michael@0: the build will cause swap activity, slowing down your system and the michael@0: build. Even if you never get to the point of swapping, the build system michael@0: performs a lot of I/O and having all accessed files in memory and the michael@0: page cache can significantly reduce the influence of the I/O layer on michael@0: the build system. michael@0: michael@0: **We recommend building with no less than 8 GB of system memory.** As michael@0: always, the more memory you have, the better. For a bare bones machine michael@0: doing nothing more than building the source tree, anything more than 16 michael@0: GB is likely entering the point of diminishing returns. michael@0: michael@0: This cause impacts both clobber and incremental builds. michael@0: michael@0: You are building with pymake michael@0: ============================ michael@0: michael@0: Pymake is slower than GNU make. One reason is Python is generally slower michael@0: than C. The build system maintainers are consistently looking at michael@0: optimizing pymake. However, it is death by a thousand cuts. michael@0: michael@0: This cause impacts both clobber and incremental builds. michael@0: michael@0: You are building on Windows michael@0: =========================== michael@0: michael@0: Builds on Windows are slow for a few reasons. First, Windows builds use michael@0: pymake, not GNU make (because of compatibility issues with GNU make). michael@0: But, there are other sources of slowness. michael@0: michael@0: New processes on Windows are about a magnitude slower to spawn than on michael@0: UNIX-y systems such as Linux. This is because Windows has optimized new michael@0: threads while the \*NIX platforms typically optimize new processes. michael@0: Anyway, the build system spawns thousands of new processes during a michael@0: build. Parts of the build that rely on rapid spawning of new processes michael@0: are slow on Windows as a result. This is most pronounced when running michael@0: *configure*. The configure file is a giant shell script and shell michael@0: scripts rely heavily on new processes. This is why configure on Windows michael@0: can run over a minute slower on Windows. michael@0: michael@0: Another reason Windows builds are slower is because Windows lacks proper michael@0: symlink support. On systems that support symlinks, we can generate a michael@0: file into a staging area then symlink it into the final directory very michael@0: quickly. On Windows, we have to perform a full file copy. This incurs michael@0: much more I/O. And if done poorly, can muck with file modification michael@0: times, messing up build dependencies. As of the summer of 2013, the michael@0: impact of symlinks is being mitigated through the use michael@0: of an :term:`install manifest`. michael@0: michael@0: These issues impact both clobber and incremental builds. michael@0: michael@0: Recursive make traversal is slow michael@0: ================================ michael@0: michael@0: The build system has traditionally been built by employing recursive michael@0: make. Recursive make involves make iterating through directories / make michael@0: files sequentially and executing each in turn. This is inefficient for michael@0: directories containing few targets/tasks because make could be *starved* michael@0: for work when processing these directories. Any time make is starved, michael@0: the build isn't using all available CPU cycles and the build is slower michael@0: as a result. michael@0: michael@0: Work has started in bug 907365 to fix this issue by changing the way michael@0: make traverses all the make files. michael@0: michael@0: The impact of slow recursive make traversal is mostly felt on michael@0: incremental builds. Traditionally, most of the wall time during a michael@0: no-op build is spent in make traversal. michael@0: michael@0: make is inefficient michael@0: =================== michael@0: michael@0: Compared to modern build backends like Tup or Ninja, make is slow and michael@0: inefficient. We can only make make so fast. At some point, we'll hit a michael@0: performance plateau and will need to use a different tool to make builds michael@0: faster. michael@0: michael@0: Please note that clobber and incremental builds are different. A clobber michael@0: build with make will likely be as fast as a clobber build with e.g. Tup. michael@0: However, Tup should vastly outperform make when it comes to incremental michael@0: builds. Therefore, this issue is mostly seen when performing incremental michael@0: builds. michael@0: michael@0: C++ header dependency hell michael@0: ========================== michael@0: michael@0: Modifying a *.h* file can have significant impact on the build system. michael@0: If you modify a *.h* that is used by 1000 C++ files, all of those 1000 michael@0: C++ files will be recompiled. michael@0: michael@0: Our code base has traditionally been sloppy managing the impact of michael@0: changed headers on build performance. Bug 785103 tracks improving the michael@0: situation. michael@0: michael@0: This issue mostly impacts the times of an :term:`incremental build`. michael@0: michael@0: A search/indexing service on your machine is running michael@0: ==================================================== michael@0: michael@0: Many operating systems have a background service that automatically michael@0: indexes filesystem content to make searching faster. On Windows, you michael@0: have the Windows Search Service. On OS X, you have Finder. michael@0: michael@0: These background services sometimes take a keen interest in the files michael@0: being produced as part of the build. Since the build system produces michael@0: hundreds of megabytes or even a few gigabytes of file data, you can michael@0: imagine how much work this is to index! If this work is being performed michael@0: while the build is running, your build will be slower. michael@0: michael@0: OS X's Finder is notorious for indexing when the build is running. And, michael@0: it has a tendency to suck up a whole CPU core. This can make builds michael@0: several minutes slower. If you build with ``mach`` and have the optional michael@0: ``psutil`` package built (it requires Python development headers - see michael@0: :ref:`python` for more) and Finder is running during a build, mach will michael@0: print a warning at the end of the build, complete with instructions on michael@0: how to fix it.