michael@0:                       Trace Malloc Tools
michael@0:               Chris Waterson <waterson@netscape.com>
michael@0:                       November 27, 2000
michael@0: 
michael@0: This is a short primer on how to use the `trace malloc' tools
michael@0: contained in this directory.
michael@0: 
michael@0: 
michael@0: WHAT IS TRACE MALLOC?
michael@0: =====================
michael@0: 
michael@0: Trace malloc is an optional facility that is built in to XPCOM. It
michael@0: uses `weak linking' to intercept all calls to malloc(), calloc(),
michael@0: realloc() and free(). It does two things:
michael@0: 
michael@0: 1. Writes information about allocations to a filehandle that you
michael@0: specify. As each call to malloc(), et. al. is made, a record is logged
michael@0: to the filehandle.
michael@0: 
michael@0: 2. Maintains a table of all `live objects' -- that is, objects that
michael@0: have been allocated by malloc(), calloc() or realloc(), but have not
michael@0: yet been free()'d. The contents of this table can be called by making
michael@0: a `secret' call to JavaScript.
michael@0: 
michael@0: 
michael@0: MAKING A TRACE MALLOC BUILD
michael@0: ===========================
michael@0: 
michael@0: As of this writing, trace malloc only works on Linux, but work is
michael@0: underway to port it to Windows.
michael@0: 
michael@0: On Linux, start with a clean tree, and configure your build with the
michael@0: following flags:
michael@0: 
michael@0:   --enable-trace-malloc
michael@0:   --enable-cpp-rtti
michael@0: 
michael@0: Be sure that `--enable-boehm' is *not* set. I don't think that the
michael@0: values for `--enable-debug' and `--enable-optimize' matter, but I've
michael@0: typically had debug on and optimize off.
michael@0: 
michael@0: 
michael@0: COLLECTING LIVE OBJECT DATA
michael@0: ===========================
michael@0: 
michael@0: To collect `live object' data from `mozilla' using a build that has
michael@0: trace malloc enabled,
michael@0: 
michael@0:   1. Run `mozilla' as follows:
michael@0: 
michael@0:      % mozilla --trace-malloc /dev/null
michael@0: 
michael@0:   2. Do whatever operations in mozilla you'd like to test.
michael@0: 
michael@0:   3. Open the `live-bloat.html' file contained in this directory.
michael@0: 
michael@0:   4. Press the button that says `Dump to /tmp/dump.log'
michael@0: 
michael@0: An enormous file (typically 300MB) called `dump.log' will be dropped
michael@0: in your `/tmp' directory.
michael@0: 
michael@0: To collect live object data from `gtkEmbed' using a build that has
michael@0: trace malloc enabled:
michael@0: 
michael@0:   1. Run `gtkEmbed' as follows:
michael@0: 
michael@0:      % gtkEmbed --trace-malloc /dev/null
michael@0: 
michael@0:   2. Do whatever operations in gtkEmbed that you'd like to test.
michael@0: 
michael@0:   3. Press the `Dump Memory' button at the bottom of gtkEmbed.
michael@0: 
michael@0: The enormous file will be dropped in the current directory, and is
michael@0: called `allocations.log'.
michael@0: 
michael@0: 
michael@0: About Live Object Logs
michael@0: ----------------------
michael@0: 
michael@0: A typical entry from the `live object' dump file will look like:
michael@0: 
michael@0:   Address     Type      Size
michael@0:   |           |         |
michael@0:   v           v         v
michael@0:   0x40008080 <nsFooBar> 16
michael@0:         0x00000001 <- Fields
michael@0:         0x40008084
michael@0:         0x80004001
michael@0:         0x00000036
michael@0:   __builtin_new[./libxpcom.so +0x10E9DC]  <- Stack at allocation time
michael@0:   nsFooBar::CreateFooBar(nsFooBar **)[./libfoobar.so +0x408C]
michael@0:   main+C7E5AFB5[(null) +0xC7E5AFB5]
michael@0: 
michael@0: One will be printed for each object that was allocated.
michael@0: 
michael@0: 
michael@0: TOOLS TO PARSE LIVE OBJECT LOGS
michael@0: ===============================
michael@0: 
michael@0: This directory is meant to house the tools that you can use to parse
michael@0: live-object logs.
michael@0: 
michael@0: Object Histograms - histogram.pl
michael@0: --------------------------------
michael@0: 
michael@0: This program parses a `live object' dump and produces a histogram of
michael@0: the objects, sorted from objects that take the most memory to objects
michael@0: that take the least. The output of this program is rather spartan: on
michael@0: each line, it prints the object type, the number of objects of that
michael@0: type, and the total number of bytes that the objects consume.
michael@0: 
michael@0: There are a two simple programs to `pretty print' the output from
michael@0: histogram.pl:
michael@0: 
michael@0:   1. histogram-pretty.sh takes a single histogram and produces a table
michael@0:      of objects.
michael@0: 
michael@0:        Type                    Count    Bytes %Total
michael@0:        TOTAL                   67348  4458127 100.00
michael@0:        nsImageGTK                 76   679092  15.23
michael@0:        void*                    8956   563572  12.64
michael@0:        ...
michael@0:        PRLock                    732    61488   1.38
michael@0:        OTHER                   24419   940235  21.09
michael@0: 
michael@0:   2. histogram-diff.sh takes two histograms and computes the difference
michael@0:      between them.
michael@0: 
michael@0:                   ---- Base ----   ---- Incr ----   ----- Difference ----
michael@0:        Type       Count    Bytes   Count    Bytes   Count    Bytes %Total
michael@0:        TOTAL      40241  1940945   73545  5315142   33304  3374197 100.00
michael@0:        nsImageGTK    16   106824     151   832816     135   725992  21.52
michael@0:        PresShell     16    51088     198   340706     182   289618   8.58
michael@0:        ...
michael@0:        OTHER      27334  1147033   38623  1493385   11289   346352  10.26
michael@0: 
michael@0: Both of these scripts accept `-c' parameter that specifies how many
michael@0: rows you'd like to see (by default, twenty). Any rows past the first
michael@0: `n' rows are lumped into a single `OTHER' row. This allows you to keep
michael@0: your reports short n' sweet.
michael@0: 
michael@0: Stack-based Type Inference - types.dat
michael@0: --------------------------------------
michael@0: 
michael@0: Trace malloc uses `speculative RTTI' to determine the types of objects
michael@0: as it dumps them. Unfortunately, RTTI can only deduce the type name
michael@0: for C++ objects with a virtual destructor.
michael@0: 
michael@0: This leaves:
michael@0: 
michael@0:  . C++ object without a virtual destructor
michael@0:  . array allocated C++ objects, and
michael@0:  . objects allocated with the C runtime function (malloc
michael@0:    and friends)
michael@0: 
michael@0: out in the cold. Trace malloc reports objects allocated this was as
michael@0: having type `void*'.
michael@0: 
michael@0: The good news is that you can almost always determine the object's
michael@0: type by looking at the stack trace that's taken at the time the object
michael@0: is allocated.
michael@0: 
michael@0: The file `types.dat' consists of rules to classify objects based on
michael@0: stack trace.
michael@0: 
michael@0: 
michael@0: Uncategorized Objects - uncategorized.pl
michael@0: ----------------------------------------
michael@0: 
michael@0: Categorizing objects in `types.dat' is sweaty work, and the
michael@0: `uncategorized.pl' script is a tool that makes it a bit
michael@0: easier. Specifically, it reads a `live object' dump file and sorts the
michael@0: stack traces. Stack traces that account for the most uncategorized
michael@0: objects are placed first.
michael@0: 
michael@0: Using this tool, you can add the `most effective' rules to
michael@0: `types.dat': rules that account for most of the uncategorized data.