|
1 ============== |
|
2 Crash Reporter |
|
3 ============== |
|
4 |
|
5 Overview |
|
6 ======== |
|
7 |
|
8 The **crash reporter** is a subsystem to record and manage application |
|
9 crash data. |
|
10 |
|
11 While the subsystem is known as *crash reporter*, it helps to think of |
|
12 it more as a *process dump manager*. This is because the heart of this |
|
13 subsystem is really managing process dump files and these files are |
|
14 created not only from process crashes but also from hangs and other |
|
15 exceptional events. |
|
16 |
|
17 The crash reporter subsystem is composed of a number of pieces working |
|
18 together. |
|
19 |
|
20 Breakpad |
|
21 Breakpad is a library and set of tools to make collecting process |
|
22 information (notably dumps from crashes) easy. Breakpad is a 3rd |
|
23 party project (originaly developed by Google) that is imported into |
|
24 the tree. |
|
25 |
|
26 Dump files |
|
27 Breakpad produces files called *dump files* that hold process data |
|
28 (stacks, heap data, etc). |
|
29 |
|
30 Crash Reporter Client |
|
31 The crash reporter client is a standalone executable that is launched |
|
32 to handle dump files. This application optionally submits crashes to |
|
33 Mozilla (or the configured server). |
|
34 |
|
35 How Main-Process Crash Handling Works |
|
36 ===================================== |
|
37 |
|
38 The crash handler is hooked up very early in the Gecko process lifetime. |
|
39 It all starts in ``XREMain::XRE_mainInit()`` from ``nsAppRunner.cpp``. |
|
40 Assuming crash reporting is enabled, this startup function registers an |
|
41 exception handler for the process and tells the crash reporter subsystem |
|
42 about basic metadata such as the application name and version. |
|
43 |
|
44 The registration of the crash reporter exception handler doubles as |
|
45 initialization of the crash reporter itself. This happens in |
|
46 ``CrashReporter::SetExceptionHandler()`` from ``nsExceptionHandler.cpp``. |
|
47 The crash reporter figures out what application to use for reporting |
|
48 dumped crashes and where to store these dump files on disk. The Breakpad |
|
49 exception handler (really just a mechanism for dumping process state) is |
|
50 initialized as part of this function. The Breakpad exception handler is |
|
51 a ``google_breakpad::ExceptionHandler`` instance and it's stored as |
|
52 ``gExceptionHandler``. |
|
53 |
|
54 As the application runs, various other systems may write *annotations* |
|
55 or *notes* to the crash reporter to indicate state of the application, |
|
56 help with possible reasons for a current or future crash, etc. These are |
|
57 performed via ``CrashReporter::AnnotateCrashReport()`` and |
|
58 ``CrashReporter::AppendAppNotesToCrashReport()`` from |
|
59 ``nsExceptionHandler.h``. |
|
60 |
|
61 For well running applications, this is all that happens. However, if a |
|
62 crash or similar exceptional event occurs (such as a hang), we need to |
|
63 write a crash report. |
|
64 |
|
65 When an event worthy of writing a dump occurs, the Breakpad exception |
|
66 handler is invoked and Breakpad does its thing. When Breakpad has |
|
67 finished, it calls back into ``CrashReporter::MinidumpCallback()`` from |
|
68 ``nsExceptionHandler.cpp`` to tell the crash reporter about what was |
|
69 written. |
|
70 |
|
71 ``MinidumpCallback()`` performs a number of actions once a dump has been |
|
72 written. It writes a file with the time of the crash so other systems can |
|
73 easily determine the time of the last crash. It supplements the dump |
|
74 file with an *extra* file containing Mozilla-specific metadata. This data |
|
75 includes the annotations set via ``CrashReporter::AnnotateCrashReport()`` |
|
76 as well as time since last crash, whether garbage collection was active at |
|
77 the time of the crash, memory statistics, etc. |
|
78 |
|
79 If the *crash reporter client* is enabled, ``MinidumpCallback()`` invokes |
|
80 it. It simply tries to create a new *crash reporter client* process (e.g. |
|
81 *crashreporter.exe*) with the path to the written minidump file as an |
|
82 argument. |
|
83 |
|
84 The *crash reporter client* performs a number of roles. There's a lot going |
|
85 on, so you may want to look at ``main()`` in ``crashreporter.cpp``. First, |
|
86 it verifies the dump data is sane. If it isn't (e.g. required metadata is |
|
87 missing), the dump data is ignored. If dump data looks sane, the dump data |
|
88 is moved into the *pending* directory for the configured data directory |
|
89 (defined via the ``MOZ_CRASHREPORTER_DATA_DIRECTORY`` environment variable |
|
90 or from the UI). Once this is done, the main crash reporter UI is displayed |
|
91 via ``UIShowCrashUI()``. The crash reporter UI is platform specific: there |
|
92 are separate versions for Windows, OS X, and various \*NIX presentation |
|
93 flavors (such as GTK). The basic gist is a dialog is displayed to the user |
|
94 and the user has the opportunity to submit this dump data to a remote |
|
95 server. |
|
96 |
|
97 If a dump is submitted via the crash reporter, the raw dump files are |
|
98 removed from the *pending* directory and a file containing the |
|
99 crash ID from the remote server for the submitted dump is created in the |
|
100 *submitted* directory. |
|
101 |
|
102 If the user chooses not to submit a dump in the crash reporter UI, the dump |
|
103 files are deleted. |
|
104 |
|
105 And that's pretty much what happens when a crash/dump is written! |
|
106 |
|
107 Plugin and Child Process Crashes |
|
108 ================================ |
|
109 |
|
110 Crashes in plugin and child processes are also managed by the crash |
|
111 reporting subsystem. |
|
112 |
|
113 Child process crashes are handled by the ``mozilla::dom::CrashReporterParent`` |
|
114 class defined in ``dom/ipc``. When a child process crashes, the toplevel IPDL |
|
115 actor should check for it by calling TakeMinidump in its ``ActorDestroy`` |
|
116 Method: see ``mozilla::plugins::PluginModuleParent::ActorDestroy`` and |
|
117 ``mozilla::plugins::PluginModuleParent::ProcessFirstMinidump``. That method |
|
118 is responsible for calling |
|
119 ``mozilla::dom::CrashReporterParent::GenerateCrashReportForMinidump`` with |
|
120 appropriate crash annotations specific to the crash. All child-process |
|
121 crashes are annotated with a ``ProcessType`` annotation, such as "content" or |
|
122 "plugin". |
|
123 |
|
124 Submission of child process crashes is handled by application code. This |
|
125 code prompts the user to submit crashes in context-appropriate UI and then |
|
126 submits the crashes using ``CrashSubmit.jsm``. |
|
127 |
|
128 Flash Process Crashes |
|
129 ===================== |
|
130 |
|
131 On Windows Vista+, the Adobe Flash plugin creates two extra processes in its |
|
132 Firefox plugin to implement OS-level sandboxing. In order to catch crashes in |
|
133 these processes, Firefox injects a crash report handler into the process using the code at ``InjectCrashReporter.cpp``. When these crashes occur, the |
|
134 ProcessType=plugin annotation is present, and an additional annotation |
|
135 FlashProcessDump has the value "Sandbox" or "Broker". |
|
136 |
|
137 Plugin Hangs |
|
138 ============ |
|
139 |
|
140 Plugin hangs are handled as crash reports. If a plugin doesn't respond to an |
|
141 IPC message after 60 seconds, the plugin IPC code will take minidumps of all |
|
142 of the processes involved and then kill the plugin. |
|
143 |
|
144 In this case, there will be only one .ini file with the crash report metadata, |
|
145 but there will be multiple dump files: at least one for the browser process and |
|
146 one for the plugin process, and perhaps also additional dumps for the Flash |
|
147 sandbox and broker processes. All of these files are submitted together as a |
|
148 unit. Before submission, the filenames of the files are linked: |
|
149 |
|
150 - **uuid.ini** - *annotations, includes an additional_minidumps field* |
|
151 - **uuid.dmp** - *plugin process dump file* |
|
152 - **uuid-<other>.dmp** - *other process dump file as listed in additional_minidumps* |
|
153 |
|
154 Browser Hangs |
|
155 ============= |
|
156 |
|
157 There is a feature of Firefox that will crash Firefox if it stops processing |
|
158 messages after a certain period of time. This feature doesn't work well and is |
|
159 disabled by default. See ``xpcom/threads/HangMonitor.cpp``. Hang crashes |
|
160 are annotated with ``Hang=1``. |
|
161 |
|
162 about:crashes |
|
163 ============= |
|
164 |
|
165 If the crash reporter subsystem is enabled, the *about:crashes* |
|
166 page will be registered with the application. This page provides |
|
167 information about previous and submitted crashes. |
|
168 |
|
169 It is also possible to submit crashes from *about:crashes*. |