• file
• revisions
• annotate
• diff
• comparison
• raw

services/healthreport/docs/architecture.rst@97036ab72558 (annotated)

services/healthreport/docs/architecture.rst

Tue, 06 Jan 2015 21:39:09 +0100

author

Michael Schloh von Bennewitz <michael@schloh.com>

date

Tue, 06 Jan 2015 21:39:09 +0100

branch

TOR_BUG_9701

changeset 8

97036ab72558

permissions

-rw-r--r--

Conditionally force memory storage according to privacy.thirdparty.isolate;
This solves Tor bug #9701, complying with disk avoidance documented in
https://www.torproject.org/projects/torbrowser/design/#disk-avoidance.

michael@0	1	.. _healthreport_architecture:
michael@0	2
michael@0	3	============
michael@0	4	Architecture
michael@0	5	============
michael@0	6
michael@0	7	``healthreporter.jsm`` contains the main interface for FHR, the
michael@0	8	``HealthReporter`` type. An instance of this is created by the
michael@0	9	:ref:`data_reporting_service`.
michael@0	10
michael@0	11	``providers.jsm`` contains numerous ``Metrics.Provider`` and
michael@0	12	``Metrics.Measurement`` used for collecting application metrics. If you
michael@0	13	are looking for the FHR probes, this is where they are.
michael@0	14
michael@0	15	Storage
michael@0	16	=======
michael@0	17
michael@0	18	Firefox Health Report stores data in 3 locations:
michael@0	19
michael@0	20	* Metrics measurements and provider state is stored in a SQLite database
michael@0	21	(via ``Metrics.Storage``).
michael@0	22	* Service state (such as the IDs of documents uploaded) is stored in a
michael@0	23	JSON file on disk (via OS.File).
michael@0	24	* Lesser state and run-time options are stored in preferences.
michael@0	25
michael@0	26	Preferences
michael@0	27	===========
michael@0	28
michael@0	29	Preferences controlling behavior of Firefox Health Report live in the
michael@0	30	``datareporting.healthreport.*`` branch.
michael@0	31
michael@0	32	Service and Data Control
michael@0	33	------------------------
michael@0	34
michael@0	35	The follow preferences control behavior of the service and data upload.
michael@0	36
michael@0	37	service.enabled
michael@0	38	Controls whether the entire health report service runs. The overall
michael@0	39	service performs data collection, storing, and submission.
michael@0	40
michael@0	41	This is the primary kill switch for Firefox Health Report
michael@0	42	outside of the build system variable. i.e. if you are using an
michael@0	43	official Firefox build and wish to disable FHR, this is what you
michael@0	44	should set to false to prevent FHR from not only submitting but
michael@0	45	also collecting data.
michael@0	46
michael@0	47	uploadEnabled
michael@0	48	Whether uploading of data is enabled. This is the preference the
michael@0	49	checkbox in the preferences UI reflects. If this is
michael@0	50	disabled, FHR still collects data - it just doesn't upload it.
michael@0	51
michael@0	52	service.loadDelayMsec
michael@0	53	How long (in milliseconds) after initial application start should FHR
michael@0	54	wait before initializing.
michael@0	55
michael@0	56	FHR may initialize sooner than this if the FHR service is requested.
michael@0	57	This will happen if e.g. the user goes to ``about:healthreport``.
michael@0	58
michael@0	59	service.loadDelayFirstRunMsec
michael@0	60	How long (in milliseconds) FHR should wait to initialize on first
michael@0	61	application run.
michael@0	62
michael@0	63	FHR waits longer than normal to initialize on first application run
michael@0	64	because first-time initialization can use a lot of I/O to initialize
michael@0	65	the SQLite database and this I/O should not interfere with the
michael@0	66	first-run user experience.
michael@0	67
michael@0	68	documentServerURI
michael@0	69	The URI of a Bagheera server that FHR should interface with for
michael@0	70	submitting documents.
michael@0	71
michael@0	72	You typically do not need to change this.
michael@0	73
michael@0	74	documentServerNamespace
michael@0	75	The namespace on the document server FHR should upload documents to.
michael@0	76
michael@0	77	You typically do not need to change this.
michael@0	78
michael@0	79	infoURL
michael@0	80	The URL of a page containing more info about FHR, it's privacy
michael@0	81	policy, etc.
michael@0	82
michael@0	83	about.reportUrl
michael@0	84	The URL to load in ``about:healthreport``.
michael@0	85
michael@0	86	service.providerCategories
michael@0	87	A comma-delimited list of category manager categories that contain
michael@0	88	registered ``Metrics.Provider`` records. Read below for how provider
michael@0	89	registration works.
michael@0	90
michael@0	91	If the entire service is disabled, you lose data collection. This means
michael@0	92	that **local** data analysis won't be available because there is no data
michael@0	93	to analyze! Keep in mind that Firefox Health Report can be useful even
michael@0	94	if it's not submitting data to remote servers!
michael@0	95
michael@0	96	Logging
michael@0	97	-------
michael@0	98
michael@0	99	The following preferences allow you to control the logging behavior of
michael@0	100	Firefox Health Report.
michael@0	101
michael@0	102	logging.consoleEnabled
michael@0	103	Whether to write log messages to the web console. This is true by
michael@0	104	default.
michael@0	105
michael@0	106	logging.consoleLevel
michael@0	107	The minimum log level FHR messages must have to be written to the
michael@0	108	web console. By default, only FHR warnings or errors will be written
michael@0	109	to the web console. During normal/expected operation, no messages of
michael@0	110	this type should be produced.
michael@0	111
michael@0	112	logging.dumpEnabled
michael@0	113	Whether to write log messages via ``dump()``. If true, FHR will write
michael@0	114	messages to stdout/stderr.
michael@0	115
michael@0	116	This is typically only enabled when developing FHR.
michael@0	117
michael@0	118	logging.dumpLevel
michael@0	119	The minimum log level messages must have to be written via
michael@0	120	``dump()``.
michael@0	121
michael@0	122	State
michael@0	123	-----
michael@0	124
michael@0	125	currentDaySubmissionFailureCount
michael@0	126	How many submission failures the client has encountered while
michael@0	127	attempting to upload the most recent document.
michael@0	128
michael@0	129	lastDataSubmissionFailureTime
michael@0	130	The time of the last failed document upload.
michael@0	131
michael@0	132	lastDataSubmissionRequestedTime
michael@0	133	The time of the last document upload attempt.
michael@0	134
michael@0	135	lastDataSubmissionSuccessfulTime
michael@0	136	The time of the last successful document upload.
michael@0	137
michael@0	138	nextDataSubmissionTime
michael@0	139	The time the next data submission is scheduled for. FHR will not
michael@0	140	attempt to upload a new document before this time.
michael@0	141
michael@0	142	pendingDeleteRemoteData
michael@0	143	Whether the client currently has a pending request to delete remote
michael@0	144	data. If true, the client will attempt to delete all remote data
michael@0	145	before an upload is performed.
michael@0	146
michael@0	147	FHR stores various state in preferences.
michael@0	148
michael@0	149	Registering Providers
michael@0	150	=====================
michael@0	151
michael@0	152	Firefox Health Report providers are registered via the category manager.
michael@0	153	See ``HealthReportComponents.manifest`` for providers defined in this
michael@0	154	directory.
michael@0	155
michael@0	156	Essentially, the category manager receives the name of a JS type and the
michael@0	157	URI of a JSM to import that exports this symbol. At run-time, the
michael@0	158	providers registered in the category manager are instantiated.
michael@0	159
michael@0	160	Providers are registered via the category manager to make registration
michael@0	161	simple and less prone to errors. Any XPCOM component can create a
michael@0	162	category manager entry. Therefore, new data providers can be added
michael@0	163	without having to touch core Firefox Health Report code. Additionally,
michael@0	164	category manager registration means providers are more likely to be
michael@0	165	registered on FHR's terms, when it wants. If providers were registered
michael@0	166	in code at application run-time, there would be the risk of other
michael@0	167	components prematurely instantiating FHR (causing a performance hit if
michael@0	168	performed at an inopportune time) or semi-complicated code around
michael@0	169	observers or listeners. Category manager entries are only 1 line per
michael@0	170	provider and leave FHR in control: they are simple and safe.
michael@0	171
michael@0	172	Document Generation and Lifecycle
michael@0	173	=================================
michael@0	174
michael@0	175	FHR will attempt to submit a JSON document containing data every 24 wall
michael@0	176	clock hours.
michael@0	177
michael@0	178	At upload time, FHR will query the database for **all** information from
michael@0	179	the last 180 days and assemble this data into a JSON document. We
michael@0	180	attempt to upload this JSON document with a client-generated UUID to the
michael@0	181	configured server.
michael@0	182
michael@0	183	Before we attempt upload, the generated UUID is stored in the JSON state
michael@0	184	file on local disk. At this point, the client assumes the document with
michael@0	185	that UUID has been successfully stored on the server.
michael@0	186
michael@0	187	If the client is aware of other document UUIDs that presumably exist on
michael@0	188	the server, those UUIDs are sent with the upload request so the client
michael@0	189	can request those UUIDs be deleted. This helps ensure that each client
michael@0	190	only has 1 document/UUID on the server at any one time.
michael@0	191
michael@0	192	Importance of Persisting UUIDs
michael@0	193	------------------------------
michael@0	194
michael@0	195	The choices of how, where, and when document UUIDs are stored and updated
michael@0	196	are very important. One should not attempt to change things unless she
michael@0	197	has a very detailed understanding of why things are the way they are.
michael@0	198
michael@0	199	The client is purposefully very conservative about forgetting about
michael@0	200	generated UUIDs. In other words, once a UUID is generated, the client
michael@0	201	deliberately holds on to that UUID until it's very confident that UUID
michael@0	202	is no longer stored on the server. The reason we do this is because
michael@0	203	*orphaned* documents/UUIDs on the server can lead to faulty analysis,
michael@0	204	such as over-reporting the number of Firefox installs that stop being
michael@0	205	used.
michael@0	206
michael@0	207	When uploading a new UUID, we update the state and save the state file
michael@0	208	to disk *before* an upload attempt because if the upload succeeds but
michael@0	209	the response never makes it back to the client, we want the client to
michael@0	210	know about the uploaded UUID so it can delete it later to prevent an
michael@0	211	orphan.
michael@0	212
michael@0	213	We maintain a list of UUIDs locally (not simply the last UUID) because
michael@0	214	multiple upload attempts could fail the same way as the previous
michael@0	215	paragraph describes and we have no way of knowing which (if any)
michael@0	216	actually succeeded. The safest approach is to assume every document
michael@0	217	produced managed to get uploaded some how.
michael@0	218
michael@0	219	We store the UUIDs on a file on disk and not anywhere else because we
michael@0	220	want storage to be robust. We originally stored UUIDs in preferences,
michael@0	221	which only flush to disk periodically. Writes to preferences were
michael@0	222	apparently getting lost. We switched to writing directly to files to
michael@0	223	eliminate this window.