1.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
1.2 +++ b/browser/experiments/docs/manifest.rst Wed Dec 31 06:09:35 2014 +0100
1.3 @@ -0,0 +1,429 @@
1.4 +.. _experiments_manifests:
1.5 +
1.6 +=====================
1.7 +Experiments Manifests
1.8 +=====================
1.9 +
1.10 +*Experiments Manifests* are documents that describe the set of active
1.11 +experiments a client may run.
1.12 +
1.13 +*Experiments Manifests* are fetched periodically by clients. When
1.14 +fetched, clients look at the experiments within the manifest and
1.15 +determine which experiments are applicable. If an experiment is
1.16 +applicable, the client may download and start the experiment.
1.17 +
1.18 +Manifest Format
1.19 +===============
1.20 +
1.21 +Manifests are JSON documents where the main element is an object.
1.22 +
1.23 +The *schema* of the object is versioned and defined by the presence
1.24 +of a top-level ``version`` property, whose integer value is the
1.25 +schema version used by that manifest. Each version is documented
1.26 +in the sections below.
1.27 +
1.28 +Version 1
1.29 +---------
1.30 +
1.31 +Version 1 is the original manifest format.
1.32 +
1.33 +The following properties may exist in the root object:
1.34 +
1.35 +experiments
1.36 + An array of objects describing candidate experiments. The format of
1.37 + these objects is documented below.
1.38 +
1.39 + An array is used to create an explicit priority of experiments.
1.40 + Experiments listed at the beginning of the array take priority over
1.41 + experiments that follow.
1.42 +
1.43 +Experiments Objects
1.44 +^^^^^^^^^^^^^^^^^^^
1.45 +
1.46 +Each object in the ``experiments`` array may contain the following
1.47 +properties:
1.48 +
1.49 +id
1.50 + (required) String identifier of this experiment. The identifier should
1.51 + be treated as opaque by clients. It is used to uniquely identify an
1.52 + experiment for all of time.
1.53 +
1.54 +xpiURL
1.55 + (required) String URL of the XPI that implements this experiment.
1.56 +
1.57 + If the experiment is activated, the client will download and install this
1.58 + XPI.
1.59 +
1.60 +xpiHash
1.61 + (required) String hash of the XPI that implements this experiment.
1.62 +
1.63 + The value is composed of a hash identifier followed by a colon
1.64 + followed by the hash value. e.g.
1.65 + `sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`
1.66 + are the two supported hashing mechanisms. The hash value is the hex
1.67 + encoding of the binary hash.
1.68 +
1.69 + When the client downloads the XPI for the experiment, it should compare
1.70 + the hash of that XPI against this value. If the hashes don't match,
1.71 + the client should not install the XPI.
1.72 +
1.73 + Clients may also use this hash as a means of determining when an
1.74 + experiment's XPI has changed and should be refreshed.
1.75 +
1.76 +startTime
1.77 + Integer seconds since UNIX epoch that this experiment should
1.78 + start. Clients should not start an experiment if *now()* is less than
1.79 + this value.
1.80 +
1.81 +maxStartTime
1.82 + (optional) Integer seconds since UNIX epoch after which this experiment
1.83 + should no longer start.
1.84 +
1.85 + Some experiments may wish to impose hard deadlines after which no new
1.86 + clients should activate the experiment. This property may be used to
1.87 + facilitate that.
1.88 +
1.89 +endTime
1.90 + Integer seconds since UNIX epoch after which this experiment
1.91 + should no longer run. Clients should cease an experiment when the current
1.92 + time is beyond this value.
1.93 +
1.94 +maxActiveSeconds
1.95 + Integer seconds defining the max wall time this experiment should be
1.96 + active for.
1.97 +
1.98 + The client should deactivate the experiment this many seconds after
1.99 + initial activation.
1.100 +
1.101 + This value only involves wall time, not browser activity or session time.
1.102 +
1.103 +appName
1.104 + Array of application names this experiment should run on.
1.105 +
1.106 + An application name comes from ``nsIXULAppInfo.name``. It is a value
1.107 + like ``Firefox``, ``Fennec``, or `B2G`.
1.108 +
1.109 + The client should compare its application name against the members of
1.110 + this array. If a match is found, the experiment is applicable.
1.111 +
1.112 +minVersion
1.113 + (optional) String version number of the minimum application version this
1.114 + experiment should run on.
1.115 +
1.116 + A version number is something like ``27.0.0`` or ``28``.
1.117 +
1.118 + The client should compare its version number to this value. If the client's
1.119 + version is greater or equal to this version (using a version-aware comparison
1.120 + function), the experiment is applicable.
1.121 +
1.122 + If this is not specified, there is no lower bound to versions this
1.123 + experiment should run on.
1.124 +
1.125 +maxVersion
1.126 + (optional) String version number of the maximum application version this
1.127 + experiment should run on.
1.128 +
1.129 + This is similar to ``minVersion`` except it sets the upper bound for
1.130 + application versions.
1.131 +
1.132 + If the client's version is less than or equal to this version, the
1.133 + experiment is applicable.
1.134 +
1.135 + If this is not specified, there is no upper bound to versions this
1.136 + experiment should run on.
1.137 +
1.138 +version
1.139 + (optional) Array of application versions this experiment should run on.
1.140 +
1.141 + This is similar to ``minVersion`` and ``maxVersion`` except only a
1.142 + whitelisted set of specific versions are allowed.
1.143 +
1.144 + The client should compare its version to members of this array. If a match
1.145 + is found, the experiment is applicable.
1.146 +
1.147 +minBuildID
1.148 + (optional) String minimum Build ID this experiment should run on.
1.149 +
1.150 + Build IDs are values like ``201402261424``.
1.151 +
1.152 + The client should perform a string comparison of its Build ID against this
1.153 + value. If its value is greater than or equal to this value, the experiment
1.154 + is applicable.
1.155 +
1.156 +maxBuildID
1.157 + (optional) String maximum Build ID this experiment should run on.
1.158 +
1.159 + This is similar to ``minBuildID`` except it sets the upper bound
1.160 + for Build IDs.
1.161 +
1.162 + The client should perform a string comparison of its Build ID against
1.163 + this value. If its value is less than or equal to this value, the
1.164 + experiment is applicable.
1.165 +
1.166 +buildIDs
1.167 + (optional) Array of Build IDs this experiment should run on.
1.168 +
1.169 + This is similar to ``minBuildID`` and ``maxBuildID`` except only a
1.170 + whitelisted set of Build IDs are considered.
1.171 +
1.172 + The client should compare its Build ID to members of this array. If a
1.173 + match is found, the experiment is applicable.
1.174 +
1.175 +os
1.176 + (optional) Array of operating system identifiers this experiment should
1.177 + run on.
1.178 +
1.179 + Values for this array come from ``nsIXULRuntime.OS``.
1.180 +
1.181 + The client will compare its operating system identifier to members
1.182 + of this array. If a match is found, the experiment is applicable to the
1.183 + client.
1.184 +
1.185 +channel
1.186 + (optional) Array of release channel identifiers this experiment should run
1.187 + on.
1.188 +
1.189 + The client will compare its channel to members of this array. If a match
1.190 + is found, the experiment is applicable.
1.191 +
1.192 + If this property is not defined, the client should assume the experiment
1.193 + is to run on all channels.
1.194 +
1.195 +locale
1.196 + (optional) Array of locale identifiers this experiment should run on.
1.197 +
1.198 + A locale identifier is a string like ``en-US`` or ``zh-CN`` and is
1.199 + obtained by looking at
1.200 + ``nsIXULChromeRegistry.getSelectedLocale("global")``.
1.201 +
1.202 + The client should compare its locale identifier to members of this array.
1.203 + If a match is found, the experiment is applicable.
1.204 +
1.205 + If this property is not defined, the client should assume the experiment
1.206 + is to run on all locales.
1.207 +
1.208 +sample
1.209 + (optional) Decimal number indicating the sampling rate for this experiment.
1.210 +
1.211 + This will contain a value between ``0.0`` and ``1.0``. The client should
1.212 + generate a random decimal between ``0.0`` and ``1.0``. If the randomly
1.213 + generated number is less than or equal to the value of this field, the
1.214 + experiment is applicable.
1.215 +
1.216 +disabled
1.217 + (optional) Boolean value indicating whether an experiment is disabled.
1.218 +
1.219 + Normally, experiments are deactivated after a certain time has passed or
1.220 + after the experiment itself determines it no longer needs to run (perhaps
1.221 + it collected sufficient data already).
1.222 +
1.223 + This property serves as a backup mechanism to remotely disable an
1.224 + experiment before it was scheduled to be disabled. It can be used to
1.225 + kill experiments that are found to be doing wrong or bad things or that
1.226 + aren't useful.
1.227 +
1.228 + If this property is not defined or is false, the client should assume
1.229 + the experiment is active and a candidate for activation.
1.230 +
1.231 +frozen
1.232 + (optional) Boolean value indicating this experiment is frozen and no
1.233 + longer accepting new enrollments.
1.234 +
1.235 + If a client sees a true value in this field, it should not attempt to
1.236 + activate an experiment.
1.237 +
1.238 +jsfilter
1.239 + (optional) JavaScript code that will be evaluated to determine experiment
1.240 + applicability.
1.241 +
1.242 + This property contains the string representation of JavaScript code that
1.243 + will be evaluated in a sandboxed environment using JavaScript's
1.244 + ``eval()``.
1.245 +
1.246 + The string is expected to contain the definition of a JavaScript function
1.247 + ``filter(context)``. This function receives as its argument an object
1.248 + holding application state. See the section below for the definition of
1.249 + this object.
1.250 +
1.251 + The purpose of this property is to allow experiments to define complex
1.252 + rules and logic for evaluating experiment applicability in a manner
1.253 + that is privacy conscious and doesn't require the transmission of
1.254 + excessive data.
1.255 +
1.256 + The return value of this filter indicates whether the experiment is
1.257 + applicable. Functions should return true if the experiment is
1.258 + applicable.
1.259 +
1.260 + If an experiment is not applicable, they should throw an Error whose
1.261 + message contains the reason the experiment is not applicable. This
1.262 + message may be logged and sent to remote servers, so it should not
1.263 + contain private or otherwise sensitive data that wouldn't normally
1.264 + be submitted.
1.265 +
1.266 + If a falsey (or undefined) value is returned, the client should
1.267 + assume the experiment is not applicable.
1.268 +
1.269 + If this property is not defined, the client does not consider a custom
1.270 + JavaScript filter function when determining whether an experiment is
1.271 + applicable.
1.272 +
1.273 +JavaScript Filter Context Objects
1.274 +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.275 +
1.276 +The object passed to a ``jsfilter`` ``filter()`` function contains the
1.277 +following properties:
1.278 +
1.279 +healthReportSubmissionEnabled
1.280 + This property contains a boolean indicating whether Firefox Health
1.281 + Report has its data submission flag enabled (whether Firefox Health
1.282 + Report is sending data to remote servers).
1.283 +
1.284 +healthReportPayload
1.285 + This property contains the current Firefox Health Report payload.
1.286 +
1.287 + The payload format is documented at :ref:`healthreport_dataformat`.
1.288 +
1.289 +telemetryPayload
1.290 + This property contains the current Telemetry payload.
1.291 +
1.292 +The evaluation sandbox for the JavaScript filters may be destroyed
1.293 +immediately after ``filter()`` returns. This function should not assume
1.294 +async code will finish.
1.295 +
1.296 +Experiment Applicability and Client Behavior
1.297 +============================================
1.298 +
1.299 +The point of an experiment manifest is to define which experiments are
1.300 +available and where and how to run them. This section explains those
1.301 +rules in more detail.
1.302 +
1.303 +Many of the properties in *Experiment Objects* are related to determining
1.304 +whether an experiment should run on a given client. This evaluation is
1.305 +performed client side.
1.306 +
1.307 +1. Multiple conditions in an experiment
1.308 +---------------------------------------
1.309 +
1.310 +If multiple conditions are defined for an experiment, the client should
1.311 +combine each condition with a logical *AND*: all conditions must be
1.312 +satisfied for an experiment to run. If one condition fails, the experiment
1.313 +is not applicable.
1.314 +
1.315 +2. Active experiment disappears from manifest
1.316 +---------------------------------------------
1.317 +
1.318 +If a specific experiment disappears from the manifest, the client should
1.319 +continue conducting an already-active experiment. Furthermore, the
1.320 +client should remember what the expiration events were for an experiment
1.321 +and honor them.
1.322 +
1.323 +The rationale here is that we want to prevent an accidental deletion
1.324 +or temporary failure on the server to inadvertantly deactivate
1.325 +supposed-to-be-active experiments. We also don't want premature deletion
1.326 +of an experiment from the manifest to result in indefinite activation
1.327 +periods.
1.328 +
1.329 +3. Inactive experiment disappears from manifest
1.330 +-----------------------------------------------
1.331 +
1.332 +If an inactive but scheduled-to-be-active experiment disappears from the
1.333 +manifest, the client should not activate the experiment.
1.334 +
1.335 +If that experiment reappears in the manifest, the client should not
1.336 +treat that experiment any differently than any other new experiment. Put
1.337 +another way, the fact an inactive experiment disappears and then
1.338 +reappears should not be significant.
1.339 +
1.340 +The rationale here is that server operators should have complete
1.341 +control of an inactive experiment up to it's go-live date.
1.342 +
1.343 +4. Re-evaluating applicability on manifest refresh
1.344 +--------------------------------------------------
1.345 +
1.346 +When an experiment manifest is refreshed or updated, the client should
1.347 +re-evaluate the applicability of each experiment therein.
1.348 +
1.349 +The rationale here is that the server may change the parameters of an
1.350 +experiment and want clients to pick those up.
1.351 +
1.352 +5. Activating a previously non-applicable experiment
1.353 +----------------------------------------------------
1.354 +
1.355 +If the conditions of an experiment change or the state of the client
1.356 +changes to allow an experiment to transition from previously
1.357 +non-applicable to applicable, the experiment should be activated.
1.358 +
1.359 +For example, if a client is running version 28 and the experiment
1.360 +initially requires version 29 or above, the client will not mark the
1.361 +experiment as applicable. But if the client upgrades to version 29 or if
1.362 +the manifest is updated to require 28 or above, the experiment will
1.363 +become applicable.
1.364 +
1.365 +6. Deactivating a previously active experiment
1.366 +----------------------------------------------
1.367 +
1.368 +If the conditions of an experiment change or the state of the client
1.369 +changes and an active experiment is no longer applicable, that
1.370 +experiment should be deactivated.
1.371 +
1.372 +7. Calculation of sampling-based applicability
1.373 +----------------------------------------------
1.374 +
1.375 +For calculating sampling-based applicability, the client will associate
1.376 +a random value between ``0.0`` and ``1.0`` for each observed experiment
1.377 +ID. This random value will be generated the first time sampling
1.378 +applicability is evaluated. This random value will be persisted and used
1.379 +in future applicability evaluations for this experiment.
1.380 +
1.381 +By saving and re-using the value, the client is able to reliably and
1.382 +consistently evaluate applicability, even if the sampling threshold
1.383 +in the manifest changes.
1.384 +
1.385 +Clients should retain the randomly-generated sampling value for
1.386 +experiments that no longer appear in a manifest for a period of at least
1.387 +30 days. The rationale is that if an experiment disappears and reappears
1.388 +from a manifest, the client will not have multiple opportunities to
1.389 +generate a random value that satisfies the sampling criteria.
1.390 +
1.391 +8. Incompatible version numbers
1.392 +-------------------------------
1.393 +
1.394 +If a client receives a manifest with a version number that it doesn't
1.395 +recognize, it should ignore the manifest.
1.396 +
1.397 +9. Usage of old manifests
1.398 +-------------------------
1.399 +
1.400 +If a client experiences an error fetching a manifest (server not
1.401 +available) or if the manifest is corrupt, not readable, or compatible,
1.402 +the client may use a previously-fetched (cached) manifest.
1.403 +
1.404 +10. Updating XPIs
1.405 +-----------------
1.406 +
1.407 +If the URL or hash of an active experiment's XPI changes, the client
1.408 +should fetch the new XPI, uninstall the old XPI, and install the new
1.409 +XPI.
1.410 +
1.411 +Examples
1.412 +========
1.413 +
1.414 +Here is an example manifest::
1.415 +
1.416 + {
1.417 + "version": 1,
1.418 + "experiments": [
1.419 + {
1.420 + "id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",
1.421 + "xpiURL": "https://experiments.mozilla.org/foo.xpi",
1.422 + "xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",
1.423 + "startTime": 1393000000,
1.424 + "endTime": 1394000000,
1.425 + "appName": ["Firefox", "Fennec"],
1.426 + "minVersion": "28",
1.427 + "maxVersion": "30",
1.428 + "os": ["windows", "linux", "osx"],
1.429 + "jsfilter": "function filter(context) { return context.healthReportEnabled; }"
1.430 + }
1.431 + ]
1.432 + }
