The Tor Browser / file revision

 .. _experiments_manifests:

 =====================

 Experiments Manifests

 =====================

 *Experiments Manifests* are documents that describe the set of active

 experiments a client may run.

 *Experiments Manifests* are fetched periodically by clients. When

 fetched, clients look at the experiments within the manifest and

 determine which experiments are applicable. If an experiment is

 applicable, the client may download and start the experiment.

 Manifest Format

 ===============

 Manifests are JSON documents where the main element is an object.

 The *schema* of the object is versioned and defined by the presence

 of a top-level ``version`` property, whose integer value is the

 schema version used by that manifest. Each version is documented

 in the sections below.

 Version 1

 ---------

 Version 1 is the original manifest format.

 The following properties may exist in the root object:

 experiments

    An array of objects describing candidate experiments. The format of

    these objects is documented below.

    An array is used to create an explicit priority of experiments.

    Experiments listed at the beginning of the array take priority over

    experiments that follow.

 Experiments Objects

 ^^^^^^^^^^^^^^^^^^^

 Each object in the ``experiments`` array may contain the following

 properties:

 id

    (required) String identifier of this experiment. The identifier should

    be treated as opaque by clients. It is used to uniquely identify an

    experiment for all of time.

 xpiURL

    (required) String URL of the XPI that implements this experiment.

    If the experiment is activated, the client will download and install this

    XPI.

 xpiHash

    (required) String hash of the XPI that implements this experiment.

    The value is composed of a hash identifier followed by a colon

    followed by the hash value. e.g.

    `sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`

    are the two supported hashing mechanisms. The hash value is the hex

    encoding of the binary hash.

    When the client downloads the XPI for the experiment, it should compare

    the hash of that XPI against this value. If the hashes don't match,

    the client should not install the XPI.

    Clients may also use this hash as a means of determining when an

    experiment's XPI has changed and should be refreshed.

 startTime

    Integer seconds since UNIX epoch that this experiment should

    start. Clients should not start an experiment if *now()* is less than

    this value.

 maxStartTime

    (optional) Integer seconds since UNIX epoch after which this experiment

    should no longer start.

    Some experiments may wish to impose hard deadlines after which no new

    clients should activate the experiment. This property may be used to

    facilitate that.

 endTime

    Integer seconds since UNIX epoch after which this experiment

    should no longer run. Clients should cease an experiment when the current

    time is beyond this value.

 maxActiveSeconds

    Integer seconds defining the max wall time this experiment should be

    active for.

    The client should deactivate the experiment this many seconds after

    initial activation.

    This value only involves wall time, not browser activity or session time.

 appName

    Array of application names this experiment should run on.

    An application name comes from ``nsIXULAppInfo.name``. It is a value

    like ``Firefox``, ``Fennec``, or `B2G`.

    The client should compare its application name against the members of

    this array. If a match is found, the experiment is applicable.

 minVersion

    (optional) String version number of the minimum application version this

    experiment should run on.

    A version number is something like ``27.0.0`` or ``28``.

    The client should compare its version number to this value. If the client's

    version is greater or equal to this version (using a version-aware comparison

    function), the experiment is applicable.

    If this is not specified, there is no lower bound to versions this

    experiment should run on.

 maxVersion

    (optional) String version number of the maximum application version this

    experiment should run on.

    This is similar to ``minVersion`` except it sets the upper bound for

    application versions.

    If the client's version is less than or equal to this version, the

    experiment is applicable.

    If this is not specified, there is no upper bound to versions this

    experiment should run on.

 version

    (optional) Array of application versions this experiment should run on.

    This is similar to ``minVersion`` and ``maxVersion`` except only a

    whitelisted set of specific versions are allowed.

    The client should compare its version to members of this array. If a match

    is found, the experiment is applicable.

 minBuildID

    (optional) String minimum Build ID this experiment should run on.

    Build IDs are values like ``201402261424``.

    The client should perform a string comparison of its Build ID against this

    value. If its value is greater than or equal to this value, the experiment

    is applicable.

 maxBuildID

    (optional) String maximum Build ID this experiment should run on.

    This is similar to ``minBuildID`` except it sets the upper bound

    for Build IDs.

    The client should perform a string comparison of its Build ID against

    this value. If its value is less than or equal to this value, the

    experiment is applicable.

 buildIDs

    (optional) Array of Build IDs this experiment should run on.

    This is similar to ``minBuildID`` and ``maxBuildID`` except only a

    whitelisted set of Build IDs are considered.

    The client should compare its Build ID to members of this array. If a

    match is found, the experiment is applicable.

 os

    (optional) Array of operating system identifiers this experiment should

    run on.

    Values for this array come from ``nsIXULRuntime.OS``.

    The client will compare its operating system identifier to members

    of this array. If a match is found, the experiment is applicable to the

    client.

 channel

    (optional) Array of release channel identifiers this experiment should run

    on.

    The client will compare its channel to members of this array. If a match

    is found, the experiment is applicable.

    If this property is not defined, the client should assume the experiment

    is to run on all channels.

 locale

    (optional) Array of locale identifiers this experiment should run on.

    A locale identifier is a string like ``en-US`` or ``zh-CN`` and is

    obtained by looking at

    ``nsIXULChromeRegistry.getSelectedLocale("global")``.

    The client should compare its locale identifier to members of this array.

    If a match is found, the experiment is applicable.

    If this property is not defined, the client should assume the experiment

    is to run on all locales.

 sample

    (optional) Decimal number indicating the sampling rate for this experiment.

    This will contain a value between ``0.0`` and ``1.0``. The client should

    generate a random decimal between ``0.0`` and ``1.0``. If the randomly

    generated number is less than or equal to the value of this field, the

    experiment is applicable.

 disabled

    (optional) Boolean value indicating whether an experiment is disabled.

    Normally, experiments are deactivated after a certain time has passed or

    after the experiment itself determines it no longer needs to run (perhaps

    it collected sufficient data already).

    This property serves as a backup mechanism to remotely disable an

    experiment before it was scheduled to be disabled. It can be used to

    kill experiments that are found to be doing wrong or bad things or that

    aren't useful.

    If this property is not defined or is false, the client should assume

    the experiment is active and a candidate for activation.

 frozen

    (optional) Boolean value indicating this experiment is frozen and no

    longer accepting new enrollments.

    If a client sees a true value in this field, it should not attempt to

    activate an experiment.

 jsfilter

     (optional) JavaScript code that will be evaluated to determine experiment

     applicability.

     This property contains the string representation of JavaScript code that

     will be evaluated in a sandboxed environment using JavaScript's

     ``eval()``.

     The string is expected to contain the definition of a JavaScript function

     ``filter(context)``. This function receives as its argument an object

     holding application state. See the section below for the definition of

     this object.

     The purpose of this property is to allow experiments to define complex

     rules and logic for evaluating experiment applicability in a manner

     that is privacy conscious and doesn't require the transmission of

     excessive data.

     The return value of this filter indicates whether the experiment is

     applicable. Functions should return true if the experiment is

     applicable.

     If an experiment is not applicable, they should throw an Error whose

     message contains the reason the experiment is not applicable. This

     message may be logged and sent to remote servers, so it should not

     contain private or otherwise sensitive data that wouldn't normally

     be submitted.

     If a falsey (or undefined) value is returned, the client should

     assume the experiment is not applicable.

     If this property is not defined, the client does not consider a custom

     JavaScript filter function when determining whether an experiment is

     applicable.

 JavaScript Filter Context Objects

 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

 The object passed to a ``jsfilter`` ``filter()`` function contains the

 following properties:

 healthReportSubmissionEnabled

    This property contains a boolean indicating whether Firefox Health

    Report has its data submission flag enabled (whether Firefox Health

    Report is sending data to remote servers).

 healthReportPayload

    This property contains the current Firefox Health Report payload.

    The payload format is documented at :ref:`healthreport_dataformat`.

 telemetryPayload

    This property contains the current Telemetry payload.

 The evaluation sandbox for the JavaScript filters may be destroyed

 immediately after ``filter()`` returns. This function should not assume

 async code will finish.

 Experiment Applicability and Client Behavior

 ============================================

 The point of an experiment manifest is to define which experiments are

 available and where and how to run them. This section explains those

 rules in more detail.

 Many of the properties in *Experiment Objects* are related to determining

 whether an experiment should run on a given client. This evaluation is

 performed client side.

 1. Multiple conditions in an experiment

 ---------------------------------------

 If multiple conditions are defined for an experiment, the client should

 combine each condition with a logical *AND*: all conditions must be

 satisfied for an experiment to run. If one condition fails, the experiment

 is not applicable.

 2. Active experiment disappears from manifest

 ---------------------------------------------

 If a specific experiment disappears from the manifest, the client should

 continue conducting an already-active experiment. Furthermore, the

 client should remember what the expiration events were for an experiment

 and honor them.

 The rationale here is that we want to prevent an accidental deletion

 or temporary failure on the server to inadvertantly deactivate

 supposed-to-be-active experiments. We also don't want premature deletion

 of an experiment from the manifest to result in indefinite activation

 periods.

 3. Inactive experiment disappears from manifest

 -----------------------------------------------

 If an inactive but scheduled-to-be-active experiment disappears from the

 manifest, the client should not activate the experiment.

 If that experiment reappears in the manifest, the client should not

 treat that experiment any differently than any other new experiment. Put

 another way, the fact an inactive experiment disappears and then

 reappears should not be significant.

 The rationale here is that server operators should have complete

 control of an inactive experiment up to it's go-live date.

 4. Re-evaluating applicability on manifest refresh

 --------------------------------------------------

 When an experiment manifest is refreshed or updated, the client should

 re-evaluate the applicability of each experiment therein.

 The rationale here is that the server may change the parameters of an

 experiment and want clients to pick those up.

 5. Activating a previously non-applicable experiment

 ----------------------------------------------------

 If the conditions of an experiment change or the state of the client

 changes to allow an experiment to transition from previously

 non-applicable to applicable, the experiment should be activated.

 For example, if a client is running version 28 and the experiment

 initially requires version 29 or above, the client will not mark the

 experiment as applicable. But if the client upgrades to version 29 or if

 the manifest is updated to require 28 or above, the experiment will

 become applicable.

 6. Deactivating a previously active experiment

 ----------------------------------------------

 If the conditions of an experiment change or the state of the client

 changes and an active experiment is no longer applicable, that

 experiment should be deactivated.

 7. Calculation of sampling-based applicability

 ----------------------------------------------

 For calculating sampling-based applicability, the client will associate

 a random value between ``0.0`` and ``1.0`` for each observed experiment

 ID. This random value will be generated the first time sampling

 applicability is evaluated. This random value will be persisted and used

 in future applicability evaluations for this experiment.

 By saving and re-using the value, the client is able to reliably and

 consistently evaluate applicability, even if the sampling threshold

 in the manifest changes.

 Clients should retain the randomly-generated sampling value for

 experiments that no longer appear in a manifest for a period of at least

 30 days. The rationale is that if an experiment disappears and reappears

 from a manifest, the client will not have multiple opportunities to

 generate a random value that satisfies the sampling criteria.

 8. Incompatible version numbers

 -------------------------------

 If a client receives a manifest with a version number that it doesn't

 recognize, it should ignore the manifest.

 9. Usage of old manifests

 -------------------------

 If a client experiences an error fetching a manifest (server not

 available) or if the manifest is corrupt, not readable, or compatible,

 the client may use a previously-fetched (cached) manifest.

 10. Updating XPIs

 -----------------

 If the URL or hash of an active experiment's XPI changes, the client

 should fetch the new XPI, uninstall the old XPI, and install the new

 XPI.

 Examples

 ========

 Here is an example manifest::

    {

      "version": 1,

      "experiments": [

        {

          "id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",

          "xpiURL": "https://experiments.mozilla.org/foo.xpi",

          "xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",

          "startTime": 1393000000,

          "endTime": 1394000000,

          "appName": ["Firefox", "Fennec"],

          "minVersion": "28",

          "maxVersion": "30",

          "os": ["windows", "linux", "osx"],

          "jsfilter": "function filter(context) { return context.healthReportEnabled; }"

        }

      ]

    }