michael@0: .. _experiments_manifests:
michael@0: 
michael@0: =====================
michael@0: Experiments Manifests
michael@0: =====================
michael@0: 
michael@0: *Experiments Manifests* are documents that describe the set of active
michael@0: experiments a client may run.
michael@0: 
michael@0: *Experiments Manifests* are fetched periodically by clients. When
michael@0: fetched, clients look at the experiments within the manifest and
michael@0: determine which experiments are applicable. If an experiment is
michael@0: applicable, the client may download and start the experiment.
michael@0: 
michael@0: Manifest Format
michael@0: ===============
michael@0: 
michael@0: Manifests are JSON documents where the main element is an object.
michael@0: 
michael@0: The *schema* of the object is versioned and defined by the presence
michael@0: of a top-level ``version`` property, whose integer value is the
michael@0: schema version used by that manifest. Each version is documented
michael@0: in the sections below.
michael@0: 
michael@0: Version 1
michael@0: ---------
michael@0: 
michael@0: Version 1 is the original manifest format.
michael@0: 
michael@0: The following properties may exist in the root object:
michael@0: 
michael@0: experiments
michael@0:    An array of objects describing candidate experiments. The format of
michael@0:    these objects is documented below.
michael@0: 
michael@0:    An array is used to create an explicit priority of experiments.
michael@0:    Experiments listed at the beginning of the array take priority over
michael@0:    experiments that follow.
michael@0: 
michael@0: Experiments Objects
michael@0: ^^^^^^^^^^^^^^^^^^^
michael@0: 
michael@0: Each object in the ``experiments`` array may contain the following
michael@0: properties:
michael@0: 
michael@0: id
michael@0:    (required) String identifier of this experiment. The identifier should
michael@0:    be treated as opaque by clients. It is used to uniquely identify an
michael@0:    experiment for all of time.
michael@0: 
michael@0: xpiURL
michael@0:    (required) String URL of the XPI that implements this experiment.
michael@0: 
michael@0:    If the experiment is activated, the client will download and install this
michael@0:    XPI.
michael@0: 
michael@0: xpiHash
michael@0:    (required) String hash of the XPI that implements this experiment.
michael@0: 
michael@0:    The value is composed of a hash identifier followed by a colon
michael@0:    followed by the hash value. e.g.
michael@0:    `sha1:f677428b9172e22e9911039aef03f3736e7f78a7`. `sha1` and `sha256`
michael@0:    are the two supported hashing mechanisms. The hash value is the hex
michael@0:    encoding of the binary hash.
michael@0: 
michael@0:    When the client downloads the XPI for the experiment, it should compare
michael@0:    the hash of that XPI against this value. If the hashes don't match,
michael@0:    the client should not install the XPI.
michael@0: 
michael@0:    Clients may also use this hash as a means of determining when an
michael@0:    experiment's XPI has changed and should be refreshed.
michael@0: 
michael@0: startTime
michael@0:    Integer seconds since UNIX epoch that this experiment should
michael@0:    start. Clients should not start an experiment if *now()* is less than
michael@0:    this value.
michael@0: 
michael@0: maxStartTime
michael@0:    (optional) Integer seconds since UNIX epoch after which this experiment
michael@0:    should no longer start.
michael@0: 
michael@0:    Some experiments may wish to impose hard deadlines after which no new
michael@0:    clients should activate the experiment. This property may be used to
michael@0:    facilitate that.
michael@0: 
michael@0: endTime
michael@0:    Integer seconds since UNIX epoch after which this experiment
michael@0:    should no longer run. Clients should cease an experiment when the current
michael@0:    time is beyond this value.
michael@0: 
michael@0: maxActiveSeconds
michael@0:    Integer seconds defining the max wall time this experiment should be
michael@0:    active for.
michael@0: 
michael@0:    The client should deactivate the experiment this many seconds after
michael@0:    initial activation.
michael@0: 
michael@0:    This value only involves wall time, not browser activity or session time.
michael@0: 
michael@0: appName
michael@0:    Array of application names this experiment should run on.
michael@0: 
michael@0:    An application name comes from ``nsIXULAppInfo.name``. It is a value
michael@0:    like ``Firefox``, ``Fennec``, or `B2G`.
michael@0: 
michael@0:    The client should compare its application name against the members of
michael@0:    this array. If a match is found, the experiment is applicable.
michael@0: 
michael@0: minVersion
michael@0:    (optional) String version number of the minimum application version this
michael@0:    experiment should run on.
michael@0: 
michael@0:    A version number is something like ``27.0.0`` or ``28``.
michael@0: 
michael@0:    The client should compare its version number to this value. If the client's
michael@0:    version is greater or equal to this version (using a version-aware comparison
michael@0:    function), the experiment is applicable.
michael@0: 
michael@0:    If this is not specified, there is no lower bound to versions this
michael@0:    experiment should run on.
michael@0: 
michael@0: maxVersion
michael@0:    (optional) String version number of the maximum application version this
michael@0:    experiment should run on.
michael@0: 
michael@0:    This is similar to ``minVersion`` except it sets the upper bound for
michael@0:    application versions.
michael@0: 
michael@0:    If the client's version is less than or equal to this version, the
michael@0:    experiment is applicable.
michael@0: 
michael@0:    If this is not specified, there is no upper bound to versions this
michael@0:    experiment should run on.
michael@0: 
michael@0: version
michael@0:    (optional) Array of application versions this experiment should run on.
michael@0: 
michael@0:    This is similar to ``minVersion`` and ``maxVersion`` except only a
michael@0:    whitelisted set of specific versions are allowed.
michael@0: 
michael@0:    The client should compare its version to members of this array. If a match
michael@0:    is found, the experiment is applicable.
michael@0: 
michael@0: minBuildID
michael@0:    (optional) String minimum Build ID this experiment should run on.
michael@0: 
michael@0:    Build IDs are values like ``201402261424``.
michael@0: 
michael@0:    The client should perform a string comparison of its Build ID against this
michael@0:    value. If its value is greater than or equal to this value, the experiment
michael@0:    is applicable.
michael@0: 
michael@0: maxBuildID
michael@0:    (optional) String maximum Build ID this experiment should run on.
michael@0: 
michael@0:    This is similar to ``minBuildID`` except it sets the upper bound
michael@0:    for Build IDs.
michael@0: 
michael@0:    The client should perform a string comparison of its Build ID against
michael@0:    this value. If its value is less than or equal to this value, the
michael@0:    experiment is applicable.
michael@0: 
michael@0: buildIDs
michael@0:    (optional) Array of Build IDs this experiment should run on.
michael@0: 
michael@0:    This is similar to ``minBuildID`` and ``maxBuildID`` except only a
michael@0:    whitelisted set of Build IDs are considered.
michael@0: 
michael@0:    The client should compare its Build ID to members of this array. If a
michael@0:    match is found, the experiment is applicable.
michael@0: 
michael@0: os
michael@0:    (optional) Array of operating system identifiers this experiment should
michael@0:    run on.
michael@0: 
michael@0:    Values for this array come from ``nsIXULRuntime.OS``.
michael@0: 
michael@0:    The client will compare its operating system identifier to members
michael@0:    of this array. If a match is found, the experiment is applicable to the
michael@0:    client.
michael@0: 
michael@0: channel
michael@0:    (optional) Array of release channel identifiers this experiment should run
michael@0:    on.
michael@0: 
michael@0:    The client will compare its channel to members of this array. If a match
michael@0:    is found, the experiment is applicable.
michael@0: 
michael@0:    If this property is not defined, the client should assume the experiment
michael@0:    is to run on all channels.
michael@0: 
michael@0: locale
michael@0:    (optional) Array of locale identifiers this experiment should run on.
michael@0: 
michael@0:    A locale identifier is a string like ``en-US`` or ``zh-CN`` and is
michael@0:    obtained by looking at
michael@0:    ``nsIXULChromeRegistry.getSelectedLocale("global")``.
michael@0: 
michael@0:    The client should compare its locale identifier to members of this array.
michael@0:    If a match is found, the experiment is applicable.
michael@0: 
michael@0:    If this property is not defined, the client should assume the experiment
michael@0:    is to run on all locales.
michael@0: 
michael@0: sample
michael@0:    (optional) Decimal number indicating the sampling rate for this experiment.
michael@0: 
michael@0:    This will contain a value between ``0.0`` and ``1.0``. The client should
michael@0:    generate a random decimal between ``0.0`` and ``1.0``. If the randomly
michael@0:    generated number is less than or equal to the value of this field, the
michael@0:    experiment is applicable.
michael@0: 
michael@0: disabled
michael@0:    (optional) Boolean value indicating whether an experiment is disabled.
michael@0: 
michael@0:    Normally, experiments are deactivated after a certain time has passed or
michael@0:    after the experiment itself determines it no longer needs to run (perhaps
michael@0:    it collected sufficient data already).
michael@0: 
michael@0:    This property serves as a backup mechanism to remotely disable an
michael@0:    experiment before it was scheduled to be disabled. It can be used to
michael@0:    kill experiments that are found to be doing wrong or bad things or that
michael@0:    aren't useful.
michael@0: 
michael@0:    If this property is not defined or is false, the client should assume
michael@0:    the experiment is active and a candidate for activation.
michael@0: 
michael@0: frozen
michael@0:    (optional) Boolean value indicating this experiment is frozen and no
michael@0:    longer accepting new enrollments.
michael@0: 
michael@0:    If a client sees a true value in this field, it should not attempt to
michael@0:    activate an experiment.
michael@0: 
michael@0: jsfilter
michael@0:     (optional) JavaScript code that will be evaluated to determine experiment
michael@0:     applicability.
michael@0: 
michael@0:     This property contains the string representation of JavaScript code that
michael@0:     will be evaluated in a sandboxed environment using JavaScript's
michael@0:     ``eval()``.
michael@0: 
michael@0:     The string is expected to contain the definition of a JavaScript function
michael@0:     ``filter(context)``. This function receives as its argument an object
michael@0:     holding application state. See the section below for the definition of
michael@0:     this object.
michael@0: 
michael@0:     The purpose of this property is to allow experiments to define complex
michael@0:     rules and logic for evaluating experiment applicability in a manner
michael@0:     that is privacy conscious and doesn't require the transmission of
michael@0:     excessive data.
michael@0: 
michael@0:     The return value of this filter indicates whether the experiment is
michael@0:     applicable. Functions should return true if the experiment is
michael@0:     applicable.
michael@0: 
michael@0:     If an experiment is not applicable, they should throw an Error whose
michael@0:     message contains the reason the experiment is not applicable. This
michael@0:     message may be logged and sent to remote servers, so it should not
michael@0:     contain private or otherwise sensitive data that wouldn't normally
michael@0:     be submitted.
michael@0: 
michael@0:     If a falsey (or undefined) value is returned, the client should
michael@0:     assume the experiment is not applicable.
michael@0: 
michael@0:     If this property is not defined, the client does not consider a custom
michael@0:     JavaScript filter function when determining whether an experiment is
michael@0:     applicable.
michael@0: 
michael@0: JavaScript Filter Context Objects
michael@0: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
michael@0: 
michael@0: The object passed to a ``jsfilter`` ``filter()`` function contains the
michael@0: following properties:
michael@0: 
michael@0: healthReportSubmissionEnabled
michael@0:    This property contains a boolean indicating whether Firefox Health
michael@0:    Report has its data submission flag enabled (whether Firefox Health
michael@0:    Report is sending data to remote servers).
michael@0: 
michael@0: healthReportPayload
michael@0:    This property contains the current Firefox Health Report payload.
michael@0: 
michael@0:    The payload format is documented at :ref:`healthreport_dataformat`.
michael@0: 
michael@0: telemetryPayload
michael@0:    This property contains the current Telemetry payload.
michael@0: 
michael@0: The evaluation sandbox for the JavaScript filters may be destroyed
michael@0: immediately after ``filter()`` returns. This function should not assume
michael@0: async code will finish.
michael@0: 
michael@0: Experiment Applicability and Client Behavior
michael@0: ============================================
michael@0: 
michael@0: The point of an experiment manifest is to define which experiments are
michael@0: available and where and how to run them. This section explains those
michael@0: rules in more detail.
michael@0: 
michael@0: Many of the properties in *Experiment Objects* are related to determining
michael@0: whether an experiment should run on a given client. This evaluation is
michael@0: performed client side.
michael@0: 
michael@0: 1. Multiple conditions in an experiment
michael@0: ---------------------------------------
michael@0: 
michael@0: If multiple conditions are defined for an experiment, the client should
michael@0: combine each condition with a logical *AND*: all conditions must be
michael@0: satisfied for an experiment to run. If one condition fails, the experiment
michael@0: is not applicable.
michael@0: 
michael@0: 2. Active experiment disappears from manifest
michael@0: ---------------------------------------------
michael@0: 
michael@0: If a specific experiment disappears from the manifest, the client should
michael@0: continue conducting an already-active experiment. Furthermore, the
michael@0: client should remember what the expiration events were for an experiment
michael@0: and honor them.
michael@0: 
michael@0: The rationale here is that we want to prevent an accidental deletion
michael@0: or temporary failure on the server to inadvertantly deactivate
michael@0: supposed-to-be-active experiments. We also don't want premature deletion
michael@0: of an experiment from the manifest to result in indefinite activation
michael@0: periods.
michael@0: 
michael@0: 3. Inactive experiment disappears from manifest
michael@0: -----------------------------------------------
michael@0: 
michael@0: If an inactive but scheduled-to-be-active experiment disappears from the
michael@0: manifest, the client should not activate the experiment.
michael@0: 
michael@0: If that experiment reappears in the manifest, the client should not
michael@0: treat that experiment any differently than any other new experiment. Put
michael@0: another way, the fact an inactive experiment disappears and then
michael@0: reappears should not be significant.
michael@0: 
michael@0: The rationale here is that server operators should have complete
michael@0: control of an inactive experiment up to it's go-live date.
michael@0: 
michael@0: 4. Re-evaluating applicability on manifest refresh
michael@0: --------------------------------------------------
michael@0: 
michael@0: When an experiment manifest is refreshed or updated, the client should
michael@0: re-evaluate the applicability of each experiment therein.
michael@0: 
michael@0: The rationale here is that the server may change the parameters of an
michael@0: experiment and want clients to pick those up.
michael@0: 
michael@0: 5. Activating a previously non-applicable experiment
michael@0: ----------------------------------------------------
michael@0: 
michael@0: If the conditions of an experiment change or the state of the client
michael@0: changes to allow an experiment to transition from previously
michael@0: non-applicable to applicable, the experiment should be activated.
michael@0: 
michael@0: For example, if a client is running version 28 and the experiment
michael@0: initially requires version 29 or above, the client will not mark the
michael@0: experiment as applicable. But if the client upgrades to version 29 or if
michael@0: the manifest is updated to require 28 or above, the experiment will
michael@0: become applicable.
michael@0: 
michael@0: 6. Deactivating a previously active experiment
michael@0: ----------------------------------------------
michael@0: 
michael@0: If the conditions of an experiment change or the state of the client
michael@0: changes and an active experiment is no longer applicable, that
michael@0: experiment should be deactivated.
michael@0: 
michael@0: 7. Calculation of sampling-based applicability
michael@0: ----------------------------------------------
michael@0: 
michael@0: For calculating sampling-based applicability, the client will associate
michael@0: a random value between ``0.0`` and ``1.0`` for each observed experiment
michael@0: ID. This random value will be generated the first time sampling
michael@0: applicability is evaluated. This random value will be persisted and used
michael@0: in future applicability evaluations for this experiment.
michael@0: 
michael@0: By saving and re-using the value, the client is able to reliably and
michael@0: consistently evaluate applicability, even if the sampling threshold
michael@0: in the manifest changes.
michael@0: 
michael@0: Clients should retain the randomly-generated sampling value for
michael@0: experiments that no longer appear in a manifest for a period of at least
michael@0: 30 days. The rationale is that if an experiment disappears and reappears
michael@0: from a manifest, the client will not have multiple opportunities to
michael@0: generate a random value that satisfies the sampling criteria.
michael@0: 
michael@0: 8. Incompatible version numbers
michael@0: -------------------------------
michael@0: 
michael@0: If a client receives a manifest with a version number that it doesn't
michael@0: recognize, it should ignore the manifest.
michael@0: 
michael@0: 9. Usage of old manifests
michael@0: -------------------------
michael@0: 
michael@0: If a client experiences an error fetching a manifest (server not
michael@0: available) or if the manifest is corrupt, not readable, or compatible,
michael@0: the client may use a previously-fetched (cached) manifest.
michael@0: 
michael@0: 10. Updating XPIs
michael@0: -----------------
michael@0: 
michael@0: If the URL or hash of an active experiment's XPI changes, the client
michael@0: should fetch the new XPI, uninstall the old XPI, and install the new
michael@0: XPI.
michael@0: 
michael@0: Examples
michael@0: ========
michael@0: 
michael@0: Here is an example manifest::
michael@0: 
michael@0:    {
michael@0:      "version": 1,
michael@0:      "experiments": [
michael@0:        {
michael@0:          "id": "da9d7f4f-f3f9-4f81-bacd-6f0626ffa360",
michael@0:          "xpiURL": "https://experiments.mozilla.org/foo.xpi",
michael@0:          "xpiHash": "sha1:cb1eb32b89d86d78b7326f416cf404548c5e0099",
michael@0:          "startTime": 1393000000,
michael@0:          "endTime": 1394000000,
michael@0:          "appName": ["Firefox", "Fennec"],
michael@0:          "minVersion": "28",
michael@0:          "maxVersion": "30",
michael@0:          "os": ["windows", "linux", "osx"],
michael@0:          "jsfilter": "function filter(context) { return context.healthReportEnabled; }"
michael@0:        }
michael@0:      ]
michael@0:    }