Making addons a better path for experimention: List of ideas to change in firefox to eliminate needing privileged shield utils

[gregglind]

rev: 3

Current Assumptions about Experiment Addons that need revision

1. They need to "pack their own lunch". Instead, some change to code in the tree, including some possible new "mozilla-only" permissions or APIs could help.
2. All Experiments run in Firefox. It may be that we build addons that are designed to run on other browsers, and could benefit from CI, testing, and instrumentation.
3. They should tie to existing Telemetry (including client Id).
4. They should "store nothing" in firefox between runs.
5. They should "leave no trace" after finishing.

Help me to stop avoiding firefox.

Lifecycle Event Challenges

First install vs. Second install matters for experiments (and probably other addons)

• runtime.onInstall 'details' has a little more details (a 'firefox upgraded', 'is this first run for this addon', maybe other things)
• runtime.onStartup could gets the 'details' from runtime.onInstall, and be called every time.
• runtime.onStartup() doesn't run in the "incognito" mode, see: mdn: runtime.onStartup docs

Study Endings and Uninstall is hard to untangle.

• uninstall / shutdown for addons can get messages about the 'full reason' so that normandy vs about:addons click, etc.
• handle the 'removed by rm from profile' case, if possible

Sending Data

Telemetry requires privileges to

• fill the payload (clientId, other payload data)

• send with validation. (using TelemetryClient repsects the prefs, formats data, etc.)

• semi-permission for telemetry, telmetry client id

• un-privileged telemetry equivalent for when addons are running on other browsers

Reading and Writing Preferences

Studies care about:

• permissions for telemetry, shield, pioneer, etc.
• prefs (or other mechanisms) set to simulate different conditions, such as "expired".
• storing data across sessions (notably firstRun / secondRun so that we can know if this is a fresh install or not, so that we can decide a variation and send appropriate pings.

Persistent Data and Setup issues

• to eliminate prefs, have a method (in web-ext cli or elsewhere) for setting storage.local 'initial values'. The allows simiulating 2nd run, force a variation to be selected, etc.

• "local prefs" or "prefs for this addon" feels like something that would be nice to add here, if we are sticking with prefs. In web-ext configs, allow "prefs" AND "addonPrefs". Having to figure out the name for those prefs as a mangle of the addonId is tedious!

QA / CI

As we discussed, there are some possible beneficial tests that would help to lower the risk profile of these addons

• does it break the updater?
• does using it break performance (esp compared to an empty addon)
• are there templates and tests that would be beneficial to addons we make for other browsers, or other common addons that users might install on Fx?
• ci for signing and building
• mach try-with-addon or npm try with addon

@aswan, @rhelmer, @motin @biancadanforth thoughts on if we should turn some of this into true bugs / patches.

[gregglind]

Note: "onStartup isn't fired in incognito mode". This also is an issue :)

https://developer.mozilla.org/en-US/Add-ons/WebExtensions/API/runtime/onStartup

[jonathanKingston]

1. They need to "pack their own lunch". Instead, some change to code in the tree, including some possible new "mozilla-only" permissions or APIs could help.

This would be great, especially if we can safely check that this is only Mozilla signed addons. The mozaddons permission should protect against this and we could security review it once. For example with the HTTP DNS work I did, there is a narrow API that reads a JSON API. This allows the experiment to only set known states that are bundled with the addon.

The same is also true for the notifications API that I also proposed into central this was inspired by the work in the extension too.

to eliminate prefs, have a method (in web-ext cli or elsewhere) for setting storage.local 'initial values'. The allows simiulating 2nd run, force a variation to be selected, etc.

This wouldn't have been suitable for the DNS study as we want follow up ones, however having the ability to easily set storage certainly would help. In a way the storage API I proposed above should likely default to using storage.local instead of setting prefs.

"local prefs" or "prefs for this addon" feels like something that would be nice to add here, if we are sticking with prefs. In web-ext configs, allow "prefs" AND "addonPrefs". Having to figure out the name for those prefs as a mangle of the addonId is tedious!

Yeah perhaps just default a known prefix to the prefs, however if it's specific to this addon what is the benefit over storage? (unless you expect the addon to be used like HTTP DNS, differing version access the previous versions prefs?)

uninstall / shutdown for addons can get messages about the 'full reason' so that normandy vs about:addons click, etc.

My understanding here is we don't want Shield tightly coupled to Normandy however perhaps we could provide an "unknown" or "server-initiated-state" for all non chrome UI addon uninstalls.

are there templates and tests that would be beneficial to addons we make for other browsers, or other common addons that users might install on Fx?

Providing a simpler way to throw it at ./mach try as a system addon certainly would solve most of the issues.

[gregglind]

Yeah perhaps just default a known prefix to the prefs, however if it's specific to this addon what is the benefit over storage? (unless you expect the addon to be used like HTTP DNS, differing version access the previous versions prefs?)

The benefit over storage is that it can be set / read from the 'outside', and before the addon exists. If storage grew those features / paved that path, that also works.

[ianb]

to eliminate prefs, have a method (in web-ext cli or elsewhere) for setting storage.local 'initial values'. The allows simiulating 2nd run, force a variation to be selected, etc.

This would also be beneficial to me – currently in Screenshots and other add-ons I am resorting to dynamically creating JS files with these settings.

"local prefs" or "prefs for this addon" feels like something that would be nice to add here, if we are sticking with prefs. In web-ext configs, allow "prefs" AND "addonPrefs". Having to figure out the name for those prefs as a mangle of the addonId is tedious!

Do you want to do about:config prefs, or add-on preferences generally? I guess the benefit of about:config prefs is you can offer the user or developer some transparency, and you have a built-in UI, but you aren't trying to offer those settings to the general public. Otherwise the ability to set storage from web-ext plus a library for managing the settings page would seem sufficient? (There's also some open questions about how to manage using libraries in an extension, but for now there are some lazy ways to approach it.)

does using it break performance (esp compared to an empty addon)
mach try-with-addon or npm try with addon

I'm kind of inclined for this to be a script that copies the built add-on into a firefox tree, makes sure the add-on will install, and then commits the result. By doing it as file copies and a commit (as opposed to a mach command) it will work with Try and Talos without any new code in the build system.

This could be implemented as a Node script that you install into the extension (assuming you are creating a separate repository for the extension with its own package.json).

[aswan]

Before getting into the details, a meta-comment: this seems like a collection of small issues, not anything major or architectural. I don't mean that they aren't important, just want to make sure I'm not overlooking some larger architectural issue, please correct me if I did. I think some of these may be solvable with just some additional documentation/communication. But if there's some not common theme that I missed, could we have separate issues for each of the topics here? I fear that the unthreaded conversation view here is going to make it all to easy for some of these issues to fall through the cracks.

I have more to say about some of the individual items but a few assorted other thoughts:

First install vs. Second install matters for experiments (and probably other addons)

This could use some elaboration. What's the distinction? And perhaps I'm reading this too literally but I can't square this with:

They should "store nothing" in firefox between runs.

As for runtime.onStartup and runtime.onInstalled, I think the onStartup event is mostly useful in Chrome extensions since Chrome does event pages. I'm curious how you're using it in Firefox. As for this:

runtime.onStartup could gets the 'details' from runtime.onInstall, and be called every time.

I'm confused, the details passed to an onInstalled handler are about the install event, I don't know what we would pass during a regular browser startup. Perhaps it would be easier if you could explain the problem you're trying to solve, separately from the proposed solution?

runtime.onStartup() doesn't run in the "incognito" mode, see: mdn: runtime.onStartup docs

I think this is a documentation bug, Firefox doesn't have any special handling for extensions in private browsing mode (https://bugzilla.mozilla.org/show_bug.cgi?id=1460738)

As for prefs, again a clear statement of the problem that extension/study authors need to solve, independent of proposed solutions, would be a big help here. e.g., "we need to be able to write data from Normandy and read it from an extension" without any assumptions about whether the implementation should be prefs, or the extension storage API, or whatever.

[biancadanforth]

At @MattGMoz's request, here is my take on the top 3 most important improvements to the Shield study development process:

1. Get a working, well-tested WEE template.
   1. Cull old/redundant issues.
   2. Identify existing challenges, file new issues with context, prior discussions, etc. with a one-concept-per-issue approach.
   3. Triage those issues with P1, P2, P3 flags.
2. P1 issue: Centralize documentation (a la Read the Docs, like Normandy has) with sections for different roles involved in the study (developer, QA, analyst, survey writer, PM, ...).
   1. Developer/QA docs should include general behaviors all Shield studies should have, including:
      1. When a study is enabled/disabled/installed/uninstalled by the user in about:addons.
      2. How the study should behave in Private Browsing mode (we should get an answer from the Data/Privacy/Legal folks and link to that answer).
      3. To what extent these behaviors are already built-in to the template/utils, and to what extent the developer has to do something about it.
3. P1 issue: Integrate regression testing in Firefox with CI, either using a Taskcluster integration or Travis/Circle/non-Mozilla-specific CI, with a flag for more complete in-tree testing to be done less frequently.
   1. Identify what subset of tests are most important (ex: browser chrome Mochitests); which should be per-commit, and which should be checked at all, but less frequently?
      1. Do they all already exist?
      2. Do we need to write some of our own?
         1. Is there a test to confirm we are not breaking the updater?
         2. Is there a test to confirm study expiration (or other persistent state) works between restarts?
   2. For the less frequent, more complete in-tree testing, establish a set of go/no-go requirements to provide some indication to developers who are less familiar with Mozilla-specific testing.
      1. Could there be a green/red indicator?

Edit: I should mention I am working 100% time on another project, so I will not be heavily involved in the day-to-day of this transformation (at least for the next two quarters), but of course I am very interested in the high-level discussions.

[gregglind]

All discussion of ENDINGS goes to #246, not here.

I will try break out the issues around storage, and what a study "needs to do" into separate issues.

[motin]

Closing this in favor of #278, which includes proposed and ongoing engineering efforts towards a simplified experimentation process.