Feature flipping support for experiment addons

[lmorchard]

A notion for testing features in an experimental addon is to ship lots of alternate code branches and flip conditionally between them. We've done this a lot on Django sites at Mozilla with django-waffle.

What might be interesting is to have experiment addons talk to the Idea Town addon (e.g. via nsIObserverService) to obtain feature flipping configuration data - which in turn can come from the Idea Town server based on the authenticated user. This could let us do things like A/B testing, incremental roll out, etc.

Breaking this out into subtasks off the top of my head:

• Data model for features defined for a given experiment.
• Data model for feature flags for a particular user experiment installation
• Server-side API resource for a user's experiment feature flags
• Admin integration for managing features, both for defining features and for individual user/experiment flag state
• Any logic needed to come up with the feature flags for a particular user installation (e.g. segmentation, incremental rollout, etc. django-waffle could be an inspiration)
• Metrics integration so that we can measure things based on the feature state for any given user
• Example experiment addon code to show how to fetch feature flag data from the main addon and how to use it in conditionally enabling features

[lmorchard]

Also implied here is that we'd have per-user, per-experiment feature profiles on the server side. Could just be a blob of "name": {true,false} pairs served up as JSON from the API. Flags, switches, and samples seem handy in django-waffle as logic for controlling these things - but it's worth thinking more about.

[jaredhirsch]

👍

If Idea Town handles segmenting users, that information could be added to metrics packets that are sent over from individual experiments. Also allows individual product teams to not think about stuff like how big the sample population needs to be for a given test.

[lmorchard]

In a status meeting, there was talk of experiments that don't necessarily have a start and an end. Instead, an experiment might lead to a release in Firefox proper, but then continue on as an experiment as new features & tweaks roll in.

Might be interesting to have feature flipping as a way to manage that? That is, the "released" version could be much the same as the "experimental" version, just with all the experimental things and Idea Town metrics turned off.

[lmorchard]

Doing some checklist thinking in the issue description in advance of possibly creating new issues to capture the work involved in this overall thing