Feature Request: bucket overrides on a per-request basis

[chrskrchr]

Is your feature request related to a problem? Please describe.
I would love the ability to force myself into a specific variant of a flag on a per-request basis in order to facilitate the testing of flags that are currently either disabled or set to 0%.

Describe the solution you'd like
As a developer, I need the ability to force myself into a specific variant or combination of variants across flags in order to test things in a production environment before we've begun targeting live production traffic with those flags. Ideally, I would be able to specify the overrides via a query param or HTTP header and then pass those overrides to the LD Client during bucketing.

There may be security concerns associated with users being allowed to force bucket themselves into different variants, but the responsibility of addressing those concerns should fall on me as the application developer and not the LD client. For example, I should check the client IP and only respect the requested overrides if the request is from an IP address on my internal network.

Describe alternatives you've considered
Today, I can achieve the above via custom targeting rules on a flag. However, this becomes cumbersome when trying to implement this opt-in behavior in a consistent manner across all flags.

Additional context
Other feature flag providers provide this functionality:

• Optimizely: https://docs.developers.optimizely.com/full-stack/docs/use-forced-bucketing
• Rollout.io (in a limited capacity only for browser and native clients): https://docs.cloudbees.com/docs/cloudbees-feature-flags/latest/debugging/override-view

[eli-darkly]

I think I understand the overall goal here, but not the key details of where and how you want to specify this behavior. Could you provide more of an example of what the desired program flow would be, and how it would affect some example flags? I appreciate that you've provided links to documentation for other companies' products, but they have fairly different models to LD in some ways (I think; as an SDK maintainer, I'm not very familiar with those products) so I'm not sure I see how you intend those to map to our model.

For instance, in our server-side SDKs such as Node, a feature flag evaluation does not involve an HTTP request; it uses only the flag configuration data that's already been received from LD. I'm also not sure how a "per-request override", if it involved having to pass extra parameters to the variation call, would differ from simply having conditional logic in your code that hard-codes the answer you want to get, and skips calling variation (as opposed to having conditional logic that hard-codes passing that answer as an extra parameter). If what you want is instead to set some kind of property in the overall SDK configuration, like "this is the special test user, here are the overrides for them", then that is not per-request.

A feature we've added in some of the other SDKs, which we're planning to add to Node, is a bit different. In those SDKs (such as Go), rather than having the SDK use the flag data from LaunchDarkly as-is except for some particular flag that you've overridden for a single request, we've provided the ability to set the SDK to use an in-app fixture for all flag configurations. Your code can then tell this component exactly what value the flag should return for all evaluations, or set up targeting rules similar to what you can set on the LD dashboard. This is mainly designed for programmatic testing, as in unit tests, where the test code needs to be able to completely specify the desired inputs without referring to an outside service. I'm not sure if it would be applicable to your use case.

[eli-darkly]

If the kind of test fixture I mentioned isn't applicable, then it sounds like this isn't necessarily something specific to the Node SDK but more of a product-wide feature request. If so, I'll file it in the same list where feature requests sent to support@launchdarkly.com normally go and I'll close this issue, but then we would comment on it to let you know if/when we implement something.

[chrskrchr]

Hi Eli,
Thanks for the quick reply!

First, you're absolutely correct that this is more of a product-wide feature request. I only entered the request against the Node SDK since this is the first SDK where we'd take advantage of the requested functionality.

Second, you're also correct that we could implement this functionality ourselves via conditional logic around our flags or via custom audience targeting rules on the flags themselves. However, this becomes a pain if we want this override functionality to be available for use on ANY feature flag in a consistent manner. We'd essentially need to build a wrapper around the LD client that handled the logic for checking if an override was present, otherwise falling back on .variation() to do the its bucketing. Our wrapper would also need to dig around in flag metadata to pluck out the correct return value for the overridden variant as needed.

If the LD client provided this functionality, the flow would be as follows:

1. When a request hits our Node app, we check for a query param or HTTP header on the request that specifies the flags+variants the user wants to override for that request, e.g. ?ldOverride=example-flag-1:2&ldOverride=example-flag-2:5 to indicate "force variant 2 for example-flag-1" and `"force variant 5 for example-flag-2"
2. We would pass this map of overrides alongside the user context when calling ldClient.allFlagsState() or ldClient.variation()
   1. When executing its bucketing logic, the LD client would first look in the provided map of overrides to see if an override had been provided for the flag, otherwise it would fall back on its standard hashing/bucketing logic to determine the variant. It would then look up the value for the variant the same as it does today.

Does that make sense? This makes handling overrides a lot easier from our POV because we build the override map in nearly the same place where we're constructing the user context today.

The test fixture functionality you described sounds great for unit tests, but not quite what we want for per-request overrides at runtime.

[eli-darkly]

Ah— you mean "per-request" as in per request to your application, not a request to LaunchDarkly.

From this description, it sounds like this would be a feature entirely implemented within the SDK, not involving the LaunchDarkly service itself at all. However, I think we will still need to discuss it at a broader level rather than as a Node SDK feature, because we'll need to come up with a design that we can keep relatively consistent across platforms.

One concern is that we probably would want to avoid adding more parameters to variation() or allFlagsState(), because optional parameters are not supported in many languages and can cause unwanted syntactic ambiguity.

An alternative approach might be to have the overrides actually be part of the user object. If it's the case that within any given request lifecycle, the overrides should be in place for all evaluations against that user (that is, if you would not want flag F to return regular value V1 within one code path, but override value V2 within another, for this request) then that might be feasible. However, if you ever do things with the user JSON other than passing it to evaluation methods (like passing it to front-end JS code) then the presence of that stuff might be unwanted.

[chrskrchr]

Ah— you mean "per-request" as in per request to your application, not a request to LaunchDarkly.

Yes! Sorry for the confusion there.

I think that adding this to the user context object would be a great solution - maybe with a reserved name like $overrides / __overrides or something similar that's very unlikely to conflict with any existing keys in the context.

the overrides should be in place for all evaluations against that user (that is, if you would not want flag F to return regular value V1 within one code path, but override value V2 within another, for this request)

Correct - we'd want these overrides to apply consistently for all evaluations that occur in the context of that request.

[bwoskow-ld]

Hi @chrskrchr ,

Thanks for the feature request! I have added it to our internal feature request tracker so that our product managers can assess the priority. While we can't commit to implementing this feature or any particular delivery timeframes, I can at least ensure you that the request is filed where the right people can see it.

I'm not sure how you currently manage your LaunchDarkly environments, or if you'd be open to restructuring your environments in a different manner... but our best practices are such that every developer on your team would have their own environment. This best practice allows each developer to make changes to their own environment however they see fit -- for example, to test a particular variation of a particular flag -- without impacting any other developers or environments. If you follow this best practice, you may find that you no longer have a need for your SDK feature request.

As this feature request isn't specific to the Node server-side SDK, I'm going to close the request.

Cheers,
@bwoskow-ld

[chrskrchr]

Thanks, @bwoskow-ld! Glad to hear it made it into your tracker.

but our best practices are such that every developer on your team would have their own environment

I could see how this would work on a dev's local machine where the dev injected their personal SDK key into the app via an env var (or something similar), but how would this help in a non-local environment? i.e., a test/staging/pre-prod type environment that's configured to use a specific LD SDK key.

This feature request is really targeted at non-engineers that:

1. aren't running the app on their local machine
2. don't necessarily have access to modify our LD flag config (i.e., where they could temporarily adjust a flag's targeting rules to enable it for themselves)
3. want to force themselves into a specific variant of a flag to do things like:
   1. demo a dark launched feature to execs
   2. test a specific configuration of flags
   3. etc...

[bwoskow-ld]

Good question. For the reasons you pointed out my suggestion is targeting local development environments and not demo/test purposes.

For configuring a flag variation for a particular user, especially on non-local environments, we recommend that customers target individual users on the desired environments. This way no code changes are needed and no one (other than the specified users) are affected by the targeting change. As you pointed out, in order for this feature to be utilized as intended, your colleagues will need sufficient access so they can set the flag state. If you don't want to grant them full writer access and if you are on an enterprise account, you can use custom roles to create a custom permission level where, for example, users can modify flags' targeting rules but not other aspects.

In the end this will be a much cleaner solution as it won't require any code changes (e.g. to override variations for a particular user) and you'll be ensured that the desired change will be isolated to that particular user on that particular environment.