feat(tb-lf): env-cohort subcommand for cross-environment flag analysis#28
Merged
Conversation
Adds `tb-lf env-cohort`, a third cohort analysis command alongside flag-cohort and flag-cohort-stratified. Pivots on environment membership instead of flag-tag value, so it works for the common review-env pattern where a feature is forced-ON in code (`true || flag_check()`) regardless of what the flag's metadata tag says. Calls the new devportal endpoint /traces/env_cohort. Treatment env is where the feature actually runs; control envs are where the feature is OFF / not deployed. --ignore-flags excludes specified flags from the fingerprint (typically the target flag itself, whose tag value is unreliable in the treatment env). Cohorts are paired across envs by matching the remaining flag set. Output uses TREATMENT/CONTROL labels to keep the semantic distinct from the flag-tag-based ON/OFF cohorts. Updated SKILL.md with workflow guidance and interpretation caveats (cross-env noise on top of small-N task variance). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tb-lf env-cohort, a third cohort analysis command alongsideflag-cohortandflag-cohort-stratified. Pivots on environment membership instead of flag-tag value, so it works for the common review-env pattern where a feature is forced-ON in code (true || flag_check()) regardless of what the flag's trace metadata tag says.Depends on devportal PR #263 — the new
/spa_api/ai/traces/env_cohortendpoint must be deployed before this CLI command works against production DevPortal.Why
The existing
flag-cohort-stratifiedrequires both ON and OFF traces (by flag-tag) in the same env. That breaks for review-env-driven feature rollouts:lazy-output-redesign): code forces feature-ON, but trace metadata reports flag-tag OFF for ~99% of traces. No real ON pool.Result: stratified excludes every cohort. The pre-rollout cost/quality validation that's the whole point of cohort analysis can't run.
env-cohortsolves this by using environment as the AB pivot:--treatment-env= the review env where the feature actually runs--control-envs= comma-separated envs where the feature is off / not deployed--ignore-flags= flags whose tag value is unreliable (target feature flag) → excluded from fingerprintCohorts pair traces by matching flag fingerprint across these envs. Larger control-side N (production has thousands vs review env's hundreds) → more reliable comparison than the stratified single-env path could provide.
Usage
Output mirrors
flag-cohort-stratified(cohort fingerprint, flag diff, cohort metrics, delta % cost) but uses TREATMENT/CONTROL labels to keep the semantic distinct from flag-tag-based ON/OFF cohorts.Implementation
Commands::EnvCohortvariant with all the usual stratified params (--min-cohort-size,--max-cohorts,--detail, time range)EnvCohortResponse/EnvCohorttypes in types.rs (treatment/control instead of on/off)print_cohortandprint_cohort_diffhelpers/traces/env_cohortdevportal endpointTest plan
tb-lf env-cohort --helprenders correctly with all params and examples