Skip to content

feat(tb-lf): env-cohort subcommand for cross-environment flag analysis#28

Merged
trogulja merged 2 commits into
mainfrom
feat/tb-lf-env-cohort
May 4, 2026
Merged

feat(tb-lf): env-cohort subcommand for cross-environment flag analysis#28
trogulja merged 2 commits into
mainfrom
feat/tb-lf-env-cohort

Conversation

@trogulja
Copy link
Copy Markdown
Collaborator

@trogulja trogulja commented May 4, 2026

Summary

Adds tb-lf env-cohort, a third cohort analysis command alongside flag-cohort and flag-cohort-stratified. Pivots on environment membership instead of flag-tag value, so it works for the common review-env pattern where a feature is forced-ON in code (true || flag_check()) regardless of what the flag's trace metadata tag says.

Depends on devportal PR #263 — the new /spa_api/ai/traces/env_cohort endpoint must be deployed before this CLI command works against production DevPortal.

Why

The existing flag-cohort-stratified requires both ON and OFF traces (by flag-tag) in the same env. That breaks for review-env-driven feature rollouts:

  • Review env (e.g. lazy-output-redesign): code forces feature-ON, but trace metadata reports flag-tag OFF for ~99% of traces. No real ON pool.
  • Production env: feature not deployed, all flag-tag OFF. No ON pool either.

Result: stratified excludes every cohort. The pre-rollout cost/quality validation that's the whole point of cohort analysis can't run.

env-cohort solves this by using environment as the AB pivot:

  • --treatment-env = the review env where the feature actually runs
  • --control-envs = comma-separated envs where the feature is off / not deployed
  • --ignore-flags = flags whose tag value is unreliable (target feature flag) → excluded from fingerprint

Cohorts pair traces by matching flag fingerprint across these envs. Larger control-side N (production has thousands vs review env's hundreds) → more reliable comparison than the stratified single-env path could provide.

Usage

tb-lf env-cohort \
  --treatment-env lazy-output-redesign \
  --control-envs production,latest \
  --ignore-flags aiAgentLazyOutput \
  --from 14d --to today

Output mirrors flag-cohort-stratified (cohort fingerprint, flag diff, cohort metrics, delta % cost) but uses TREATMENT/CONTROL labels to keep the semantic distinct from flag-tag-based ON/OFF cohorts.

Implementation

  • New Commands::EnvCohort variant with all the usual stratified params (--min-cohort-size, --max-cohorts, --detail, time range)
  • New EnvCohortResponse / EnvCohort types in types.rs (treatment/control instead of on/off)
  • Reuses print_cohort and print_cohort_diff helpers
  • Calls the new /traces/env_cohort devportal endpoint

Test plan

  • cargo fmt --check clean
  • cargo clippy --workspace -D warnings clean
  • cargo test --workspace passes
  • tb-lf env-cohort --help renders correctly with all params and examples
  • End-to-end against deployed devportal endpoint (depends on #263 merge + deploy)

Adds `tb-lf env-cohort`, a third cohort analysis command alongside
flag-cohort and flag-cohort-stratified. Pivots on environment membership
instead of flag-tag value, so it works for the common review-env pattern
where a feature is forced-ON in code (`true || flag_check()`) regardless
of what the flag's metadata tag says.

Calls the new devportal endpoint /traces/env_cohort. Treatment env is
where the feature actually runs; control envs are where the feature is
OFF / not deployed. --ignore-flags excludes specified flags from the
fingerprint (typically the target flag itself, whose tag value is
unreliable in the treatment env). Cohorts are paired across envs by
matching the remaining flag set.

Output uses TREATMENT/CONTROL labels to keep the semantic distinct from
the flag-tag-based ON/OFF cohorts.

Updated SKILL.md with workflow guidance and interpretation caveats
(cross-env noise on top of small-N task variance).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@trogulja trogulja marked this pull request as ready for review May 4, 2026 13:28
@trogulja trogulja merged commit e1299af into main May 4, 2026
1 check passed
@trogulja trogulja deleted the feat/tb-lf-env-cohort branch May 4, 2026 13:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant