Skip to content

Test fidelity check should fail CI when upstream tests aren't ported #53

@patrick-chinchill

Description

@patrick-chinchill

Process gap — silent test coverage erosion.

The problem

scripts/verify_test_fidelity.py reports every TS it("...") that has no matching Python test_*():

```
MISSING: [Options Load] should call onOptionsLoad handler for a matching action ID
MISSING: [Options Load] should prefer specific handlers before catch-all handlers
...
TOTAL: 548/588 matched (93%), 40 missing, 3 absorbers
```

But it exits 0 regardless of how many are missing. CI prints "Run with --fix to generate stubs for missing tests" and moves on. New upstream tests land unported, undetected, for releases at a time.

Why it's bad

This is how we ended up with:

Every one of those shows up in fidelity output. Every one got ignored because the script exits 0.

Fix

Option A — strict: exit non-zero when `missing > 0`. Simple, forces discipline.

Option B — graduated: exit non-zero when `missing > K` (where K is the current count, i.e. treat today's gap as the baseline and ratchet down). Allows incremental adoption without a big-bang PR.

Option C — per-file threshold: allow configured gaps per file in a `fidelity-baseline.json` (like pyrefly's). New misses must be explicitly acknowledged.

I'd recommend Option A for post-4.26 PRs + a separate catch-up PR that either ports the 40 missing tests or explicitly marks them as "intentionally skipped" with a reason.

Acceptance

  • `verify_test_fidelity.py --strict` (or equivalent) exits non-zero on missing tests
  • CI workflow invokes the strict mode
  • Existing 40 gaps handled (either ported or baselined with rationale)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions