Skip to content

feat: rivet mechanical oracle (validate + impact → findings)#26

Merged
avrabe merged 1 commit intomainfrom
feat/rivet-oracle-module
Apr 26, 2026
Merged

feat: rivet mechanical oracle (validate + impact → findings)#26
avrabe merged 1 commit intomainfrom
feat/rivet-oracle-module

Conversation

@avrabe
Copy link
Copy Markdown
Contributor

@avrabe avrabe commented Apr 26, 2026

Why

The hardened AI review (PR #24) closes the model's escape hatches but the model itself is still the only source of findings. For repos that ship rivet traceability (rivet, spar, ...), we have a mechanical oracle that already emits the exact shape of finding we want — name-anchored, deterministic, falsifiable. Wire it in.

This is the pulseengine.eu "model proposes, oracle decides" pattern. The discovery here is that `rivet validate --format json` and `rivet impact --since= --format json` already produce the per-artifact diagnostic output we need; no model is involved in the judgement, only in proposing which lines to look at — which we get for free from the diff.

What

New module `src/rivet-oracle.js` with three exports:

Function Job
`isRivetProject(repoPath)` Detect `rivet.yaml` at the tree root.
`runRivetValidate(binary, repoPath, opts)` Run the validate CLI, parse JSON, convert errors + warnings + lifecycle gaps + broken cross-refs to findings. Severity `info` is dropped (66 on rivet/main alone — too noisy).
`runRivetImpact(binary, repoPath, baseRef, opts)` Run impact against a baseline ref. Surfaces removed artifacts and directly-affected. Transitive list summarised, not enumerated. `added`/`changed` not surfaced — diff metadata, not concerns.
`runRivetOracle(binary, repoPath, opts)` One-shot helper: both, merged, degraded-but-useful when one fails.

The module is side-effect-free: no cloning, no checkout, no install. The caller provides a working tree path. The `runner` option is injected for tests so we can stub `execFile` deterministically.

Each finding carries `{source, severity, artifact_id, claim}`. Claims are constructed from rivet output verbatim — no hedging is possible by construction. PR-A's slop filter is a no-op on oracle findings (correct: oracle findings don't pass through the model path).

What's NOT here

  • Integration with `ai-review.js` — that's PR-C. This is the building block; wiring needs repo-tarball fetching, base-ref handling, finding-source styling.
  • Auto-installer + scheduled update task — that's PR-D.
  • Binary on netcup — manual install (one shot) until the installer ships.

Test plan

  • All 752 tests pass (was 737 — added 15 covering parse, severity-drop, non-zero-exit handling, spawn failure, summary aggregation, degraded-mode behaviour)
  • eslint clean
  • Manual: ran `rivet validate --format json` against `pulseengine/rivet@v0.4.3` locally — emits 6 errors / 10 warnings / 66 infos / 1 lifecycle gap. Module converts to 17 findings (errors+warnings+gaps+broken; infos dropped).
  • Manual: ran `rivet impact --since=v0.4.2 --format json` on the same — shape matches.

Risk & rollout

  • Risk: low. No behaviour change — module is exported but not called yet. Pure addition.
  • Rollout: self-update on merge is a no-op for end users.

🤖 Generated with Claude Code

## Why
The hardened AI review (PR #24) closes the model's escape hatches but the
model itself is still the only source of findings. For repos that ship
rivet traceability (rivet, spar, ...), we have a *mechanical* oracle that
already emits the exact shape of finding we want — name-anchored,
deterministic, falsifiable. Wire it in.

This is the pulseengine.eu "model proposes, oracle decides" pattern. The
discovery here is that `rivet validate --format json` and `rivet impact
--since=<ref> --format json` already produce the per-artifact diagnostic
output we need; no model is involved in the *judgement* — only in the
proposal of which lines to look at, which we get for free from the diff.

## What
New module `src/rivet-oracle.js` with three exports:

| Function | Job |
|---|---|
| `isRivetProject(repoPath)` | Detect `rivet.yaml` at the tree root. |
| `runRivetValidate(binary, repoPath, opts)` | Run the validate CLI, parse JSON, convert error+warning diagnostics + lifecycle gaps + broken cross-refs to Finding records. Severity 'info' is dropped (66 of them on rivet/main alone — too noisy). |
| `runRivetImpact(binary, repoPath, baseRef, opts)` | Run the impact CLI against a baseline ref. Surfaces removed artifacts (risky) and directly_affected (the trace contract this PR is changing). Transitive list is summarised, not enumerated. `added`/`changed` are *not* surfaced — those are diff metadata, not concerns. |
| `runRivetOracle(binary, repoPath, opts)` | One-shot helper: runs both, merges findings, returns degraded-but-useful results when one of the two fails. |

The module is *side-effect-free* on the caller's repo: it does not clone,
checkout, or install. The caller provides a working tree path. The `runner`
option is injected for tests so we can stub `execFile` deterministically.

Each finding carries `{source: 'oracle:rivet-validate' | 'oracle:rivet-impact',
severity, artifact_id, claim}`. Claims are constructed from the rivet output
verbatim — no hedging language is possible by construction. The PR-A
slop filter is therefore a no-op on these findings (which is fine — they
don't go through the model path anyway).

## What's NOT in this PR
- Integration with `ai-review.js` — that's PR-C. This PR is the building
  block; wiring it requires repo-tarball fetching, base-ref handling, and
  finding-source styling in the renderer.
- Auto-installer + scheduled update task — that's PR-D.
- The binary itself on netcup — manual install (one shot) until the
  installer ships.

## Test plan
- [x] All 752 tests pass (was 737 — added 15 covering parse, severity-drop,
      non-zero-exit handling, spawn-failure, summary aggregation,
      degraded-mode behaviour)
- [x] eslint clean
- [x] Manual: ran `rivet validate --format json` against
      `pulseengine/rivet@v0.4.3` locally — it emits 6 errors / 10 warnings /
      66 infos / 1 lifecycle gap. Module correctly converts to 17 findings
      (errors + warnings + gaps + broken cross-refs; infos dropped).
- [x] Manual: ran `rivet impact --since=v0.4.2 --format json` on the same —
      JSON shape matches what the parser expects.

## Risk & rollout
- Risk: **low**. No behaviour change yet — module is exported but not yet
  called from `ai-review.js`. Pure addition.
- Rollout: self-update on merge is a no-op for end users.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@avrabe avrabe merged commit 0e9e9de into main Apr 26, 2026
5 checks passed
@avrabe avrabe deleted the feat/rivet-oracle-module branch April 26, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant