Checkpoint diffs never resolve: ProviderRuntimeIngestion placeholder preempts CheckpointReactor git capture

[eggfriedrice24]

Bug

Turn diffs ("Loading checkpoint diff…" / "Filesystem checkpoint is unavailable for turn N") never resolve after a Codex turn completes, even when the project is a valid git repository.

Root cause

There is a race between two independent event handlers that both react to turn completion:

1. ProviderRuntimeIngestion handles turn.diff.updated events and immediately dispatches thread.turn.diff.complete with a placeholder checkpoint entry:

   • checkpointRef: "provider-diff:<eventId>" (not a real git ref)
   • status: "missing"
   • files: []

   (apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts, lines 1064-1083)

2. CheckpointReactor handles turn.completed events and is supposed to capture a real git-ref-based checkpoint via CheckpointStore.captureCheckpoint, then dispatch thread.turn.diff.complete with the actual diff data.

   (apps/server/src/orchestration/Layers/CheckpointReactor.ts, lines 150-298)

The problem: ProviderRuntimeIngestion wins the race and inserts the placeholder into thread.checkpoints first. When CheckpointReactor later processes turn.completed, it checks:

if (thread.checkpoints.some((checkpoint) => checkpoint.turnId === turnId)) {
  return; // ← early exit, real capture never happens
}

(line 169)

Since the placeholder already has the same turnId, the real git checkpoint is never captured. The DiffPanel then tries to resolve provider-diff:UUID via git rev-parse, which fails because it's not an actual git ref.

Observed behavior

• Server logs show thread.turn-diff-completed events with checkpointRef: "provider-diff:..." and status: "missing"
• DiffPanel shows "Loading checkpoint diff..." indefinitely, then "Filesystem checkpoint is unavailable for turn N"
• No refs/t3/checkpoints/ git refs are created for the affected turns

Reproduction

1. Add any git-initialized project
2. Send a message that triggers file changes (e.g., "add a comment to README")
3. Wait for turn to complete
4. Open the diff panel → stuck on "Loading checkpoint diff..."

Suggested fix

The dedup check in CheckpointReactor (line 169) should either:

• Only skip if an existing checkpoint has a non-placeholder status (i.e., skip "missing" entries), or
• Replace/upgrade placeholder checkpoints with the real git-based checkpoint when one becomes available

Additionally, there is a secondary CWD inconsistency worth noting: CheckpointReactor turn-completion capture prefers sessionRuntime.cwd while CheckpointDiffQuery only uses resolveThreadWorkspaceCwd. If these ever diverge, refs would be written to one repo but looked up in another.

Environment

• T3 Code dev (main branch)
• Codex provider with GPT-5.4
• Linux (Arch)

[eggfriedrice24]

Working on a fix locally, will post results shortly.

[eggfriedrice24]

Update: Root cause found and fix verified

The dedup race described in the issue is real, but it turned out to be a secondary problem. The primary root cause is deeper:

Primary: Shared PubSub subscription

Stream.fromPubSub(pubsub) in effect-smol creates a subscription eagerly when called. Both OrchestrationEngine.streamDomainEvents and ProviderService.streamEvents stored the result as a plain property on the returned service object:

// Before (broken): single subscription shared by all consumers
const streamDomainEvents = Stream.fromPubSub(eventPubSub);
return { streamDomainEvents, ... };

Every consumer that reads the property gets the same stream/subscription. The first consumer to start reading (ProviderRuntimeIngestion, wsServer) drains events from the queue; subsequent consumers (CheckpointReactor) receive zero events.

This means CheckpointReactor never saw turn.completed runtime events OR domain events, so it could never capture git checkpoints regardless of the dedup check.

Fix: Change both properties to getters so each access creates a fresh Stream.fromPubSub() call:

// After (fixed): each consumer gets its own subscription
get streamDomainEvents() {
  return Stream.fromPubSub(eventPubSub);
},

Applied to:

• apps/server/src/orchestration/Layers/OrchestrationEngine.ts (streamDomainEvents)
• apps/server/src/provider/Layers/ProviderService.ts (streamEvents)

Secondary: Placeholder checkpoints not replaced

Even with events flowing, the original dedup check would still block real captures. Two fixes:

1. Updated dedup check in captureCheckpointFromTurnCompletion to only skip if a non-placeholder checkpoint exists (checkpoint.status !== "missing")

2. Added captureCheckpointFromPlaceholder handler that reacts to thread.turn-diff-completed domain events where status === "missing". When ProviderRuntimeIngestion creates a placeholder, this handler captures the real git checkpoint and dispatches a replacement with status: "ready" and actual file diffs. This is more reliable than depending solely on runtime turn.completed events.

Files changed

• apps/server/src/orchestration/Layers/OrchestrationEngine.ts (getter fix)
• apps/server/src/provider/Layers/ProviderService.ts (getter fix)
• apps/server/src/orchestration/Layers/CheckpointReactor.ts (placeholder handler + dedup fix)

Verified

After applying the fix, CheckpointReactor successfully captures real git checkpoints (refs/t3/checkpoints/.../turn/N) and the diff panel shows actual file diffs.

Will open a PR shortly.

[eggfriedrice24]

this was PAIN to fix

[akarabach]

the same