Skip to content

fix(gastown): stop reconciler log spam from orphaned bead_cancelled events#3047

Merged
kilo-code-bot[bot] merged 1 commit intogastown-stagingfrom
gt/toast/f2600010
May 5, 2026
Merged

fix(gastown): stop reconciler log spam from orphaned bead_cancelled events#3047
kilo-code-bot[bot] merged 1 commit intogastown-stagingfrom
gt/toast/f2600010

Conversation

@jrf0110
Copy link
Copy Markdown
Contributor

@jrf0110 jrf0110 commented May 5, 2026

Summary

Production logs were flooded every alarm tick with:

["reconciler: applyEvent failed",{"eventType":"bead_cancelled","error":"Bead <id> not found"}]

Two independent bugs composed to produce this: deleteBead did not clean up the town_events queue, and applyEvent threw on the missing bead while the drain loop never marked the failing event processed — so the error repeated forever.

  • Fix 1deleteBead / deleteBeads now purge town_events rows that reference the deleted bead by either bead_id or agent_id (agents are themselves beads, so the single cleanup covers both).
  • Fix 2areconciler.applyEvent('bead_cancelled') pre-checks that the target bead exists and returns with a warn if it doesn't, instead of throwing.
  • Fix 2b — the Town.do.ts drain loop now recognises Bead/Agent <uuid> not found as a terminal error, logs at warn, and marks the event processed so it stops retrying. Acts as a safety net for any future handler that forgets to pre-check existence, and also drains any pre-existing stuck events in production town DBs on the next alarm tick — no migration required.

Added debugTownEvents, debugInsertTownEvent, and debugRecordContainerStatus test-only RPCs so integration tests can observe the event queue and inject orphan events.

Verification

  • cd services/gastown && pnpm test -- event-cleanup — new integration tests cover all three fixes (delete-then-drain, direct applyEvent on missing bead, drain loop classifying a terminal error).

Full pnpm typecheck was skipped per repo guidance (too heavy); the gastown-local checks above exercise the changed code paths.

Visual Changes

N/A — backend only.

Reviewer Notes

  • The town_events sweep uses bead_id OR agent_id because deleteBead is used for both bead and agent rows in this system.
  • The terminal-error detection is deliberately narrow (\b(Bead|Agent) [0-9a-f-]{36} not found\b) so other transient errors continue to be retried.
  • The debug RPCs are namespaced under debug* and only used by tests; they are not part of the public surface.

…vents

Two independent bugs compose to flood production logs every alarm tick
with 'Bead <id> not found' errors:

1. deleteBead / deleteBeads did not clean up the town_events queue,
   leaving bead_cancelled and container_status rows pointing at deleted
   beads/agents.
2. applyEvent threw on missing beads and the drain loop never marked
   the failing event processed — so it retried forever.

Fix 1: purge town_events rows (by bead_id OR agent_id, since agents are
beads) from deleteBead and the deleteBeads bulk path.

Fix 2a: reconciler.applyEvent('bead_cancelled') checks for the target
bead up front and returns (with a warn) when it's missing, instead of
throwing.

Fix 2b: the Town.do.ts drain loop recognises 'Bead/Agent <uuid> not
found' terminal errors, logs them at warn, and marks the offending
event processed so it stops retrying.

Adds debug RPCs (debugTownEvents, debugInsertTownEvent,
debugRecordContainerStatus) and integration coverage in
event-cleanup.test.ts.
@kilo-code-bot
Copy link
Copy Markdown
Contributor

kilo-code-bot Bot commented May 5, 2026

Code Review Summary

Status: No Issues Found | Recommendation: Merge

Files Reviewed (4 files)
  • services/gastown/src/dos/Town.do.ts
  • services/gastown/src/dos/town/beads.ts
  • services/gastown/src/dos/town/reconciler.ts
  • services/gastown/test/integration/event-cleanup.test.ts

Reviewed by gpt-5.5-20260423 · 674,327 tokens

@kilo-code-bot kilo-code-bot Bot merged commit db58406 into gastown-staging May 5, 2026
2 checks passed
@kilo-code-bot kilo-code-bot Bot deleted the gt/toast/f2600010 branch May 5, 2026 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant