fix(gastown): stop reconciler log spam from orphaned bead_cancelled events#3047
Merged
kilo-code-bot[bot] merged 1 commit intogastown-stagingfrom May 5, 2026
Merged
fix(gastown): stop reconciler log spam from orphaned bead_cancelled events#3047kilo-code-bot[bot] merged 1 commit intogastown-stagingfrom
kilo-code-bot[bot] merged 1 commit intogastown-stagingfrom
Conversation
…vents
Two independent bugs compose to flood production logs every alarm tick
with 'Bead <id> not found' errors:
1. deleteBead / deleteBeads did not clean up the town_events queue,
leaving bead_cancelled and container_status rows pointing at deleted
beads/agents.
2. applyEvent threw on missing beads and the drain loop never marked
the failing event processed — so it retried forever.
Fix 1: purge town_events rows (by bead_id OR agent_id, since agents are
beads) from deleteBead and the deleteBeads bulk path.
Fix 2a: reconciler.applyEvent('bead_cancelled') checks for the target
bead up front and returns (with a warn) when it's missing, instead of
throwing.
Fix 2b: the Town.do.ts drain loop recognises 'Bead/Agent <uuid> not
found' terminal errors, logs them at warn, and marks the offending
event processed so it stops retrying.
Adds debug RPCs (debugTownEvents, debugInsertTownEvent,
debugRecordContainerStatus) and integration coverage in
event-cleanup.test.ts.
Contributor
Code Review SummaryStatus: No Issues Found | Recommendation: Merge Files Reviewed (4 files)
Reviewed by gpt-5.5-20260423 · 674,327 tokens |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Production logs were flooded every alarm tick with:
Two independent bugs composed to produce this:
deleteBeaddid not clean up thetown_eventsqueue, andapplyEventthrew on the missing bead while the drain loop never marked the failing event processed — so the error repeated forever.deleteBead/deleteBeadsnow purgetown_eventsrows that reference the deleted bead by eitherbead_idoragent_id(agents are themselves beads, so the single cleanup covers both).reconciler.applyEvent('bead_cancelled')pre-checks that the target bead exists and returns with a warn if it doesn't, instead of throwing.Town.do.tsdrain loop now recognisesBead/Agent <uuid> not foundas a terminal error, logs atwarn, and marks the event processed so it stops retrying. Acts as a safety net for any future handler that forgets to pre-check existence, and also drains any pre-existing stuck events in production town DBs on the next alarm tick — no migration required.Added
debugTownEvents,debugInsertTownEvent, anddebugRecordContainerStatustest-only RPCs so integration tests can observe the event queue and inject orphan events.Verification
cd services/gastown && pnpm test -- event-cleanup— new integration tests cover all three fixes (delete-then-drain, directapplyEventon missing bead, drain loop classifying a terminal error).Full
pnpm typecheckwas skipped per repo guidance (too heavy); the gastown-local checks above exercise the changed code paths.Visual Changes
N/A — backend only.
Reviewer Notes
town_eventssweep usesbead_id OR agent_idbecausedeleteBeadis used for both bead and agent rows in this system.\b(Bead|Agent) [0-9a-f-]{36} not found\b) so other transient errors continue to be retried.debug*and only used by tests; they are not part of the public surface.