Skip to content

Bug: Reconciler invariant violations spamming logs every 5s across all towns #1364

@jrf0110

Description

@jrf0110

Parent

Sub-issue of #204 (Phase 4: Hardening)

Summary

The reconciler invariant checker is logging violations every 5 seconds for every active town, generating massive log noise. Two distinct violations:

Invariant 7: "Working agent has no hook"

Affected towns: 8a6f9375 (your town), 505b54c4

For your town, the working agent is 9640a1a6 — the mayor. The mayor is always working with no hook (it doesn't work on beads). Invariant 7 should exclude mayors from the "working agent must have a hook" check.

For town 505b54c4, agent c07585bd is a non-mayor working agent with no hook. This is the #1358 bug — dispatch timeout race leaves the agent in an inconsistent state.

Invariant 5: "Convoy bead is in_progress (should only be open or closed)"

Affected towns: 505b54c4 (5 convoy beads), be20bf4b (3 convoy beads)

The invariant expects convoy parent beads to only be open or closed. But in_progress is a legitimate transient state — it means agents are actively working on the convoy's child beads. The invariant is too strict.

Either:

  1. Allow in_progress and in_review as valid convoy bead states in the invariant check
  2. Or the convoy bead status should not change from open until all children are closed (then it goes straight to closed). If convoy beads are being set to in_progress by dispatchAgent or hookBead, that's the bug.

Impact

  • Log spam: ~12 error log lines per 5-second alarm tick per affected town
  • No direct functional impact (invariants are detection-only, don't block reconciler actions)
  • Masks real errors in the Workers observability dashboard

Fix

  1. Invariant 7: AND agent_metadata.role != 'mayor'
  2. Invariant 5: Either allow in_progress/in_review for convoy beads, or fix the code path that sets convoy beads to in_progress

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Blocks soft launchbugSomething isn't workinggt:coreReconciler, state machine, bead lifecycle, convoy flowgt:observabilityLogging, metrics, Grafana, debuggingkilo-auto-fixAuto-generated label by Kilokilo-triagedAuto-generated label by Kilo

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions