fix(watcher): prune graph nodes on file delete + rename (closes #9)#12
Merged
NickCirv merged 3 commits intoApr 18, 2026
Merged
Conversation
Used by the watcher to gate the new onDelete callback so it only fires for files that were actually indexed (avoids noise when unrelated files are deleted under the watched root). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…amed (NickCirv#9) The watcher only subscribed to "change" events, but fs.watch (recursive) fires "rename" for create/unlink on every platform. Deletions were silently ignored, leaving stale nodes in the graph. Renames left duplicate nodes — the old sourceFile lingered while the new path was indexed afresh. Changes: * Extract `syncFile(absPath, root)` exported helper that encodes "exists -> reindex; gone (and was indexed) -> prune". Shared contract that the `engram reindex` subcommand from NickCirv#8 will reuse. * Add `onDelete?: (filePath, prunedCount) => void` to WatchOptions alongside the existing onReindex. * Subscribe the debounced handler to both "change" and "rename". * Renames fall out naturally — the old path fires rename, prunes; the new path fires rename, indexes. Directory deletion (`rm -rf src/foo`) is intentionally out of scope for this PR — fs.watch fires a single rename event on the dir path which carries no per-file information. A full `engram init` handles that case today; per-file directory walking is a v2.2 follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Wire the new watcher onDelete callback into:
* `engram watch` — prints `× <path> pruned (N nodes)` distinct
from the existing green `↻ <path> (N nodes)` reindex line.
* `gen-cursor --watch`, `gen-aider --watch`, `gen-windsurfrules
--watch` — regenerate output files when a source file is
deleted, same as on reindex, so generated artifacts don't
keep stale references.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NickCirv
added a commit
that referenced
this pull request
Apr 18, 2026
Captures the changes merged in PR #12 (gabiudrescu/fix/issue-9-watcher-prune-deletes) under a Keep-a-Changelog Unreleased heading. Subsequent v2.1.0 features (#8 reindex subcommand, #5 skills-miner OOM defensive limits) will accumulate under the same section. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NickCirv
added a commit
that referenced
this pull request
Apr 18, 2026
The `formatStatsSummary > includes total invocation count` test has been intermittently timing out at 5000ms on Windows Node 22 CI (most recent bite: post-merge run on 9f99f5b after PR #12). Root cause: line 152 called `summary.estimatedTokensSaved.toLocaleString()`, and first-call ICU init on Windows Node has been observed to take multiple seconds in CI VMs. Replaced with a locale-independent `formatThousands()` helper (regex thousands separator). Two wins: 1. Deterministic performance — no ICU dependency, microsecond runtime, no first-call cost. 2. Locale-independent output — previously `"2,400"` on en-US CI runners but `"2.400"` on a de-DE system, which would have broken the `toContain("2,400")` assertion downstream AND given users inconsistent CLI output depending on their shell locale. Verified: 10 sequential stats.test.ts runs locally all pass, 3-4ms each. Full suite 673/673, TS strict clean, build clean. Also adds vitest.config.ts with CI-only `retry: 1` + 15s testTimeout (defaults stay tight on dev machines so slow tests still get noticed). Defense-in-depth against any other cold-worker flake we haven't spotted. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
gabiudrescu
added a commit
to gabiudrescu/engram
that referenced
this pull request
Apr 18, 2026
…Cirv#8) Closes NickCirv#8. Two parts, one commit; the maintainer offered to split into two PRs in the acceptance comment — happy to split on request. Part 1 — CLI subcommand - `engram reindex <file> [-p, --project <path>] [--verbose]`: re-indexes a single file via the shared `syncFile()` primitive from NickCirv#9/NickCirv#12. Semantics match `engram watch`: exists → reindex; missing-but- previously-indexed → prune; unsupported ext or ignored dir → silent exit 0 (safe to fire from a PostToolUse hook on every edit). - On success: single line `engram: reindexed <file> (<N> nodes)` (or `pruned`) using locale-stable `formatThousands` — no Windows ICU flake, matching the e099bf1 migration. - Missing graph: exits 1 with `engram: no graph found at <root>. Run 'engram init' first.`, mirroring `watchProject`'s explicit error. - Errors: single stderr line; `--verbose` surfaces the stack trace. - New pure formatter `formatReindexLine(result, displayPath)` exported from `src/watcher.ts` (returns null for skipped so callers stay silent). Part 2 — optional auto-reindex wiring - `engram reindex-hook`: PostToolUse stdin entry point. Reads Claude Code's JSON payload, extracts `tool_input.file_path`, delegates to `runReindexHook()` → `syncFile()`. ALWAYS exits 0 — malformed JSON, missing fields, non-project cwd, and internal errors all resolve to a silent no-op. Same bounded-stdin + watchdog pattern as `intercept`; `process.exitCode = 0` without `process.exit` so sql.js WASM drains. - `engram install-hook --auto-reindex`: off by default so existing users aren't surprised. Appends a second PostToolUse entry with matcher `Edit|Write|MultiEdit` (per the maintainer's acceptance) whose command is `engram reindex-hook`. Idempotent. - `isEngramHookEntry()` broadened to also match `engram reindex-hook` so `engram uninstall-hook` strips the auto-reindex entry alongside the primary intercept entries. - Installer idempotence check targets `engram intercept` specifically (via a new internal `entryContainsCommand` helper) — otherwise the newly-broadened `isEngramHookEntry` would false-positive when only the reindex-hook entry is present, causing install to skip adding the missing intercept entry. Implementation notes - `runReindexHook` walks up from the FILE path (not `cwd`) to find the project root. Claude Code's session `cwd` often sits above the engram-initialized project that owns the edited file (multi-project parent, monorepo subtree); `findProjectRoot(cwd)` would silently no-op in that case. Test covers this exact shape. - MultiEdit included in the matcher per the maintainer's acceptance note (the issue originally said Edit|Write). Tests (+16 new, all green — 704 total) - `formatReindexLine`: 4 unit tests (indexed/pruned/skipped/locale-stable). - `engram reindex` E2E via subprocess: happy path, non-code silent, no-graph error, modified-file-updates-graph (old fn out / new fn in), two-process concurrency (AC 6, honestly labelled as multi-process). - `runReindexHook`: 5 unit tests (happy, relative path, malformed-shape matrix, cwd-outside-project, walks-up-from-file-not-cwd). - `engram reindex-hook` E2E via subprocess: happy + malformed JSON + empty stdin, always exits 0. - `installEngramHooks({ autoReindex: true })`: 6 tests (adds second PostToolUse entry, off-by-default, idempotent, add-onto-existing, uninstall-strips-it, uninstall-preserves-non-engram). - `buildReindexHookEntry`, broadened `isEngramHookEntry`, and `formatInstallDiff` coverage for the new entry. Live smoke test (my machine) - `engram install-hook --scope user --auto-reindex` wrote the entry cleanly with a backup. - A `Write` of `src/proof-issue-8.ts` fired `engram reindex-hook` via Claude Code; `engram query` found the new function immediately. - `engram reindex src/proof-issue-8.ts` after `rm` produced `engram: pruned src/proof-issue-8.ts (2 nodes)` and the query came back empty. - `engram uninstall-hook` removed both engram entries cleanly. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NickCirv
added a commit
that referenced
this pull request
Apr 21, 2026
…h-postool ## Closes #11 — AST/LSP reported unavailable despite enabled Credit: Tommaso Tessarolo (@ttessarolo) — precise forensics + suggested fix. ### checkAst: flattened-bundle path resolution When tsup/esbuild flattens the CLI into `engramx/dist/chunk-*.js`, import.meta.url resolves `here = engramx/dist`. The previous candidates: - `here + '../grammars'` → engramx/grammars (✗ does not exist) - `here + '../../dist/grammars'` → parent/dist/grammars (✗ does not exist) Fix: add `join(here, "grammars")` as the FIRST candidate. This resolves to `engramx/dist/grammars/` where grammars actually ship. Dev-time layout (src/intercept/) still works via the third candidate. ### checkLsp: socket candidates out of sync The check only looked for `tsserver.sock` and `typescript-language-server.sock`, but `lsp-connection.ts::candidateSockets()` also probes: - tsserver-<uid>.sock - lsp-server.sock - pyright-<uid>.sock - rust-analyzer.sock Also: the `.engram/lsp-available` flag was documented as "written by the lsp provider on successful connection" but no code path writes it. Kept the marker as an explicit user opt-in (back-compat) but synced the socket list with lsp-connection.ts so HUD availability matches actual provider availability. ### Regression test tests/intercept/component-status-11.test.ts — asserts refreshComponentStatus returns a boolean for every scenario + honors the .engram/lsp-available marker. ## Wires bash-postool into PostToolUse (issue #14 full wire-up) With @gabiudrescu's PR #12 already merged, syncFile() is available. We can wire the v2.1 Bash parser into the PostToolUse observer path without waiting for PR #13's install-hook --auto-reindex flag. - post-tool.ts imports handleBashPostTool + syncFile. - On PostToolUse with tool=Bash, parse the command; for each FileOp, call syncFile(absPath, projectRoot) fire-and-forget. - Gated by ENGRAM_AUTO_REINDEX=1 (opt-in) until PR #13 lands and we can migrate to the install-hook flag. - Observer semantics preserved: never blocks the PostToolUse response, errors swallowed, never surfaces bugs to Claude Code. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
NickCirv
added a commit
that referenced
this pull request
Apr 24, 2026
Two orthogonal improvements to the resolver's assembly pipeline. Both exported from resolver.ts so they're testable in isolation, and both run in the main resolveRichPacket() flow before the final priority sort. 1. PER-PROVIDER BUDGET ENFORCEMENT (enforcePerProviderBudget) Providers are SUPPOSED to self-truncate their content to 'tokenBudget', but a bad plugin or a non-conforming MCP server shouldn't be able to spend our entire total budget on one section. New helper truncates each result to the provider's declared budget BEFORE assembly. - Under-budget content passes through unchanged (zero-cost) - Over-budget content is line-truncated (never cut mid-word) - Edge: first line alone > budget -> hard-cap characters with marker Default budget for unknown/missing providers is 200 tokens (matches the MCP-config default from item #1). 2. MISTAKES-BOOST RERANKING (boostByMistakes) If the engram:mistakes provider fires for this file, scan OTHER providers' content for substring matches against mistake labels (extracted from the ' ! <label> (flagged <age>)' format). Matching results get confidence * 1.5 (capped at 1.0). Runs BEFORE the priority sort, but the secondary sort is now (priority asc, confidence desc) — so boost breaks ties WITHIN a priority tier without overriding priority across tiers. - Case-insensitive matching (labels normalized to lowercase) - Does NOT boost the mistakes provider itself - No-op if no mistakes are reported for this file (common case) Examples of the intended effect: - An engram:git commit message mentioning a known-broken function sorts UP within the git tier - A mempalace decision that references a mistaken architectural choice bubbles ahead of unrelated decisions TESTS (+10 cases in tests/providers/resolver.test.ts) enforcePerProviderBudget: - Under-budget untouched - Over-budget truncated by line with marker - Hard-cap when first line alone exceeds budget - Default 200 tokens when provider not found boostByMistakes: - No-op when no mistakes provider in set - Matching substring boosts confidence 0.6 -> 0.9 - Cap enforced (0.8 * 1.5 = 1.2 -> 1.0) - Non-matching results left alone - Mistakes provider itself is never self-boosted - Case-insensitive matching across upper/lower case variations Full suite: 815 -> 825 tests (+10), all passing. TypeScript clean. V3.0 PROGRESS: 8 of 12 scope items done. ✅ #1 foundation ✅ #2 ✅ #3 ✅ #6 ✅ #7 ✅ #9 ✅ #10 ✅ #11 Remaining: #4 Auto-Memory (blocked on MEMORY.md fixture), #5 SSE streaming, #8 pre-mortem warnings, #12 MCP Registry submit, and #1 completion (HTTP transport + real-server integration tests).
NickCirv
added a commit
that referenced
this pull request
Apr 24, 2026
Opt-in warnings that fire BEFORE Claude Code runs an Edit/Write/Bash
tool call against code previously flagged as a mistake. Fully gated
via ENGRAM_MISTAKE_GUARD env var — zero overhead when unset.
MODES
unset / '0' → off (default — no database read, no overhead)
'1' → permissive: tool proceeds, a warning is prepended
to any additionalContext the primary handler emits
'2' → strict: tool is denied with the warning as reason
Hooks Edit/Write/Bash only. Read already surfaces mistakes via the
engram:mistakes context provider — duplicating at tool-call time would
be noise.
MATCHING
Edit/Write:
- Normalize tool_input.file_path to relative POSIX vs projectRoot
- Indexed lookup via store.getNodesByFile() (uses idx_nodes_source_file)
- Dedupe by node id when both relative + raw shapes are stored
Bash:
- Substring match on mistake.metadata.commandPattern (length >2)
- Fallback: substring match on mistake.sourceFile (length >3 to avoid
accidentally matching single-char paths like 'a')
- Full-table scan of mistakes (unavoidable — no file axis to index on).
Bounded by project size; only runs when the guard is explicitly on.
BI-TEMPORAL FILTER (item #7 interop)
Mistakes with validUntil <= now are suppressed — they refer to code
that has since been refactored away. Prevents stale-warning fatigue.
INTEGRATION
New file: src/intercept/handlers/mistake-guard.ts
- currentGuardMode() — reads env var at call time, not module load,
so tests can flip between cases cleanly
- findMatchingMistakesAsync(target, projectRoot) — the matcher
- formatWarning(matches) — human-readable warning block
- applyMistakeGuard(rawResult, payload, kind) — wrapping fn that
augments additionalContext (permissive) or overrides to deny (strict)
src/intercept/dispatch.ts wiring: after runHandler() returns for Edit/
Write/Bash, pass result through applyMistakeGuard() before returning.
Two-line diff. Doesn't touch the existing handlers.
SAFETY
Every code path in mistake-guard is wrapped in try/catch with a null
return. A guard failure MUST NEVER break the primary handler. If the
store open fails, the env var is wrong, the payload is malformed —
guard silently returns the raw result unchanged.
TESTS (+21 cases in tests/intercept/handlers/mistake-guard.test.ts)
- currentGuardMode: off/permissive/strict recognition, bogus values
coerced to off
- formatWarning: empty-match string, single-match header, >5-match
collapse with '… and N more'
- findMatchingMistakesAsync (file): rel path, abs path normalization,
no-match, validUntil filter
- findMatchingMistakesAsync (bash): commandPattern substring match,
sourceFile-in-command match, case-insensitive, too-short pattern
guard, validUntil filter
- applyMistakeGuard: mode=off no-op, permissive augments additional
context, permissive no-match no-op, strict denies with reason,
permissive from passthrough emits fresh allow-with-warning
Full suite: 825 -> 846 tests (+21), all passing. TypeScript clean.
V3.0 PROGRESS — 9 of 12 scope items
✅ #1 foundation ✅ #2 ✅ #3 ✅ #6 ✅ #7 ✅ #8 ✅ #9 ✅ #10 ✅ #11
Remaining:
- #1 completion (HTTP transport + real-server integration tests)
- #4 Anthropic Auto-Memory bridge (blocked: needs MEMORY.md fixture)
- #5 SSE streaming for rich packet assembly
- #12 Official MCP Registry submission (post-ship)
NickCirv
added a commit
that referenced
this pull request
Apr 24, 2026
Reads Claude Code's auto-managed MEMORY.md index and surfaces entries
relevant to the current file. Closes the Auto-Dream existential risk:
when Anthropic flips the server flag and MEMORY.md becomes consolidated-
and-high-quality, this bridge lights up with no code change.
FIXTURE CAPTURE
Real MEMORY.md samples live at ~/.claude/projects/<encoded>/memory/ on
every Claude Code machine. Captured a representative sample into
tests/fixtures/memory-md/sample-index.md so integration tests don't
depend on the user's actual memory directory.
CANONICAL FORMAT (from real fixtures)
- [Title](relative-file.md) — one-line description
Flat bullet list, one entry per line. Em-dash OR en-dash OR hyphen-
space all accepted. Linked .md files contain frontmatter + body —
this provider is INDEX-ONLY (doesn't dereference bodies) so it stays
under 10 ms even on large memory sets.
PATH DERIVATION
encodeProjectPath('/Users/alice/proj') -> '-Users-alice-proj'
getMemoryIndexPath(projectRoot) -> ~/.claude/projects/<encoded>/memory/MEMORY.md
Overridable via ENGRAM_ANTHROPIC_MEMORY_PATH env var for tests and
for advanced users who maintain a manual index.
RELEVANCE SCORING
+3 title contains file basename (sans extension)
+2 description contains file basename
+2 any import name appears in title or description (length ≥3)
+1 any path segment appears in title or description (length ≥3)
Top 3 matches with score >0 are returned; no matches = null.
INTEGRATION
- New provider wired into BUILTIN_PROVIDERS (src/providers/resolver.ts)
- Inserted at PROVIDER_PRIORITY index 3, between engram:mistakes
(+2) and mempalace (+4). Rationale: own-curated memory > shared
semantic memory when both are available.
SAFETY
- MAX_INDEX_BYTES = 1 MB hard cap (pathological files returned null)
- Empty files returned null (never a noise packet)
- All errors caught -> null return (never throws into resolver path)
TESTS (+24 cases in tests/providers/anthropic-memory.test.ts)
encodeProjectPath: standard path, trailing-slash trim, Windows
separator normalize, deep path preservation
getMemoryIndexPath: ends at the right path shape
parseMemoryIndex: well-formed index, malformed-line skip, empty-
content empty array, missing-description tolerated
scoreEntry: basename match (+3), import match (+2), zero on no
relationship, case-insensitive
resolve: missing file null, empty file null, no-match null, basename
match surfaces, caps at 3, over 1 MB skipped, override wins,
imports drive matches
isAvailable: default true (defers per-project), override exists true,
override missing false
Also updates tests/providers/resolver.test.ts — PROVIDER_PRIORITY
order test picks up the new index 3 slot.
Full suite: 846 -> 870 tests (+24), all passing. TypeScript clean.
V3.0 PROGRESS — 10 of 12 scope items done.
Remaining: #5 SSE streaming + #1 completion (HTTP transport + real MCP
server fixture) + #12 registry submit (post-ship).
NickCirv
added a commit
that referenced
this pull request
Apr 24, 2026
Adds progressive delivery for rich packet assembly. Instead of blocking
on Promise.allSettled (which waits for the slowest provider — Serena
cold-start, mempalace ChromaDB warmup), clients can stream results
as they arrive and render each section immediately.
NEW — resolveRichPacketStreaming generator (src/providers/resolver.ts)
AsyncGenerator<StreamEvent> that yields:
{ type: 'provider', result: ProviderResult } — as each resolves
{ type: 'done', providerCount, durationMs } — final totals
Order = ARRIVAL order (fast providers first). Consumers who want
priority order use the non-streaming resolveRichPacket() which applies
full priority + mistakes-boost + budget logic.
Implementation: fan-out all providers, funnel outcomes into a FIFO
queue + wake-on-arrival pattern. No extra deps. Per-provider timeouts
preserved (same resolveWithTimeout path as non-streaming).
NEW — /context/stream SSE endpoint (src/server/http.ts)
GET /context/stream?file=<relative-path> (auth required).
Emits one SSE frame per StreamEvent. Frame shape matches MCP SEP-1699
(SSE resumption):
id: 0
event: provider
data: {"provider":"engram:ast", …}
id: 1
event: provider
data: {"provider":"engram:mistakes", …}
id: N
event: done
data: {"providerCount":N,"durationMs":347}
Supports Last-Event-ID header — clients reconnecting via
'Last-Event-ID: 3' skip events 0-3 and pick up from 4. Useful for
long-running sessions that drop WiFi mid-stream without losing context.
Client-disconnect aborts the stream cleanly (req.close handler short-
circuits the generator loop).
TESTS (+6 new)
resolver.test.ts (+2):
- Smoke: streaming generator terminates with a 'done' event for any
project (no hang, no runaway)
- Arrival-order invariant: toy generator mirrors production shape,
verifies fast results yield before slow ones
server/http.test.ts (+4):
- Missing 'file' param returns 400
- Valid request returns 200 + text/event-stream + ends with 'done'
- Every frame carries an 'id:' header (SEP-1699 resumption)
- Auth required — unauthenticated returns 401
Full suite: 870 -> 876 tests (+6), all passing. TypeScript clean.
V3.0 PROGRESS — 11 of 12 scope items done
✅ #1 foundation ✅ #2 ✅ #3 ✅ #4 ✅ #5 ✅ #6 ✅ #7 ✅ #8
✅ #9 ✅ #10 ✅ #11
Only remaining in-scope work:
- #12 MCP Registry submission (~2h, post-ship only)
Plus item #1 completion (HTTP transport + minimal MCP server fixture
for integration tests) — technically part of #1 which shipped its
foundation as c719591; the HTTP transport path was explicitly deferred
until this SSE work landed. Now it can.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #9. Tagging @NickCirv per your acceptance comment.
Maps 1:1 to your guidance on #9:
syncFile(absPath, root)helper — exported fromsrc/watcher.ts. Encodes "exists → reindex; gone (and was indexed) → prune". Designed as the shared contractengram reindexfrom Feature:engram reindex <file>subcommand + optional PostToolUse auto-reindex #8 will reuse — itsSyncResulthas three actions:indexed,pruned,skipped.countBySourceFile— added toGraphStorenext todeleteBySourceFile. Used insidesyncFileto gate the prune branch (soonDeleteonly fires for paths the graph actually knows about — no log noise on drive-by deletes of un-indexed files).onDeletecallback — new optional field onWatchOptionsnext toonReindex. Fires only when prune count > 0.× pruned src/foo.ts (N nodes)log —engram watchnow prints it in yellow, distinct from the existing green↻ <path> (N nodes)reindex line.Scope-callout for your review
I also wired
onDeleteinto the threegen-cursor --watch/gen-aider --watch/gen-windsurfrules --watchcommands so they regenerate their output files on delete the same way they do on reindex. Without this, those generators silently keep stale references to deleted source files until any other file changes. Happy to split this into a follow-up PR if you'd prefer the strict #9 scope — it's the third commit (f06ca95), trivial to revert.Tests
tests/store.test.ts—countBySourceFileunit test.tests/watcher.test.ts— two new poll-based integration tests mirroring the existing pattern:onDeletefires with the prune count andstore.countBySourceFile(...)returns 0 afterwards.renameevent).engram init→engram watch &→rm src/to-delete.ts→ watcher logs× src/to-delete.ts pruned (3 nodes)→engram queryreturns "No matching nodes found." (was: 3 stale nodes).Known risk
The code comment claims
fs.watch (recursive)firesrenamefor create/unlink "all platforms". I verified on macOS only; Linux behavior I'm taking on documentation, and Windows recursivefs.watchrecently got CI fixes. If Windows CI flags anything, I'll iterate.CHANGELOG
Skipped a CHANGELOG entry — your project doesn't use an Unreleased section and v2.1.0 is your call to bump. Happy to add an entry under whatever heading you'd like before merge.
🤖 Generated with Claude Code