Skip to content

reflector: port karpathy-loop's bounded reflection pattern#2

Merged
jaschadub merged 2 commits intomainfrom
worktree-karpathy-loop-learnings
Apr 19, 2026
Merged

reflector: port karpathy-loop's bounded reflection pattern#2
jaschadub merged 2 commits intomainfrom
worktree-karpathy-loop-learnings

Conversation

@jaschadub
Copy link
Copy Markdown
Contributor

Summary

Ports the reflection/knowledge-store pattern from symbiont-karpathy-loop into this pipeline. Between phases, a bounded reflector agent distils a phase's findings into subject-predicate-object lessons. The next phase's agent recalls those lessons before planning, so learning flows forward across an engagement without any agent mutating another's tools or policy.

What's added

  • New knowledge SQLite table (db/schema.sql, src/db.rs) with subject-predicate-object triples, confidence in [0.0, 1.0], engagement + phase scoping.
  • Two MCP tools (src/knowledge_tools.rs, Cedar resource PenTest::KnowledgeStore):
    • store_knowledge — reflector-only, gated by reflector.cedar
    • recall_knowledge — read-only, permitted for every phase agent
  • New reflector agent (agents/reflector.dsl) with reflect() that reads phase findings, dedups against prior triples, writes new ones.
  • Cedar policy (policies/reflector.cedar) uses forbid ... unless defensive negation. Even if someone later widens a permit elsewhere, the global forbid keeps the reflector's surface bounded to {store_knowledge, recall_knowledge, query_findings}.
  • Phase DSLs updated (recon/enum/vuln-assess/exploit/post-exploit/reporter) — each gains a recall_knowledge capability and calls it at phase entry.
  • Engagement controller (agents/engagement-controller.dsl) now invokes the reflector after each phase.
  • Policiestool-authorization.cedar authorizes recall_knowledge at LOW risk globally; evidence.cedar exempts the knowledge tools from the pending-tool-runs backlog check.
  • DocsREADME.md and AGENTS.md agent hierarchy, key files table, tool roster (now 35 tools across 8 modules).
  • Tests — db round-trip + engagement isolation + limit enforcement; reflector policy text shape assertions; knowledge tool registration checks. Tool total guard bumped 33 → 35.

Why this pattern

The pen test pipeline already has retest comparisons (compare_engagements) and a finding audit trail, but nothing carries procedural lessons across phases or engagements. The karpathy-loop repo demonstrates that a bounded reflector — strictly separated from the actor it teaches — can produce concrete, indexable lessons without becoming a recursive self-improver. That's a good fit here: later phases benefit from what earlier phases learned, with zero increase in any phase agent's capabilities.

Defense in depth

The reflector is bounded at three layers:

  1. DSL capabilitiesreflector.dsl declares only {store_knowledge, recall_knowledge, query_findings}. It has no way to name a scan/exploit tool.
  2. Cedar permit — narrowly scoped to those three tools.
  3. Cedar forbid ... unless — global deny on execute_tool with a small whitelist. Any future tool added outside the whitelist is rejected automatically.

Plus explicit forbid clauses for scan, exploit, and post_exploit actions that any reader can skim.

Test plan

  • cargo test --test db_tests knowledge_ — insert/recall round-trip, per-engagement isolation, limit cap
  • cargo test --test tool_registration_tests knowledge_ — tool count, Cedar resource, actions, no human gate
  • cargo test --test reflector_policy_test — policy shape assertions (principal, forbid-unless, tool whitelist, action forbids)
  • cargo test --test db_tests init_db_creates_tablesknowledge table appears in schema
  • cargo test --test tool_registration_tests total_tools_is_35 — total count guard
  • symbi policy evaluate policies/reflector.cedar against a reflector principal requesting nmap_scan — should deny
  • symbi policy evaluate policies/reflector.cedar against a reflector principal requesting store_knowledge — should permit
  • Full engagement dry-run: confirm ask(reflector, phase=X) is invoked after every phase and subsequent phase agents see non-empty recall_knowledge output

Notes for reviewers

  • cargo check couldn't be exercised locally because symbi-channel-adapter's relative path in Cargo.toml resolves to a location that doesn't exist on this machine — same for the main branch. All Rust additions mirror the existing evidence_tools.rs / db.rs patterns exactly, so the shape should hold. CI on the canonical repo layout should validate.
  • No changes to approval/Slack code — the recent approval work stays intact.

symbi added 2 commits April 19, 2026 11:36
Borrow symbiont-karpathy-loop's post-phase reflector idea into the pen
test pipeline. Between phases the controller now invokes a bounded
reflector agent that reads the phase's findings and writes
subject-predicate-object lessons to a new `knowledge` table. The next
phase's agent recalls those lessons before planning so learning flows
forward across the engagement without any agent mutating another's
tools or policy.

Boundary is enforced the same way as in karpathy-loop: Cedar's
`reflector.cedar` uses `forbid ... unless` defensive negation to keep
the reflector's surface bounded to `store_knowledge`,
`recall_knowledge`, and `query_findings`. Every scan/enum/exploit/post-
exploit action is rejected at the gate even if someone later widens a
permit elsewhere. The reflector's DSL capabilities list is the second
layer: it declares nothing it could use to act on targets.

Changes:
- db/schema.sql, src/db.rs: `knowledge` table + typed insert/recall
- src/knowledge_tools.rs: `store_knowledge` + `recall_knowledge` MCP
  tools, gated on `PenTest::KnowledgeStore`
- agents/reflector.dsl + policies/reflector.cedar: bounded agent
- agents/{recon,enum,vuln-assess,exploit,post-exploit,reporter}.dsl:
  add `recall_knowledge` capability + phase-entry recall
- agents/engagement-controller.dsl: invoke reflector after each phase
- policies/tool-authorization.cedar: authorize `recall_knowledge`
  globally (LOW risk, read-only)
- policies/evidence.cedar: exempt knowledge tools from the pending-
  tool-runs backlog check
- tests: db round-trip, reflector policy shape assertions, knowledge
  tool registration checks
- AGENTS.md: updated agent hierarchy, key files, and tool roster
- Agent count 7 -> 8; hierarchy shows reflector between phases and
  reporter, with a short paragraph on what it does and how it's bounded.
- MCP tool count 31 -> 35 in the architecture diagram.
- Policy table adds reflector.cedar with its defensive forbid-unless
  note; count 7 -> 8.
- Data layer section introduces the knowledge store next to findings
  and evidence, with one concrete triple example.
- Repository structure shows reflector.dsl and knowledge_tools.rs.
- Key design decisions gains a "bounded reflector" entry.
- Capability comparison table gains a cross-phase-learning row.
@jaschadub jaschadub merged commit d1cc40e into main Apr 19, 2026
2 checks passed
@jaschadub jaschadub deleted the worktree-karpathy-loop-learnings branch April 19, 2026 19:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant