Continuous learning: encoding model that improves from operational experience

[CalebisGross]

Vision

The encoding model should improve from operational experience. Every encoding it produces, every recall that succeeds or fails, every feedback signal — all of this is training data that currently gets discarded. The model that encodes memory #10,000 should be measurably better than the one that encoded memory #1.

Full design document: docs/DESIGN_continuous_learning.md

Architecture

Four tiers that build on each other:

Tier 1 — Experience Collection (always running, zero cost)

• Verification gate scores every encoding (EPR, FR, TE) at write time
• Feedback from recall sessions links back to the encodings
• Experience buffer accumulates gold + bad encoding pairs

Tier 2 — Curriculum Generation (nightly, during dreaming)

• Bad encodings get re-encoded by Gemini to produce corrected versions
• Creates (input, bad_local, good_gemini) tuples — DPO training data
• Hard example mining focuses on the model's weakest areas

Tier 3 — Model Update (triggered when ≥50 new training pairs)

• Short spoke fine-tuning (SFT, later DPO) on new experience
• 30% replay from base dataset prevents catastrophic forgetting
• Quality gate: must pass EXP-25 probes before deployment
• Atomic spoke deployment with rollback

Tier 4 — Self-Assessment (metacognition-driven)

• Rolling quality window detects encoding drift
• Domain shift detection flags new patterns
• Autonomous training trigger when quality degrades

What Makes This Special

The feedback signal is downstream and delayed — an encoding is "good" not because it looks right at write time, but because it leads to useful recalls days or weeks later. The model learns what makes memories findable and useful, not just structurally correct.

Key Dependencies

• Verification gate (designed in EXP-29, not yet implemented)
• EXP-25 probe evaluation framework (exists)
• Spoke training pipeline (exists)
• Dreaming agent as training orchestrator (exists, needs extension)
• Gemini API for corrective re-encoding (available)

Implementation Roadmap

Phase	Work	Estimate
1. Experience Collection	DB migration, verification gate, experience buffer	2-3 days
2. Recall-Encoding Linkage	recall_feedback table, feedback processing	1-2 days
3. Curriculum Generation	Dreaming phase extension, Gemini re-encoding	2-3 days
4. Automated Training	Spoke training integration, quality gate, deployment	3-5 days
5. Metacognition Integration	Quality tracking, domain shift, training triggers	2-3 days
6. DPO (stretch)	Preference optimization from corrective pairs	2-3 days

Total: ~2-3 weeks

Success Criteria

• 1 month: ≥500 experience buffer entries, ≥2 training cycles, EPR +5pp
• 3 months: ≥10 training cycles, handles new domains without quality drop
• 6 months: model adapted to user's patterns, exceeds Gemini on user's actual inputs

[CalebisGross]

Design Review Progress — 2026-04-10

Approach Selected: Phased Automation (Approach C)

Three independently valuable, independently reviewable phases. Each ships as its own PR.

Key Design Decisions

1. Model-agnostic architecture — The continuous learning pipeline works regardless of which base model wins EXP-29 (Qwen 3.5 2B, 4B, Gemma E2B, or Bespoke-pruned model from EXP-28). Verification gate is pure string matching, experience buffer stores quality metrics about encodings regardless of producer, training trigger is a config value pointing at a checkpoint.

2. Hybrid orchestration (Option C) — Go metacognition agent decides whether to train. If yes, daemon writes a training request file and exits gracefully. A systemd path unit watches for the file, runs the Python training script, validates quality gate, restarts daemon with new or old spokes. Daemon never restarts itself. Respects Unix philosophy, gives systemd proper supervision, keeps PyTorch in Python where it belongs.

3. Classification metrics: EPR + recall_score + TED — Fabrication Rate (FR) is monitoring-only, NOT used for gold/needs_improvement classification. EXP-25 showed 25.8% FR on perfect outputs due to legitimate semantic expansion ("WAL mode on." → "database"). FR has >25% false positive rate on correct outputs. Classification thresholds:

   • Gold: EPR > 0.9, TED = false, AND (recall_score > 0.8 OR no recalls yet + no flags)
   • Needs improvement: EPR < 0.7 OR TED = true OR recall_score < 0.3 (with recall_count >= 3)
   • Ambiguous: Everything else
   • The recall_count >= 3 guard prevents a single irrelevant rating from condemning an encoding (design doc risk mitigation for feedback noise).

4. Phase 1 signals: write-time verification + recall linkage — Gemini comparative baseline deferred to Phase B (curriculum generation). Write-time verification (EPR, TED, MIG) gives immediate signal. Recall feedback starts accumulating from day one; takes weeks to become dense but is the real downstream quality signal. Observe before intervening.

---

Phase A: Runtime Verification & Experience Collection (~1 week)

Pure observation. No model changes, no training, no Gemini calls.

A.1 Runtime Verification Gate

New file: internal/agent/encoding/verification.go

Pipeline insertion point: After compressAndExtractConcepts() returns (agent.go ~line 1046), before embedding generation:

compressAndExtractConcepts() → verifyFaithfulness() → generateEmbedding() → persist()

Checks computed (Go port of eval_faithfulness.py patterns):

Check	Method	Cost
EPR (Entity Preservation Rate)	Regex extraction of proper nouns, numbers, paths, versions from raw input. Fraction found in compression's content-bearing fields.	~0.1ms
TED (Template Echo Detection)	Check output against known instruction phrases from encoding prompt. Boolean.	~0.05ms
MIG (Minimal Input Guard)	If raw < 50 chars and output content > 300 chars → flagged as padded.	~0.01ms
FR (Fabrication Rate)	Extract entities from output, fraction not in input. Monitoring only.	~0.1ms

Behavior: Soft mode only (no rejections). Log warning, store metrics, persist regardless. Reduce salience by 0.1 if TED=true. No hard mode until we have baseline data to calibrate thresholds — observe, then intervene.

A.2 Schema Migration

-- Verification metrics on memories table
ALTER TABLE memories ADD COLUMN encoding_epr REAL DEFAULT NULL;
ALTER TABLE memories ADD COLUMN encoding_fr REAL DEFAULT NULL;
ALTER TABLE memories ADD COLUMN encoding_flags TEXT DEFAULT NULL;

-- Recall-encoding linkage
CREATE TABLE recall_feedback (
    id TEXT PRIMARY KEY,
    query TEXT NOT NULL,
    memory_id TEXT NOT NULL,
    feedback TEXT NOT NULL,  -- helpful, partial, irrelevant
    recall_session_id TEXT,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (memory_id) REFERENCES memories(id)
);
CREATE INDEX idx_recall_feedback_memory ON recall_feedback(memory_id);

-- Experience buffer
CREATE TABLE experience_buffer (
    id TEXT PRIMARY KEY,
    raw_id TEXT NOT NULL,
    memory_id TEXT NOT NULL,
    encoding_epr REAL,
    encoding_fr REAL,
    encoding_flags TEXT,
    recall_score REAL,
    recall_count INTEGER DEFAULT 0,
    category TEXT DEFAULT 'ambiguous',
    used_in_training INTEGER DEFAULT 0,
    created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    updated_at DATETIME DEFAULT CURRENT_TIMESTAMP,
    FOREIGN KEY (memory_id) REFERENCES memories(id)
);
CREATE INDEX idx_experience_buffer_category ON experience_buffer(category);

A.3 Experience Buffer Population

• On every encoding: Insert experience_buffer row with EPR, FR, flags. Initial category = ambiguous.
• On every recall feedback: Insert recall_feedback row. Update experience_buffer.recall_score as running weighted average (helpful=1.0, partial=0.5, irrelevant=0.0). Increment recall_count.
• Reclassification: Runs during dreaming cycle. Applies gold/needs_improvement/ambiguous thresholds. The recall_count >= 3 guard prevents noisy single-rating condemnations.

A.4 Metacognition: Quality Drift Detection

New observation type: encoding_quality_drift. Extends auditMemoryQuality():

• Rolling averages over last 100 encodings: mean EPR, TED rate, flag distribution
• Compare against baseline (first 100 encodings after Phase A ships)
• Emit trend: improving / stable / degrading
• If degrading for 2+ consecutive audit cycles → flag for attention (triggers training in Phase C)

A.5 Store Interface Additions

WriteVerificationResult(ctx, memoryID, epr, fr, flags) error
WriteExperienceEntry(ctx, entry) error
UpdateExperienceRecallScore(ctx, memoryID, feedback) error
ReclassifyExperienceBuffer(ctx) (int, error)
ListExperienceByCategory(ctx, category, limit) ([]ExperienceEntry, error)
WriteRecallFeedback(ctx, rf) error
GetRecallHistory(ctx, memoryID) ([]RecallFeedback, error)

---

Phases B and C design in progress — will be added as subsequent comments.

[CalebisGross]

Phase B: Curriculum Generation (~1 week)

Runs during dreaming hours. Builds the training data that Phase C consumes. Still no model changes.

B.1 Dreaming Agent Extension

New phase: Phase 4.5 in internal/agent/dreaming/agent.go (after noise prune, before cross-project linking).

Trigger conditions:

• Experience buffer has >= 10 entries categorized as needs_improvement
• At least 24 hours since last curriculum generation run
• Gemini API is configured and reachable (graceful skip if not)

Algorithm per entry (up to 50/run):

1. Load original raw input from raw_memories
2. Send to Gemini API with build_production_prompt() — identical prompt the local model saw
3. Run verification (EPR, FR, TED) on Gemini's output
4. If Gemini passes → store as corrected_output on experience_buffer entry (training pair created)
5. If Gemini also fails → reclassify entry as "ambiguous", log for manual review

B.2 Schema Addition

ALTER TABLE experience_buffer ADD COLUMN corrected_output TEXT DEFAULT NULL;
ALTER TABLE experience_buffer ADD COLUMN corrected_epr REAL DEFAULT NULL;
ALTER TABLE experience_buffer ADD COLUMN corrected_fr REAL DEFAULT NULL;
ALTER TABLE experience_buffer ADD COLUMN correction_source TEXT DEFAULT NULL;  -- 'gemini', 'manual'
ALTER TABLE experience_buffer ADD COLUMN corrected_at DATETIME DEFAULT NULL;

B.3 Hard Example Mining

After curriculum generation, analyze the experience buffer for systematic failure patterns:

Detection heuristics:

• Dense numbers: raw input contains >= 5 distinct numeric values AND EPR < 0.8
• Short input: raw input < 30 chars AND any flag set
• Template echo: TED = true (direct detection)
• Domain gap: cluster needs_improvement by concept overlap — if 5+ share concepts not in training data, flag as new domain

Output: curriculum_report stored as metacognition observation. Tells Phase C what kind of training data to prioritize — targeted improvement, not random sampling.

B.4 Gemini API Integration

Uses existing llm.Provider interface — daemon already has Gemini configured. Same ChatCompletion() method, production encoding prompt, temperature 0.3.

Rate limiting: Max 50 corrections per dreaming cycle. ~$0.01/night at Gemini 2.5 Flash pricing.

continuous_learning:
  curriculum:
    enabled: false              # opt-in
    max_corrections_per_cycle: 50
    min_needs_improvement: 10
    cooldown_hours: 24

B.5 Training Data Assembly

Assembled data written to training/data/continuous_learning/ with timestamped filenames. This is the Go→Python handoff point.

SFT pairs (gold examples):

{"input": "<raw input>", "output": "<local encoding rated gold>"}

Corrective SFT pairs (Gemini re-encodings):

{"input": "<raw input>", "output": "<gemini corrected encoding>"}

Future DPO triples (stretch goal):

{"input": "<raw input>", "chosen": "<gemini encoding>", "rejected": "<local bad encoding>"}

---

Phase C: Automated Training & Deployment (~1-2 weeks)

Closes the loop. Metacognition decides when to train, systemd orchestrates it, Python runs PyTorch, quality gate validates.

C.1 Training Request Protocol

When metacognition detects degrading quality trend (or manual MCP trigger), the daemon:

1. Assembles training manifest (JSON):
   • Request ID, timestamp, trigger reason
   • Experience buffer stats (gold/needs_improvement/corrected counts)
   • Current model info (base, spokes checkpoint, GGUF path, baseline metrics)
   • Training data path
2. Writes to ~/.local/share/mnemonic/training_requests/pending.json
3. Exits gracefully (clean shutdown, releases VRAM)

C.2 Systemd Integration

Three new systemd units:

mnemonic-train.path — watches for training requests:

[Path]
PathExists=%h/.local/share/mnemonic/training_requests/pending.json
Unit=mnemonic-train.service

mnemonic-train.service — runs training pipeline:

[Service]
Type=oneshot
ExecStart=%h/Projects/mem/training/scripts/continuous_train.sh
TimeoutStartSec=1800    # 30 min max

continuous_train.sh — orchestrator:

#!/bin/bash
set -euo pipefail
# 1. Stop daemon (free VRAM)
systemctl --user stop mnemonic
# 2. Run training
python3 training/scripts/continuous_train.py --request "$REQUEST" --result "$RESULT"
# 3. Archive request
mv "$REQUEST" "${REQUEST%.json}.$(date +%Y%m%d%H%M%S).completed.json"
# 4. Restart daemon (new spokes if passed, old if not)
systemctl --user start mnemonic

C.3 Training Script (continuous_train.py)

Training data mix per batch:

• 30% replay from v7 base dataset (prevents catastrophic forgetting)
• 50% gold examples from experience buffer
• 20% corrective pairs from Gemini re-encodings
• Fixed seed for reproducibility

Key parameters:

• LR: 1e-4 (conservative for incremental updates)
• max_steps: 500 (2-8 epochs on 50-200 examples)
• batch_size: 1, grad_accum: 8
• gradient_checkpointing: true
• Reuses train_qwen_spokes.py infrastructure (proven in EXP-18/20/25)

C.4 Quality Gate

Reuses existing eval infrastructure. Runs after training completes:

1. Export temporary spokes to GGUF via export_qwen35_spokes.py
2. Start llama-server with temporary GGUF
3. Run eval_faithfulness.py on 25 gold-standard probes
4. Run stress_test_hallucination.py (7 tests)
5. Compare against baseline stored in training request

Pass criteria:

Metric	Threshold	Rationale
EPR	>= 90%	EXP-25 target
SC	>= 95%	Allow 1/25 edge case
Stress test	>= 6/7	Current production baseline
No metric regression	<= 5pp	Monotonic improvement enforced

FR is NOT a gate criterion — monitoring only, consistent with classification decision.

On pass: Export final GGUF, quantize to RQ4, update config.yaml, daemon restarts with new spokes.
On fail: Discard, log failure, daemon restarts with old spokes unchanged.

C.5 Spoke Deployment

• Version-suffixed GGUFs: qwen35-2b-spokes-cl001-f16.gguf
• Quantized via rotorq_quantize_gguf.py (BetaQ RQ4)
• Config.yaml updated to point to new GGUF
• Previous 3 versions retained for rollback
• Daemon loads new spokes on restart automatically

C.6 Manual Training Trigger

New MCP tool: trigger_training(reason: string) -> {request_id, status}

Allows manual trigger without waiting for quality drift. Same flow as automated trigger. Useful during development and pipeline testing.

C.7 Full Config Block

continuous_learning:
  enabled: false                    # opt-in, off by default
  training:
    min_new_examples: 50
    max_examples_per_run: 200
    replay_ratio: 0.30
    lr: 1.0e-4
    max_steps: 500
    seed: 42
    quality_gate:
      min_epr: 0.90
      max_fr: 0.10
      min_sc: 0.95
      min_stress: 6
      max_regression_pp: 5
    rollback_versions: 3
  curriculum:
    enabled: false
    max_corrections_per_cycle: 50
    min_needs_improvement: 10
    cooldown_hours: 24
  trigger:
    auto: false
    manual: true
    training_window: "02:00-06:00"

---

Summary: What Each Phase Delivers

Phase	Ships	Value
A	Verification gate, experience buffer, recall linkage, drift detection	Encoding quality observability. Every encoding scored. Baseline established.
B	Curriculum generation, hard example mining, training data assembly	Training data curated automatically. Failure patterns identified.
C	Automated training, quality gate, spoke deployment, rollback	Self-improving model. Closed loop from experience → training → deployment.

Each phase is independently valuable and ships as its own PR.

[CalebisGross]

Implementation Spec Complete

Spec committed: docs/SPEC_continuous_learning.md at 7df03b3 on feat/exp29-candidate-eval

Key additions beyond the original design doc:

• Detailed Go types (VerificationResult, ExperienceEntry, ExperienceStats, RecallFeedback, EncodingQualityWindow)
• Complete store interface additions with method signatures
• Entity extraction regex patterns (ported from eval_faithfulness.py)
• Template echo phrase list
• Systemd unit files with trap-based failure recovery (daemon always restarts)
• Training script pseudostructure with hyperparameter justification table
• Testing strategy: unit tests, parity test (Go vs Python), integration tests
• Full config block with defaults and comments
• Risk/mitigation matrix with phase assignments
• Dependency table with current status

Next: Writing detailed implementation plan (step-by-step, file-by-file).

[CalebisGross]

Phase A Implementation Plan Complete

Plan committed: docs/PLAN_continuous_learning_phase_a.md at 5aa702d

12 Tasks, TDD throughout:

Task	What	Files
1	Schema migration (007)	migrations/007_continuous_learning.sql, schema.go
2	Store interface — ContinuousLearningStore	store.go (types + interface)
3	SQLite implementation	sqlite/continuous_learning.go
4	Verification gate (EPR, TED, FR, MIG)	encoding/verification.go
5	Verification tests	encoding/verification_test.go
6	Wire into encoding pipeline	encoding/agent.go
7	Hook feedback into experience buffer	mcp/server.go
8	Dreaming reclassification phase	dreaming/agent.go
9	Metacognition quality drift	metacognition/agent.go
10	Configuration	config/config.go, config.yaml
11	Build, test, deploy to daemon	Integration validation
12	Go/Python parity test	Verify EPR matches eval_faithfulness.py

Each task has exact file paths, complete code blocks, test commands, and commit messages. Designed for subagent execution.

Phase B and C plans will be written after Phase A ships and we have baseline quality data.

[CalebisGross]

Phase A Progress — Tasks 1-10 Complete

Runtime verification and experience collection infrastructure is implemented and passing all tests.

What shipped:

• Schema migration 007: encoding_epr/encoding_fr/encoding_flags columns on memories, plus recall_feedback and experience_buffer tables
• Store interface: ContinuousLearningStore with 9 methods (verification results, experience CRUD, recall linkage, quality window)
• SQLite implementation: Full implementation with reclassification logic
• Verification gate: Go port of eval_faithfulness.py entity extraction (numbers, paths, versions, proper nouns, CamelCase, @mentions) + EPR/FR/TED/MIG checks. Wired into encoding pipeline — every encoding now runs through verifyFaithfulness() after compression
• MCP feedback hookup: feedback tool now propagates to experience buffer (running weighted average) and records recall-encoding linkage
• Dreaming reclassification: Phase 4.5 reclassifies experience buffer entries (gold/needs_improvement/ambiguous) during dreaming cycles
• Metacognition drift detection: New encoding_quality_drift observation type — warnings at EPR < 0.85 or TED > 5%, critical at EPR < 0.70 or TED > 15%
• Config: ContinuousLearningConfig with training/trigger settings, disabled by default

Test status:

• 11 new unit tests (entity extraction, EPR, TED, MIG)
• 19/20 test packages pass (1 pre-existing failure unrelated to this work)
• golangci-lint run: 0 issues

Remaining:

• Task 12: Go/Python parity test on EXP-25 gold probes (optional validation)
• Phases B and C (Gemini re-encoding, automated training) are future work

[CalebisGross]

Phase A Complete — All 12 Tasks Done

Update to previous comment: Tasks 11-12 are now also complete.

• Task 11 (build/test/lint): 20/20 test packages pass, 0 golangci-lint issues
• Task 12 (Go/Python parity test): 25/25 gold probes within tolerance. Go entity extraction is slightly more conservative on dense-number inputs (24/25 within 15%, 25/25 within 20%)
• Migration fix: Inline migration in InitSchema (migration SQL files are reference only). Tables verified live — first encoding scored EPR 1.0
• Gemma chat template: Added chat_template config field to embedded provider ("chatml" for Qwen, "gemma" for Gemma 4 E2B). Full inference chain ready for Gemma spoke deployment

Verification gate is live in production — every encoding through the daemon now gets EPR/FR/TED/MIG scoring with results persisted to the experience buffer.

Phase A observation period begins now. Phases B and C depend on data collected during this period.