feat: optional RAG verification tier via Exa.ai or similar

[Jamie-BitFlight]

Summary

Add an optional second-pass verification tier that uses RAG (Retrieval-Augmented Generation) to fact-check claims against real-world sources, complementing the existing pattern-based linguistic analysis.

Motivation

The current detector catches linguistic red flags (speculation, pseudo-quantification, etc.) — patterns that suggest unreliability. But it cannot verify whether a factual claim is actually true. A RAG tier would catch factual errors that sound confident.

Example the current detector misses:

"React 19 was released in March 2024 and introduced Server Actions."

This contains no speculation language — it sounds authoritative. But if the date or feature attribution is wrong, only source verification can catch it.

Approach (inspired by agenticassets/exa-hallucination-detector)

Three-stage pipeline:

1. Claim extraction — identify verifiable factual assertions from assistant output
2. Source retrieval — search for corroborating/contradicting sources (Exa.ai, web search, or local docs)
3. Verification — compare claims against retrieved sources, produce True/False/Insufficient verdict

Design Constraints

• Must be optional — zero-dependency pattern matching remains the default tier
• Must not slow down normal operation — RAG verification should be explicitly opted into or run as a separate pass
• Must support multiple backends — Exa.ai, WebSearch, local document search, custom API
• Must degrade gracefully — if no API key configured, skip RAG tier silently

Configuration

// .hallucination-detectorrc.cjs
module.exports = {
  ragVerification: {
    enabled: false,
    backend: 'exa',        // 'exa' | 'websearch' | 'local' | 'custom'
    apiKey: process.env.EXA_API_KEY,
    maxClaimsPerCheck: 5,
    confidenceThreshold: 70, // 0-100, below this → flag as unverified
  },
};

Acceptance Criteria

• Claim extraction works on assistant text output
• At least one backend (web search) works without paid API keys
• Results integrate with existing trigger match output format
• Zero impact on performance when disabled (default)
• Tests cover extraction, verification, and graceful degradation

Related Issues

• feat: config file handling — cascading settings from multiple sources #10 (config file handling — needed for RAG settings)
• feat: add short-termism / temporal bias detection #5 (source grounding — overlaps with this feature)

References

• agenticassets/exa-hallucination-detector — RAG-based approach using Exa.ai + Claude

[Jamie-BitFlight]

Grouping: Group F — Advanced Opt-In Features

Implement together with: #15

Rationale: Both are large, opt-in features that extend the detector beyond regex pattern matching:

• feat: optional RAG verification tier via Exa.ai or similar #11 adds RAG-based fact-checking against external sources
• feat: cognitive bias detection from LLM financial reasoning research #15 adds cognitive bias detection from LLM financial reasoning research

Both require:

• Opt-in configuration (disabled by default)
• External API integration or complex heuristics
• Graceful degradation when dependencies unavailable

Dependencies: Both require #10 (config) to be implemented first for the opt-in settings.

[Jamie-BitFlight]

Deep Analysis: Implementation Impact

Classification

• Impact type: EXTERNAL — network I/O, API keys, new dependencies
• Risk: HIGH — introduces async into a synchronous pipeline, timeout risk
• Phase: 7 — External/isolated (implement as separate module/package)

Breaks zero-dependency

The project currently has zero runtime dependencies (only node:fs, node:os, node:path). This issue requires:

• HTTP client (Exa SDK or fetch in Node 18+)
• API key management (EXA_API_KEY)
• Optional: embedding library for local-docs backend

Implementation constraint: Must use lazy require() with try/catch, optionalDependencies, or a separate sub-package with its own package.json. Must NOT add to root dependencies.

Async in a synchronous pipeline

The Stop hook is fully synchronous (fs.readFileSync, JSON.parse, regex matching). RAG verification requires HTTP requests. Options:

1. Convert main() to async — changes the entry point fundamentally
2. Use child_process.execSync with curl — blocks but synchronous
3. Run as a separate post-processing step — decoupled but more complex

All options introduce latency. Claude Code has a hook timeout. If RAG hangs (DNS failure, API down, slow response), the hook times out → Claude Code kills it → treated as "allow" → detection silently disabled for that response.

Sequencing

• Blocked by: feat: config file handling — cascading settings from multiple sources #10 (config for ragVerification.* settings)
• Independent of: feat: corrected text suggestions alongside flags #13, feat: structured JSON output mode for programmatic consumption #17 (can be parallel after feat: config file handling — cascading settings from multiple sources #10)
• Benefits from: feat: sentence-level granularity and weighted multi-signal scoring #21 (sentence-level claims are better inputs for claim extraction than whole-response text)

Failure modes

• Network timeout → hook timeout → fail-open (silent)
• API key missing/invalid → must skip silently, not crash
• API rate limit → must skip silently
• API returns garbage → must not corrupt existing pattern-matching results
• Exa SDK not installed → must skip silently (lazy require)

Graceful degradation checklist (from issue)

The issue explicitly requires: "if no API key configured, skip RAG tier silently." This must be verified for:

• Missing API key
• Missing npm package
• Network unreachable
• API returns error
• API returns slowly (> N seconds)