-
Notifications
You must be signed in to change notification settings - Fork 0
Open
0 / 60 of 6 issues completedOpen
0 / 60 of 6 issues completed
Copy link
Labels
enhancementNew feature or requestNew feature or requestimpact: infrastructureAdds new subsystem (config, MCP, state). Medium risk.Adds new subsystem (config, MCP, state). Medium risk.phase: 2-config-foundationPhase 2: Config system (unlocks phases 4-8)Phase 2: Config system (unlocks phases 4-8)risk: mediumMedium risk — new subsystem or structural changeMedium risk — new subsystem or structural changetopic: configConfiguration systemConfiguration system
Description
Summary
Add a configuration system that loads settings from multiple file formats and locations with cascading priority, enabling per-project customization of detection behavior without modifying the core script.
Config File Sources (Priority Order)
Load and merge config from these locations, highest priority wins:
- CLI/environment —
HALLUCINATION_DETECTOR_CONFIGenv var pointing to a file path - Project root —
.hallucination-detectorrc.cjs(CJS module, allows dynamic config) - Project root —
project.json→hallucination-detectorkey - Project root —
pyproject.toml→[tool.hallucination-detector]section - Home directory —
~/.hallucination-detectorrc.cjs(user-level defaults) - Built-in defaults — current hardcoded values (zero-config still works)
File Format Examples
.hallucination-detectorrc.cjs (most flexible):
module.exports = {
categories: {
speculation_language: {
enabled: true,
severity: 'warning', // 'error' | 'warning' | 'info'
customPatterns: [
{ pattern: /\bperhaps\b/i, evidence: 'perhaps' },
{ pattern: /\bmight be\b/i, evidence: 'might be' },
],
responseTemplate: 'STOP: Speculation detected — "${evidence}" at position ${offset}. Restate using observed facts.',
},
pseudo_quantification: {
enabled: false, // disable entire category
},
},
};pyproject.toml:
[tool.hallucination-detector]
severity = "warning"
[tool.hallucination-detector.categories.speculation_language]
enabled = true
severity = "error"
custom-patterns = [
{ pattern = "\\bperhaps\\b", evidence = "perhaps" },
]
[tool.hallucination-detector.categories.pseudo_quantification]
enabled = falseproject.json:
{
"hallucination-detector": {
"severity": "warning",
"categories": {
"speculation_language": {
"enabled": true,
"customPatterns": [
{ "pattern": "\\bperhaps\\b", "evidence": "perhaps" }
]
}
}
}
}Configurable Settings
Global Settings
severity— default severity for all categories ('error'|'warning'|'info'). Default:'error'maxTriggersPerResponse— max trigger matches before stopping analysis (performance guard). Default:20maxBlocksPerSession— cumulative blocks before escalating or suppressing. Default: unlimitedoutputFormat— response format ('text'|'json'|'jsonl'). Default:'text'debug— enable verbose logging to stderr. Default:false
Per-Category Settings
enabled— enable/disable individual categories. Default:truefor allseverity— override global severity per categorycustomPatterns— array of{ pattern, evidence }objects to add to built-in patternsreplacePatterns— iftrue,customPatternsreplaces built-in patterns instead of extending. Default:falseresponseTemplate— custom response template string with${evidence},${offset},${kind}interpolationsuppressionRules— custom suppression rules (e.g., additional evidence markers, context patterns)
Filtering Settings
ignorePatterns— glob patterns for file paths to skip when scanning (e.g.,['**/test/**', '*.spec.*'])ignoreBlocks— additional code block markers to strip (beyond markdown fences)evidenceMarkers— custom strings that count as "evidence nearby" for causality suppression (e.g.,['according to', 'as shown in'])allowlist— phrases that should never trigger (exact match or regex)
Response Settings
responseTemplates— override all response templates per categoryincludeContext— include surrounding text in trigger reports. Default:truecontextLines— number of surrounding lines to include. Default:2
Implementation Notes
- Config loading should happen once at startup, not per-invocation
- Use
require()for.cjsfiles (already CJS environment) - Parse TOML with a lightweight parser (no heavy dependencies — consider bundling a minimal parser or requiring
@iarna/tomlas optional peer dep) - JSON parsing is built-in
- Deep merge configs with later sources overriding earlier ones
- Validate config schema and emit clear error messages for invalid config
- Export
loadConfig()andmergeConfig()for testability
Acceptance Criteria
- Zero-config still works identically to current behavior
-
.hallucination-detectorrc.cjsloaded from project root -
pyproject.toml[tool.hallucination-detector]section parsed -
project.jsonhallucination-detectorkey parsed -
~/.hallucination-detectorrc.cjsloaded as user defaults -
HALLUCINATION_DETECTOR_CONFIGenv var overrides all file sources - Custom patterns extend built-in patterns by default
-
replacePatterns: truereplaces instead of extending - Per-category enable/disable works
- Per-category severity override works
- Custom response templates with variable interpolation
- Allowlist prevents specific phrases from triggering
- Config validation with clear error messages
- Tests cover config loading, merging, and override behavior
- Documentation updated with config file reference
Related Issues
- Extends all detection categories (feat: add absolute certainty language detection #2–feat: add methodology claims without detail detection #9) by making them configurable
Reactions are currently unavailable
Sub-issues
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestimpact: infrastructureAdds new subsystem (config, MCP, state). Medium risk.Adds new subsystem (config, MCP, state). Medium risk.phase: 2-config-foundationPhase 2: Config system (unlocks phases 4-8)Phase 2: Config system (unlocks phases 4-8)risk: mediumMedium risk — new subsystem or structural changeMedium risk — new subsystem or structural changetopic: configConfiguration systemConfiguration system