feat: degenerate repetition / collapse detection

[Jamie-BitFlight]

Summary

Detect degenerate repetition (looping/collapse) in assistant output where near-identical sentences appear multiple times. This is a specific LLM failure mode indicating low-quality generation.

Technique (from ObvioSpectre/hallucination-detector)

Check for sentence pairs with high textual similarity (>95% overlap) as an indicator of looping.

Regex-Only Approximation

Without embeddings, detect repetition via:

1. Split into sentences
2. Normalize (lowercase, strip whitespace)
3. Check for exact or near-exact duplicates (e.g., Levenshtein distance < 10% of sentence length, or longest common substring > 80%)

// Simple exact-duplicate detection
const sentences = text.split(/(?<=[.!?])\s+/);
const normalized = sentences.map(s => s.toLowerCase().trim());
const duplicates = normalized.filter((s, i) => normalized.indexOf(s) !== i);

Why This Works

When an LLM loops or produces degenerate output, it repeats the same sentence or paragraph with minor variations. This is distinct from intentional repetition (e.g., numbered lists with similar structure) and is a strong signal of generation failure.

New Category

degenerate_repetition — detects looping/collapse in assistant output.

Acceptance Criteria

• Sentence-level duplicate detection
• Near-duplicate detection (not just exact matches)
• Suppression for intentional repetition (numbered lists, table rows, code)
• Configurable similarity threshold
• Tests with looping examples and legitimate repetition controls

References

• ObvioSpectre/hallucination-detector detectors/consistency.py (similarity > 0.95 check)

[Jamie-BitFlight]

Grouping: Group C — Sentence-Level Detection

Implement together with: #4, #19

Rationale: Uses the same sentence-splitting infrastructure as #4/#19. The normalized sentence array and duplicate detection logic complement the contradiction detection — both operate on sentence pairs.

Dependency: The sentence splitter built here becomes the foundation for #21 (sentence-level scoring).

[Jamie-BitFlight]

Deep Analysis: Implementation Impact

Classification

• Impact type: ADDITIVE (depends on sentence splitting from feat: self-contradiction detection via negation polarity #19)
• Risk: Low
• Phase: 3 — Sentence infrastructure

Performance consideration

Near-duplicate detection (Levenshtein or longest common substring) is O(n^2) on sentence pairs. For a typical assistant response (10–30 sentences), this is negligible. For degenerate output (which this detects), the response may have hundreds of repeated sentences — the quadratic cost is bounded by the detection itself.

False positive suppression

The issue identifies three suppression targets: numbered lists, table rows, and code blocks. Code blocks are already stripped by `stripLowSignalRegions()`. Numbered lists and table rows need new suppression logic — detect structured repetition patterns (e.g., lines matching `/^\d+.\s/` or `/^|.*|$/`).

Sequencing

• Depends on: sentence splitter (from feat: self-contradiction detection via negation polarity #19 or feat: sentence-level granularity and weighted multi-signal scoring #21)
• Blocked by: nothing strictly (can implement its own splitter)
• Implement together with: feat: add self-contradiction detection #4, feat: self-contradiction detection via negation polarity #19 (shared sentence infrastructure)

State: None | Dependencies: None | Contract change: None (new kind only)