Add file-type context and tighten PAT confidence guidance#67
Merged
oshorefueled merged 3 commits intomainfrom Mar 4, 2026
Merged
Add file-type context and tighten PAT confidence guidance#67oshorefueled merged 3 commits intomainfrom
oshorefueled merged 3 commits intomainfrom
Conversation
…dance
- Rewrite default directive with goal, content_metadata, context,
evaluation_boundaries, and surfacing_criteria sections to reduce
false positives in structured-format files (MDX, etc.)
- Plumb file extension as EvalContext through BaseEvaluator →
runPromptStructured → RequestBuilder → {{file_type}} substitution
in the directive system prompt
- Fix {{file_type}} placeholder always replaced (empty string when
absent) to prevent literal leakage into prompts; add regression tests
- Tighten PAT schema descriptions for plausible_non_violation and
confidence to calibrate LLM scoring below 0.7 for ambiguous matches
- Expand VectorLint preset from 3 to 8 rules: add Capitalization,
Consistency, Inclusivity, PassiveVoice, UnsupportedClaims, Wordiness;
remove Directness and PseudoAdvice
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (10)
📝 WalkthroughWalkthroughThis PR adds seven new VectorLint checks, removes two deprecated presets, standardizes preset front-matter, adds many test fixtures, and threads an EvalContext (fileType) through evaluators, request builders, and LLM providers so prompts can be templated per file type. Changes
Sequence Diagram(s)sequenceDiagram
participant Caller as BaseEvaluator.evaluate(file, content)
participant Ext as Path.extname
participant Ctx as EvalContext(fileType)
participant Eval as runJudgeEvaluation / runCheckEvaluation
participant LLM as LLMProvider.runPromptStructured
participant RB as RequestBuilder.buildPromptBodyForStructured
participant Provider as VercelAIProvider
Caller->>Ext: extract file extension
Ext-->>Ctx: create context { fileType }
Caller->>Eval: call with content + context
Eval->>LLM: runPromptStructured(content, prompt, schema, context)
LLM->>RB: buildPromptBodyForStructured(originalBody, context)
RB->>RB: replace {{file_type}} with context.fileType
RB-->>LLM: return templated system prompt
LLM->>Provider: provider sends prompt to model (context-aware)
Provider-->>Eval: structured LLM result
Eval-->>Caller: aggregated evaluation result
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Suggested reviewers
Poem
🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
String.prototype.replace(string) only substitutes the first match.
Switch to replaceAll to ensure custom directives with multiple
{{file_type}} tokens are fully resolved; add regression test covering
a directive with two occurrences.
New rules get 1.0.0; AIPattern and Repetition get 2.0.0 to reflect their rewrites from the previous commit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The LLM directive gave no guidance on file structure or confidence calibration, causing excessive false positives — especially in MDX files where structural elements like cards and tabs are not continuous prose. Confidence scores were also under-constrained, surfacing too many marginal findings.
What
<goal>,<content_metadata>,<context>,<evaluation_boundaries>, and<surfacing_criteria>sections{{file_type}}so the LLM interprets structural boundaries correctly for MDX vs plain MarkdownEvalContextthroughBaseEvaluator→runPromptStructured→RequestBuilder→ directive substitution{{file_type}}placeholder is now unconditionally replaced (empty string when absent) — previously it leaked as a literal string when no file type was providedplausible_non_violationandconfidenceto push ambiguous findings below 0.7VectorLintpreset from 3 rules to 8: addCapitalization,Consistency,Inclusivity,PassiveVoice,UnsupportedClaims,Wordiness; removeDirectnessandPseudoAdviceScope
In scope
{{file_type}}templatingBaseEvaluator→LLMProvider→RequestBuilderBehavior impact
RunRules=VectorLintget 8 checks instead of 3;DirectnessandPseudoAdvicechecks no longer run — existing configs referencing them will silently produce no findings for those rulesRisk
DirectnessandPseudoAdviceare deleted without a deprecation notice; communicate in release notesmeta.jsoncount: description says "7 built-in checks" but the directory contains 8.mdfiles — cosmetic, but worth correctingHow to test / verify
Checks run
npx vitest run→ 31 test files, 218 tests passed, 0 failuresManual verification
ls presets/VectorLint/*.md | wc -l→8.mdxfile:npm run dev -- <path>.mdx --rules VectorLint/AIPattern— confirm no{{file_type}}literal in output or verbose prompt logs{{file_type}}in the promptDirectnessandPseudoAdviceno longer appear:ls presets/VectorLint/ | grep -E "directness|pseudo"→ no outputRollback
Revert to
main. No config mutations, migrations, or persistent state — rollback is immediate with no data impact. Users who relied onDirectnessorPseudoAdvicewill need to restore those files manually from git history.Summary by CodeRabbit
New Features
Changes
Removals