Rename rule types from semi-objective/subjective to check/judge by oshorefueled · Pull Request #49 · TRocket-Labs/vectorlint

oshorefueled · 2026-01-02T22:56:08Z

BREAKING CHANGE: Rule type terminology has changed

'semi-objective' → 'check'
'subjective' → 'judge'

Backward compatibility is maintained via Zod schema transform. Existing configs using old type names will continue to work.

Changes:

Update EvaluationType enum with new JUDGE/CHECK values
Add Zod transform to map deprecated values to new canonical values
Rename internal methods to runJudgeEvaluation/runCheckEvaluation
Update all VectorLint preset rules to use type: check
Update CREATING_RULES.md, CONFIGURATION.md, README.md documentation
Fix missing 'pack' property in test fixtures
Apply npm audit fixes for glob and js-yaml vulnerabilities
Bump version to 2.2.0

Summary by CodeRabbit

Release Notes - v2.2.0

Documentation
- Updated terminology across configuration documentation, rule creation guides, and README for enhanced clarity and consistency.
- Refined descriptions of scoring methodologies and evaluation approaches.
Chores
- Version bump to 2.2.0.
- Updated preset definitions and metadata with new terminology.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

…k/judge BREAKING CHANGE: Rule type terminology has changed - 'semi-objective' → 'check' - 'subjective' → 'judge' Backward compatibility is maintained via Zod schema transform. Existing configs using old type names will continue to work. Changes: - Update EvaluationType enum with new JUDGE/CHECK values - Add Zod transform to map deprecated values to new canonical values - Rename internal methods to runJudgeEvaluation/runCheckEvaluation - Update all VectorLint preset rules to use type: check - Update CREATING_RULES.md, CONFIGURATION.md, README.md documentation - Fix missing 'pack' property in test fixtures - Apply npm audit fixes for glob and js-yaml vulnerabilities - Bump version to 2.2.0

coderabbitai · 2026-01-02T22:56:16Z

📝 Walkthrough

Walkthrough

This PR renames evaluation type semantics across the codebase from "Semi-Objective/Subjective" to "Check/Judge," updating documentation, preset rules, source code (evaluators, schemas, scoring), and tests. Version bumped to 2.2.0. Backward compatibility is preserved in schema validation via a transform mapping deprecated values.

Changes

Cohort / File(s)	Summary
Documentation & Release `CONFIGURATION.md`, `CREATING_RULES.md`, `README.md`, `package.json`	Terminology replaced: "Semi-Objective Rules" → "Check Rules", "Subjective Rules" → "Judge Rules"; scoring descriptions and examples updated to reflect new semantics; version bumped from 2.1.1 to 2.2.0.
Preset Rule Definitions `presets/VectorLint/ai-pattern.md`, `presets/VectorLint/directness.md`, `presets/VectorLint/pseudo-advice.md`, `presets/VectorLint/repetition.md`	Type field changed from `semi-objective` to `check` in rule frontmatter metadata.
Evaluation Type Definitions `src/evaluators/types.ts`	EvaluationType enum members replaced: `SUBJECTIVE` → `JUDGE`, `SEMI_OBJECTIVE` → `CHECK`.
Evaluator Routing & Methods `src/evaluators/base-evaluator.ts`	Method renames: `runSubjectiveEvaluation` → `runJudgeEvaluation`, `runSemiObjectiveEvaluation` → `runCheckEvaluation`; `getEvaluationType()` now returns `JUDGE` or `CHECK`; routing logic in `evaluate()` updated accordingly.
Schema & Type Guards `src/prompts/schema.ts`	Schema builder names and result type discriminators updated: `buildSubjectiveLLMSchema` returns `vectorlint_judge_result`, `buildSemiObjectiveLLMSchema` returns `vectorlint_check_result`; `SubjectiveResult.type` now uses `JUDGE`, `SemiObjectiveResult.type` now uses `CHECK`; type guards updated.
Prompt Schema Validation `src/schemas/prompt-schemas.ts`	Type field enum expanded to include both new (`judge`, `check`) and deprecated (`subjective`, `semi-objective`) values; inline transform maps deprecated values to canonical forms for backward compatibility.
Scoring Result Labels `src/scoring/scorer.ts`	Result type labels updated in multiple scoring functions: `SEMI_OBJECTIVE` → `CHECK`, `SUBJECTIVE` → `JUDGE` in returned object type fields; documentation comments updated.
Test Suites & Fixtures `tests/scoring-types.test.ts`	Test suite names renamed ("Subjective" → "Judge", "Semi-Objective" → "Check"); prompt fixtures updated with new type values and pack metadata; result type assertions updated to expect `JUDGE`/`CHECK`.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

feat(token-usage): Add token usage tracking and cost calculation #40: Modifies the same evaluator result types and schema files (src/prompts/schema.ts, src/evaluators/base-evaluator.ts) by adding token-usage fields and LLM result wrapping alongside this PR's terminology refactor.
feat: implement content chunking and dedicated scoring logic for evaluators #39: Alters the same evaluation and scoring code paths (src/evaluators/*, src/scoring/*) with chunking logic and function implementations that interact with the renamed evaluation types and result structures.
feat: implement bundled rule packs (presets) #32: Updates bundled presets and prompt schema handling that directly relate to this PR's preset file changes and schema normalization in src/schemas/prompt-schemas.ts.

Suggested reviewers

ayo6706

Poem

🐰 From Subjective dreams to Judging days,
Semi-Objective rules now Check their ways,
Hopping through code with nomenclature cheer,
A lexicon refresh, crystal clear!
✨

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 37.50% which is insufficient. The required threshold is 80.00%.	You can run `@coderabbitai generate docstrings` to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: renaming rule types from semi-objective/subjective to check/judge, which is the primary breaking change throughout the codebase and documentation.

✨ Finishing touches

📝 Generate docstrings

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Actionable comments posted: 0

🧹 Nitpick comments (1)

src/evaluators/base-evaluator.ts (1)
32-42: Consider updating the class-level documentation to reflect new terminology.

The class documentation still references "Subjective and Semi-Objective" terminology while the code now uses "judge" and "check". Consider updating for consistency with the rest of the PR.
🔎 Suggested documentation update
 /*
- * Core LLM-based evaluator that handles Subjective and Semi-Objective evaluation modes.
+ * Core LLM-based evaluator that handles Judge and Check evaluation modes.
  * Mode is determined by prompt frontmatter 'type' field:
- * - 'subjective': Weighted average of 1-4 scores per criterion, normalized to 1-10.
- * - 'semi-objective': Density-based scoring (errors per 100 words).
+ * - 'judge': Weighted average of 1-4 scores per criterion, normalized to 1-10.
+ * - 'check': Density-based scoring (errors per 100 words).
  *
  * Content is automatically chunked for documents >600 words to improve accuracy.
  *
  * Subclasses can override protected methods to customize evaluation behavior
  * while reusing the core evaluation logic.
  */

📜 Review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 245156a and ebdc0ed.

⛔ Files ignored due to path filters (1)

package-lock.json is excluded by !**/package-lock.json

📒 Files selected for processing (14)

CONFIGURATION.md
CREATING_RULES.md
README.md
package.json
presets/VectorLint/ai-pattern.md
presets/VectorLint/directness.md
presets/VectorLint/pseudo-advice.md
presets/VectorLint/repetition.md
src/evaluators/base-evaluator.ts
src/evaluators/types.ts
src/prompts/schema.ts
src/schemas/prompt-schemas.ts
src/scoring/scorer.ts
tests/scoring-types.test.ts

🧰 Additional context used

📓 Path-based instructions (2)

src/**/*.ts

📄 CodeRabbit inference engine (AGENTS.md)

src/**/*.ts: Use TypeScript ESM with explicit imports and narrow types
Use 2-space indentation; avoid trailing whitespace
Maintain strict TypeScript with no any; use unknown + schema validation for external data
Use custom error types with proper inheritance; catch blocks use unknown type

Files:

src/schemas/prompt-schemas.ts
src/evaluators/types.ts
src/scoring/scorer.ts
src/evaluators/base-evaluator.ts
src/prompts/schema.ts

tests/**/*.test.ts

📄 CodeRabbit inference engine (AGENTS.md)

tests/**/*.test.ts: Write tests using Vitest framework with focus on config parsing, file discovery, schema/structured output, and locator
Use dependency injection in tests: mock providers; do not hit network in unit tests

Files:

tests/scoring-types.test.ts

🧠 Learnings (8)

📓 Common learnings

Learnt from: CR
Repo: TRocket-Labs/vectorlint PR: 0
File: AGENTS.md:0-0
Timestamp: 2025-12-28T19:43:51.189Z
Learning: Applies to evals/**/*.md : Eval YAML should use `name` (human), `id` (PascalCase), and criteria `id` (PascalCase)