Add setup eval suite, auto-doc pipeline, and verification phase#10
Merged
jrenaldi79 merged 12 commits intomainfrom Mar 24, 2026
Merged
Add setup eval suite, auto-doc pipeline, and verification phase#10jrenaldi79 merged 12 commits intomainfrom
jrenaldi79 merged 12 commits intomainfrom
Conversation
- Replace all em dashes with commas, periods, or colons
- Remove bold from marketing-style callout sentences
- Replace banned/sloppy words (field guide → reference, surgical → specific/targeted,
batteries-included → comprehensive, non-negotiable → required, going off the rails)
- Vary "mechanical enforcement" repetition (use "automated checks", "enforcement", etc.)
- Tighten formulaic structures ("This repo is two things" → "This repo contains",
colon-then-parallel-clauses → two sentences)
- Reduce redundant Karpathy quote paraphrases in mapping table
- Remove unnecessary bold emphasis on imperative sentences
https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
Made small edits
…sized test - Create CLAUDE.md with Commands, Architecture, Quality Gates, Code Review Checklist, Critical Gotchas, and Docs Map sections - Add .claude/settings.json with allow/deny lists for this repo's commands - Add .claude/rules/code-quality.md and tdd.md path-scoped rules - Split 491-line generate-docs.test.js into two files under 300 lines: generate-docs.test.js (243 lines) for marker/link/plan tests generate-docs-helpers.test.js (262 lines) for tree/module/jsdoc tests Addresses gaps found by /readiness analysis: missing CLAUDE.md (Pillar 4), no agent config or rules (Pillar 5), and file size violation (Pillar 6). https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
This repo has no package.json at root (it's a plugin, not an npm package). The lockfile is an artifact from npx invocations and should not be tracked. https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
- Add scripts/hooks/pre-commit: secret scanning + 300-line file size check - Add scripts/hooks/pre-push: doc drift detection + smart SHA-based caching - Add scripts/install-hooks.sh to wire hooks into .git/hooks/ - Wrap CLAUDE.md Architecture and Key Modules sections with AUTO markers - Fix module paths in Key Modules table (use full paths from repo root) - Add SessionStart hook to .claude/settings.json for drift detection - Add gotcha about two sets of hooks (repo's own vs. templates for users) Addresses remaining Level 4+ gaps: active git hooks (Pillar 3), AUTO markers (Pillar 4), enforcement hierarchy (Pillar 5), and session-start validation (Pillar 8). https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
validate-docs.js is designed for target projects (expects src/ directory and "Directory Structure" section). Replace with file size re-check across this repo's actual JS directories. Fix SessionStart hook to check repo state and hook installation instead of running target-project scripts. https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
Distinguishes scripts/ (internal repo hooks) from skills/setup/scripts/ (templates shipped to user projects by /setup). https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
Extends the existing eval infrastructure to cover the /setup skill: - setup-grader.js: validates file creation, JSON validity, hook executability, CLAUDE.md sections, settings structure, rule frontmatter - setup-eval-config.json: 2 test cases (greenfield + existing project) - Fixtures: setup-bare (empty dir) and setup-existing-node (Express app) - run-evals.sh: --config flag, per-test-case prompts, setup artifact capture https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
Tree and modules sections now reflect the setup eval infrastructure: setup-grader.js, setup-eval-config.json, run-evals.sh, and fixtures. https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
The AUTO:tree and AUTO:modules markers were hand-maintained because generate-docs.js targets user projects, not this repo. Added: - scripts/repo-generate-docs.js: repo-specific wrapper that scans skills/, scripts/, tests/ and builds tree + module index - Pre-commit hook now runs repo-generate-docs.js to auto-regenerate - Fixed extractJSDocDescription to try multi-line before single-line (was picking up @param lines instead of file-level descriptions) - Added 'results' to SKIP_DIRS so eval output dirs are excluded https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
The test cache check printed "skipping" but never exited, so tests ran anyway. Added exit 0 after the cache hit message. Added .env, .env.local, coverage/, dist/, build/ to .gitignore to match the patterns /setup installs for user projects. https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
The skill now runs 6 smoke checks after installation before presenting the summary: hook executability, enforcement scripts, CLAUDE.md sections, agent config validity, auto-documentation pipeline, and linter. Failures are fixed inline before proceeding. Summary renumbered to Phase 7 and now includes verification results. https://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (25)
📝 WalkthroughWalkthroughThe PR establishes comprehensive enforcement and evaluation infrastructure for the project. It introduces code quality and TDD rules, adds git hooks for secret scanning and file-size validation, creates a documentation generation system, extends the evaluation framework to support the Changes
Sequence Diagram(s)sequenceDiagram
participant Developer as Developer<br/>(git commit)
participant PreCommit as Pre-commit Hook<br/>(scripts/hooks/pre-commit)
participant SecretChk as Secret Checker<br/>(check-secrets.js)
participant SizeChk as Size Checker<br/>(line count validation)
participant DocGen as Doc Generator<br/>(repo-generate-docs.js)
participant Git as Git<br/>(staging area)
Developer->>PreCommit: Trigger on staged files
PreCommit->>SecretChk: Run secret scanning
SecretChk-->>PreCommit: Pass/Fail
alt secrets detected
PreCommit-->>Developer: Abort commit
end
PreCommit->>SizeChk: Check staged JS files<br/>for > 300 lines
SizeChk-->>PreCommit: Violations list
alt violations found
PreCommit-->>Developer: Abort commit
end
PreCommit->>DocGen: Regenerate markers<br/>(AUTO:tree, AUTO:modules)
DocGen->>Git: Update CLAUDE.md
DocGen-->>PreCommit: Success
PreCommit-->>Developer: Allow commit
sequenceDiagram
participant User as Claude Agent<br/>(/setup skill)
participant Skill as Setup Skill<br/>(SKILL.md flow)
participant Generator as Generate Docs<br/>(repo-generate-docs.js)
participant Hooks as Install Hooks<br/>(install-hooks.sh)
participant Grader as Setup Grader<br/>(setup-grader.js)
participant Validator as Validation Engine<br/>(checks: files, JSON, sections)
User->>Skill: Request project setup
Skill->>Generator: Auto-generate documentation
Generator-->>Skill: Update CLAUDE.md markers
Skill->>Hooks: Install git hooks
Hooks-->>Skill: Hooks installed & executable
Skill->>Skill: Phase 6: Verify Setup<br/>(6 smoke tests)
alt verification fails
Skill-->>User: Fix issues & re-run
end
Skill->>Grader: Provide artifact directory
Grader->>Validator: Run file/JSON/section checks
Validator-->>Grader: Accumulate check results
Grader-->>User: Return pass/fail score<br/>& details
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Poem
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
setup-eval-config.json,setup-grader.js, and 2 fixtures (setup-bare,setup-existing-node) that test the /setup skill's file creation, JSON validity, hook executability, CLAUDE.md sections, and agent config structure. Extendedrun-evals.shwith--configflag and per-test-case prompt overrides.scripts/repo-generate-docs.jsregeneratesAUTO:treeandAUTO:modulesmarkers in CLAUDE.md on every commit via the pre-commit hook. FixedextractJSDocDescriptionto prefer multi-line JSDoc over single-line (was picking up@paramlines instead of file-level descriptions). AddedresultstoSKIP_DIRS.exit 0on cache hit,.gitignoresafety patterns (.env,coverage/,dist/,build/).Test plan
npx jest --config '{}' tests/scripts/— all 85 unit tests pass (including 7 new repo-generate-docs tests)bash tests/evals/run-evals.sh --dry-run— readiness evals still work with default configbash tests/evals/run-evals.sh --config setup-eval-config.json --dry-run— setup evals resolve correctlynode tests/evals/setup-grader.jsvalidates file existence, JSON validity, hooks, CLAUDE.md sections, settings structure, rule frontmatterhttps://claude.ai/code/session_01Hbxy31TkbujzukGFSxLcPw
Summary by CodeRabbit
Release Notes
Documentation
Tests
Chores