Skip to content

test: add integration canaries for at, diff, review --batch, merge#221

Merged
flyingrobots merged 4 commits intomainfrom
ironclad/canary-expansion
Feb 14, 2026
Merged

test: add integration canaries for at, diff, review --batch, merge#221
flyingrobots merged 4 commits intomainfrom
ironclad/canary-expansion

Conversation

@flyingrobots
Copy link
Owner

@flyingrobots flyingrobots commented Feb 14, 2026

Summary

  • Adds 4 new CLI integration canary tests addressing CodeRabbit round-3 feedback from PR feat: IRONCLAD M9.T1 — CLI JSON Schema contracts #217
  • Tests validate at, diff, review --batch, and merge commands against their JSON Schema contracts
  • suggest is intentionally skipped (requires GITMIND_AGENT env var / LLM agent)
  • Test setup enhanced with epoch markers (via process-commit) and a second temp repo for merge testing
  • 371 tests across 22 files (was 367)

Closes #205

Test plan

  • All 371 tests pass locally (npx vitest run)
  • New canary tests validate against their respective JSON Schema contracts
  • at HEAD --json validates against at.schema.json
  • diff HEAD~1..HEAD --json validates against diff.schema.json
  • review --batch accept --json validates against review-batch.schema.json
  • merge --from <repo> --dry-run --json validates against merge.schema.json

Summary by CodeRabbit

  • Tests
    • Enhanced integration tests with a multi-repository setup and deterministic commits for consistent results
    • Added JSON-schema validation for at, diff, review (batch) and merge (dry-run) outputs
    • Introduced separate merge-focused temporary repository and improved setup/teardown cleanup
    • Clarified that some JSON-based tests rely on features unavailable in certain CI environments and may be skipped

Address round-3 CodeRabbit feedback requesting canary tests for
remaining CLI commands. Adds 4 new integration tests:

- at HEAD --json → at.schema.json
- diff HEAD~1..HEAD --json → diff.schema.json
- review --batch accept --json → review-batch.schema.json
- merge --from <repo> --dry-run --json → merge.schema.json

suggest --json is intentionally skipped (requires GITMIND_AGENT).

371 tests across 22 files.
@coderabbitai
Copy link
Contributor

coderabbitai bot commented Feb 14, 2026

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Walkthrough

Revises test/contracts.integration.test.js to replace direct git child-process calls with execFileSync, add a gitCommit(cwd, filename, message) helper, create two temporary repositories with deterministic commit history, and extend CLI JSON-schema validation tests for at, diff, review (batch), and merge (dry-run).

Changes

Cohort / File(s) Summary
Test file
test/contracts.integration.test.js
Replaced prior git init/config with execFileSync git invocations; added gitCommit(cwd, filename, message) helper; created two temp repos (tempDir, mergeDir) with epoch/deterministic commits and diff-relevant edges; updated setup/teardown to clean both repos; added JSON-schema validation checks for at, diff, review --batch, and merge --dry-run outputs; noted CI limitations for some LLM-based JSON paths.

Sequence Diagram(s)

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

Poem

🐇 I hop through commits, neat and small,

I seed two fields and answer the call,
Epochs fixed, diffs set to sing,
JSON schemas now have their ring,
A rabbit cheers: the tests stand tall.

🚥 Pre-merge checks | ✅ 6
✅ Passed checks (6 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding integration canary tests for four CLI commands (at, diff, review --batch, merge).
Linked Issues check ✅ Passed The PR adds integration tests validating CLI --json outputs against JSON schemas for 4 commands, directly supporting issue #205's Phase A1 objectives for schema contract enforcement and CI validation.
Out of Scope Changes check ✅ Passed All changes are scoped to test infrastructure improvements: new integration canary tests, test setup enhancements (execFileSync, epoch markers, merge repository), and schema validation—all aligned with PR #205 requirements.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into main

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch ironclad/canary-expansion

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@package.json`:
- Line 3: The package.json "version" field was bumped to 3.0.1 for a tests-only
change; either revert that change (restore the previous "version" value in
package.json) so the PR contains no consumer-facing version update, or if the
bump is intentional, add an explicit note in the PR description and changelog
explaining why tests require a version bump and keep the "version" change;
locate and update the "version" field in package.json accordingly and remove any
commit that only exists to bump the version if it isn't warranted.

In `@test/contracts.integration.test.js`:
- Around line 39-44: The gitCommit helper uses execSync with a
shell-interpolated string which is fragile; change it to use execFileSync with
argument arrays to avoid shell interpretation: replace execSync(`git add
${filename}` ...) with execFileSync('git', ['add', filename], {cwd, stdio:
'ignore'}), replace execSync(`git commit -m "${message}"` ...) with
execFileSync('git', ['commit', '-m', message], {cwd, stdio: 'ignore'}), and
replace the rev-parse call with execFileSync('git', ['rev-parse', 'HEAD'], {cwd,
encoding: 'utf-8'}). Keep the same option keys (cwd, stdio/encoding) as used
elsewhere.

- Use execFileSync with argument arrays in gitCommit helper to avoid
  shell interpolation (CodeRabbit)
- Revert version bump — tests-only change doesn't warrant a version
  change (CodeRabbit)
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@test/contracts.integration.test.js`:
- Around line 82-88: Replace the execSync shell-invocation calls used when
setting up mergeDir with execFileSync-style calls that pass the command and
arguments separately to avoid shell interpolation: change execSync('git init', {
cwd: mergeDir, ... }) to execFileSync('git', ['init'], { cwd: mergeDir, stdio:
'ignore' }), and change the two execSync('git config ...', { cwd: mergeDir, ...
}) lines to execFileSync('git', ['config', 'user.email', 'test@test.com'], {
cwd: mergeDir, stdio: 'ignore' }) and execFileSync('git', ['config',
'user.name', 'Test'], { cwd: mergeDir, stdio: 'ignore' }); keep the existing
execFileSync calls that run the BIN commands (execFileSync(process.execPath,
[BIN, ...], { cwd: mergeDir, stdio: 'ignore' })) unchanged; apply the same
execSync→execFileSync pattern to the tempDir setup lines if you want full
consistency with gitCommit and other helpers.

- Convert all git init/config calls to execFileSync with argument
  arrays (both tempDir and mergeDir setup)
- Remove unused execSync import
- Add 30s timeout to beforeAll (expanded setup with two repos + epochs
  exceeds vitest's default 10s hook timeout)
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Fix all issues with AI agents
In `@test/contracts.integration.test.js`:
- Around line 221-232: The test "review --batch accept --json validates against
review-batch.schema.json" mutates shared review state by accepting pending
items; add a concise inline comment above this it(...) noting the side-effect
and ordering constraint (e.g., "// NOTE: mutates review state — keep after
review --json test") so future contributors know this test must remain after
other tests that rely on pending reviews; reference the test block name,
tempDir, and runCli to locate where to add the comment.

Document the test-ordering constraint: review --batch accept mutates
shared state, so it must run after any test that expects pending items.
@flyingrobots flyingrobots merged commit 3f377ca into main Feb 14, 2026
4 of 5 checks passed
@flyingrobots flyingrobots deleted the ironclad/canary-expansion branch February 14, 2026 11:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

M9.T1: CLI JSON Schema Contracts

1 participant