Intelligent Test Prioritization Engine for Fail-Fast Feedback

[github-actions]

Strategic Improvement

Execute tests in smart order based on file changes and historical failure patterns, running high-risk tests first. When tests fail, pipeline aborts in minutes instead of completing full suite. Saves 30-50% of test execution time on failing runs, improves feedback speed, reduces wasted iterations.

Acceptance Criteria

• Test-to-file dependency mapping (which tests cover which source files)
• Historical test failure rate tracking stored in memory system
• Smart test ordering algorithm: prioritize tests covering changed files + high historical failure rate
• Fast-fail mode: abort test execution on first failure for rapid feedback
• Configurable via daemon-config.json (enable_test_prioritization, fast_fail_mode)
• Dashboard metrics showing test execution time savings from prioritization
• Documentation in CLAUDE.md under build loop and intelligence sections

Context

• Priority: P0
• Complexity: standard
• Generated by: Strategic Intelligence Agent
• Strategy alignment: P0: Reliability & Success Rate

[github-actions]

⚓ Shipwright Pipeline Starting

Field	Value
Template	standard
Complexity	standard
Max Iterations	20
Triage	Strategically valuable optimization that cuts test execution time 30-50% on failing runs via intelligent test prioritization and fast-fail mode, with clear integration into existing memory and dashboard infrastructure.
Runner	ubuntu-latest
Risk Level	unknown
Workflow	25252020937

The autonomous pipeline is processing this issue. Updates will be posted as stages complete.

[github-actions]

[github-actions]

[github-actions]

📋 Implementation Plan

Click to expand full plan (1307 lines)

I'll analyze this codebase and create a comprehensive implementation plan for the Intelligent Test Prioritization Engine.

Analysis: Current State

The Shipwright project has:

• Vitest test runner with npm package manager
• Memory persistence system (sw-memory.sh) for storing learnings
• Daemon configuration infrastructure (daemon-config.json)
• A test stage in the pipeline that runs npm test
• Dashboard infrastructure (dashboard/server.ts) for metrics

I need to integrate test prioritization into this existing framework without breaking the current test flow.

---

IMPLEMENTATION PLAN

Files to Modify

Core Implementation Files

• scripts/sw-test-dependency-mapper.sh (NEW) — Analyze project to build test-to-file mappings
• scripts/sw-test-failure-tracker.sh (NEW) — Track and update test failure rates in memory
• scripts/sw-test-prioritizer.sh (NEW) — Score and order tests for execution
• scripts/lib/test-execution-ordering.sh (NEW) — Shared test execution helpers (used by pipeline)
• scripts/sw-test-execution-ordering-test.sh (NEW) — Unit tests for test ordering logic

Integration Points

• .claude/CLAUDE.md (MODIFY) — Add documentation section on test prioritization
• scripts/lib/pipeline-stages.sh (MODIFY) — Integrate prioritizer into test stage, add fast-fail support
• scripts/sw-loop.sh (MODIFY) — Pass test failure results to tracker after each test run
• scripts/lib/daemon-config-schema.json (MODIFY) — Add test_prioritization config fields
• dashboard/server.ts (MODIFY) — Add test prioritization metrics endpoints
• package.json (MODIFY) — Add test-ordering script to npm scripts

Configuration & Documentation

• .claude/daemon-config.json (MODIFY) — Add test_prioritization section with defaults
• scripts/generated/test-execution-ordering.md (NEW) — Auto-generated documentation of ordering decisions
• .claude/test-holdout/manifest.json (MODIFY) — Track test artifacts for analysis

---

Implementation Steps

Phase 1: Test Dependency Analysis (Steps 1-4)

Step 1: Create Test Dependency Mapper

Build scripts/sw-test-dependency-mapper.sh (300 lines):

• Input: project root directory
• Parse all *.test.js and *.test.ts files to extract imports

✅ Task Checklist

• Task 1: Create sw-test-dependency-mapper.sh with import parsing and mapping logic
• Task 2: Create sw-test-failure-tracker.sh with failure rate calculation and memory updates
• Task 3: Create scripts/lib/test-execution-ordering.sh with scoring and ordering functions
• Task 4: Create sw-test-prioritizer.sh orchestrator script
• Task 5: Modify pipeline-stages.sh test stage to call prioritizer and implement fast-fail
• Task 6: Modify sw-loop.sh to track test results and call failure tracker
• Task 7: Update daemon-config.json with test_prioritization config section
• Task 8: Update daemon-config-schema.json with test_prioritization field definitions
• Task 9: Add GET /api/test-prioritization-metrics endpoint to dashboard/server.ts
• Task 10: Add test prioritization metrics widget to dashboard UI
• Task 11: Modify pipeline stage to write test execution order artifacts
• Task 12: Create unit test suite sw-test-execution-ordering-test.sh
• Task 13: Add integration test scenario to pipeline-test.sh
• Task 14: Create E2E validation with real test project
• Task 15: Update .claude/CLAUDE.md with test prioritization documentation
• Task 16: Create scripts/generated/test-execution-ordering.md auto-documentation template
• Task 17: Verify configuration is discoverable via shipwright templates list
• Task 18: Run full test suite to verify no regressions

---

Generated by shipwright pipeline at 2026-05-02T12:44:14Z

[github-actions]

[github-actions]

[github-actions]

📐 Architecture Decision Record

Click to expand ADR (1195 lines)

Architecture Decision Record: Intelligent Test Prioritization Engine for Fail-Fast Feedback

Context

Problem: In the Shipwright project, test suites run sequentially, delaying feedback on failures. When a test fails early in the suite, developers wait for the full suite to complete before learning about the failure. With 1000+ tests across diverse modules, this creates 5-10 minute feedback loops on failing builds.

Constraints:

• Language: JavaScript/TypeScript with Vitest test runner
• Existing infrastructure: Memory persistence system (sw-memory.sh), pipeline stages, daemon configuration
• Must integrate into npm test without breaking current behavior
• Must be optional (disableable for projects that don't need it)
• Must work in memory-constrained environments (CI runners with limited disk/CPU)

Opportunity: By analyzing test-to-file dependencies and tracking historical failure rates, we can order tests to run high-risk tests first, catch failures within 30-60 seconds instead of 10 minutes, and implement fast-fail to abort on first failure.

---

Decision

Implement a modular, deterministic test prioritization system with three layers:

Layer 1: Dependency Analysis (Offline)

• Parse all test files to extract import statements
• Build reverse mapping: source_file.js → [tests that import it]
• Store in memory system with cache invalidation on package.json change
• Update once per build, not per test run

Layer 2: Historical Failure Tracking (Online)

• Track pass/fail counts and failure rates per test
• Update memory atomically after each test completes
• Detect flaky tests (high failure rate but inconsistent)
• Query current rates during test prioritization

Layer 3: Test Ordering (Deterministic)

• Score each test: score = (weight_changed * coverage_match) + (weight_history * failure_rate)
• Default weights: 60% changed-file coverage, 40% historical failure rate
• Sort tests descending by score (high score = run first)
• Implement fast-fail mode: abort on first failure for quick feedback

Integration Point: Pipeline Test Stage

• Check if test prioritization is enabled in daemon-config.json
• If enabled: call prioritizer to get ordered test list, run npm test --testNamePattern
• If disabled: run npm test as normal (no behavioral change)
• Either way: update failure tracking for next build

Why This Approach?

1. Layered Responsibility: Each component has single, clear purpose

   • Mapper: build dependency index
   • Tracker: maintain failure rates
   • Prioritizer: orchestrate ordering
   • Library: reusable scoring functions

2. Deterministic: Uses concrete metrics (file dependencies + failure history), not heuristics. Scoring is reproducible and explainable.

3. Fail-Safe: If any component fails (memory unavailable, git error), falls back to full test suite. No data loss.

4. Non-Breaking: Existing test behavior unchanged when disabled. No impact on current workflows.

5. Observable: Events emitted for every scoring decision, metrics available in dashboard. Can debug ordering.

---

Generated by shipwright pipeline design stage at 2026-05-02T12:47:54Z

[github-actions]

[github-actions]

🔨 Build started — shipwright loop with 20 max iterations, 1 agent(s), model: haiku

[github-actions]

[github-actions]