Add missing test patterns to CI - cover all 353 untested functions

[Copilot]

Fix CI to run all missing tests ✅

This PR updates ci.yml to ensure all 353 missing tests are executed in CI.

Problem Statement

The canary_go job detected 353 test functions (8.9% of codebase) not being executed in any CI test job, representing significant coverage gaps.

Analysis

• Total tests in codebase: 3,980 test functions
• Tests currently executed: 13,751 (includes subtests)
• Missing top-level test functions: 353 (8.9% of all tests)

Missing Tests Breakdown by Package

pkg/cli (288 tests - 81.6%)

The majority of missing tests were in CLI command implementations:

• compile_command_test.go: 19 tests
• update_command_test.go: 22 tests
• status_command_test.go: 12 tests
• shell_completion_test.go: 11 tests
• upgrade_command_test.go: 10 tests
• completions_integration_test.go: 9 tests
• mcp_server_*.go: 25+ tests
• And many more integration tests

cmd/gh-aw (16 tests - 4.5%)

Main entry point and command structure tests:

• main_entry_test.go: 8 tests
• argument_syntax_test.go: 5 tests
• command_groups_test.go: 3 tests

pkg/workflow (38 tests - 10.8%)

Workflow compilation and validation tests:

• safe_outputs integration tests
• git commands integration tests
• custom engine tests
• Claude tools integration tests
• timestamp and validation tests

pkg/parser (11 tests - 3.1%)

Parser and frontmatter tests:

• remote_fetch_integration_test.go: 6 tests
• integration_test.go: 3 tests
• schema_location_integration_test.go: 2 tests

---

Changes Made

Enhanced Existing Test Groups

Updated patterns in existing test groups to include missing tests:

1. CLI Compile & Poutine: Added build helpers (TestBuildCommandString, TestBuildSourceString, TestBuildWorkflowDescription, TestPrintCompilationSummary, TestCompilationStats, TestModifyWorkflowForTrialMode)

2. CLI MCP Other: Added TestMCPCommand, TestIsRunningAsMCPServer, TestHeaderRoundTripper

3. CLI Compile Workflows: Expanded from single test to all workflow compilation tests (TestCompileWorkflows*, TestCollectWorkflowFiles, TestFilterWorkflowFiles, TestNormalizeWorkflowFile)

4. CLI Security Tools: Added TestParseAndDisplayZizmorOutput, TestSecurityToolsIndependentOfValidate, TestSecurity*

5. Workflow Safe Outputs: Added SafeInputs, ConsolidatedSafeOutputs, SafeOutputJobs patterns

6. Workflow GitHub & Git: Added AdditionalClaudeTools, CopilotGitCommands patterns

7. Workflow Validation: Added TestRepositoryFeaturesValidation, TestBackwardCompatibilityWithClaudeFormat

8. Workflow Permissions: Added TestExtractNpx pattern

9. Workflow Misc Part 1: Added TestCustomEngine, TestManualApprovalEnvironmentInActivationJob, TestNeutralToolsIntegration, TestIndividualGitHubToken, TestTopLevelGitHubTokenPrecedence

10. Workflow String & Sanitization: Added SingleQuoteEscaping, WorkflowTimestampCheckUsesJavaScript

11. Workflow Runtime & Setup: Added StopTimeResolution pattern

12. Parser Remote Fetch & Cache: Added TestFrontmatterLocation, TestFrontmatterOffsetCalculation, TestImprovementComparison

New Test Groups Added (17 CLI + 1 CMD = 18 total)

Created focused test groups for better organization and parallelization:

New CLI Test Groups

1. CLI Shell Completion (11 tests)

   • Pattern: TestCompletion|TestUninstall.*Completion
   • Shell completion generation and installation tests

2. CLI Secrets & Encryption (11 tests)

   • Pattern: TestSecret|TestEncrypt|TestCheckSecretsAvailability|...
   • Secret management and encryption tests

3. CLI Status & Versioning (16 tests)

   • Pattern: TestStatus|TestVersion|TestWorkflowStatus|TestWorkflowRunInfo|...
   • Status reporting and version checking tests

4. CLI Update & Upgrade (34 tests)

   • Pattern: TestUpdate|TestUpgrade|TestCheckForUpdates|...
   • Update check, version comparison, and upgrade tests

5. CLI Workflows Path & Discovery (28 tests)

   • Pattern: TestFindRunnableWorkflows|TestGetAvailableWorkflowNames|...
   • Workflow discovery, path resolution, and input validation tests

6. CLI Trial & Interactive (7 tests)

   • Pattern: TestTrial|TestLipglossImportPresent|TestSectionCompositionPattern
   • Trial mode and interactive UI rendering tests

7. CLI Run Command (11 tests)

   • Pattern: TestRun|TestPoll|TestTrackWorkflowFailure|TestInputValidation|...
   • Workflow execution and tracking tests

8. CLI Repo & Git (13 tests)

   • Pattern: TestRepo|TestGet.*RepoSlug|TestExtractBaseRepo|...
   • Repository slug parsing and git operations tests

9. CLI Spec Parsing (11 tests)

   • Pattern: TestParse.*Spec|TestResolveImportPathLocal|...
   • Workflow spec parsing and resolution tests

10. CLI Validators & Semver (8 tests)

    • Pattern: TestValidat|TestIsCommitSHA|TestIsPreciseVersion|...
    • Input validation and semantic versioning tests

11. CLI Tool Graph (3 tests)

    • Pattern: TestToolGraph
    • Tool dependency graph tests

12. CLI Signal Handling (5 tests)

    • Pattern: TestPollWithSignalHandling
    • Signal handling and graceful shutdown tests

13. CLI Makefile & Setup (3 tests)

    • Pattern: TestMakefile|TestSetupCLIAction
    • Build system and setup action tests

14. CLI Merge Content (5 tests)

    • Pattern: TestMergeWorkflowContent
    • Workflow content merging tests

15. CLI Workflows Actions (3 tests)

    • Pattern: TestGetActionSHAForTag|TestExtractNpxPackages
    • GitHub Actions version resolution tests

16. CLI Codespace Integration (5 tests)

    • Pattern: TestCodespace|TestDetectShell
    • GitHub Codespaces integration tests

17. CLI Safe Inputs (1 test)

    • Pattern: TestSafeInputsMCPServerCompilation
    • Safe inputs MCP server compilation test

New CMD Test Group

18. CMD Main & Entry (16 tests)
    • Pattern: TestMain|TestCommandGroup|TestCommandLine|TestCommand|...
    • Main entry point, command structure, and argument syntax tests

---

Error Handling Improvements

Fixed Issues with Exit Code Propagation

Problem: Using | tee in bash without set -o pipefail causes the exit code of go test to be ignored, allowing test failures to be swallowed.

Solution:

1. Added set -o pipefail to unit and integration test steps to ensure go test exit codes propagate correctly when using tee
2. Added set -euo pipefail to all inline bash scripts in canary_go job for proper error handling
3. Improved extract-executed-tests.sh to explicitly fail when no test records are found (instead of silently succeeding with || true)

Changes Made

In .github/workflows/ci.yml:

• Unit test step: Added set -o pipefail before go test | tee
• Integration test step: Added set -o pipefail before go test | tee
• All canary_go inline scripts: Added set -euo pipefail

In scripts/extract-executed-tests.sh:

• Removed || true that was swallowing errors
• Added explicit error handling with temp file approach
• Now fails with exit code 1 and error message when no test records found

These changes ensure that:
✅ Test failures will cause CI jobs to fail
✅ Missing test artifacts will be detected
✅ Script errors will propagate correctly
✅ No silent failures in the test pipeline

---

Validation

Pattern Matching Verification

All 353 missing tests are now covered by the updated patterns:

✅ pkg/cli tests: 288/288 covered (100%)
✅ cmd/gh-aw tests: 16/16 covered (100%)
✅ pkg/workflow tests: 38/38 covered (100%)
✅ pkg/parser tests: 11/11 covered (100%)

Test Execution Verification

Verified that previously missing tests now execute correctly:

$ go test -v -tags 'integration' -run 'TestArgumentSyntaxConsistency' ./cmd/gh-aw
=== RUN   TestArgumentSyntaxConsistency
--- PASS: TestArgumentSyntaxConsistency (0.03s)

YAML Validation

✅ ci.yml syntax is valid
✅ All patterns use valid regex syntax
✅ Skip patterns properly exclude covered tests
✅ Script syntax validated

Error Handling Validation

✅ Tested extract-executed-tests.sh with empty directory - properly fails with exit code 1
✅ Tested extract-executed-tests.sh with valid results - works correctly
✅ Verified set -o pipefail prevents exit code swallowing

---

Impact

Before This PR

• ❌ 353 test functions (8.9%) not executed in CI
• ❌ Test failures could be silently swallowed due to missing pipefail
• ❌ Script errors masked by || true
• ❌ Potential bugs in uncovered code paths
• ❌ False confidence in test coverage

After This PR

• ✅ All 353 previously missing tests now executed
• ✅ Test failures properly propagate and cause job failures
• ✅ Script errors detected and reported
• ✅ 100% test function coverage in CI
• ✅ Better test organization with 18 new focused test groups
• ✅ Improved parallelization through granular test grouping

---

Checklist

• Analyze existing test patterns in ci.yml
• Identify all 353 missing test patterns
• Add missing patterns to appropriate test groups
• Create new test groups for better organization
• Validate all patterns match expected tests
• Verify YAML syntax is valid
• Test sample of previously missing tests
• Update skip patterns to exclude covered tests
• Fix error handling to ensure test failures propagate
• Add pipefail to prevent exit code swallowing
• Remove || true that masked errors
• Add proper error handling to all inline scripts

---

Notes for Reviewers

1. No code changes - only CI configuration and script improvements
2. All patterns tested - manually verified each pattern matches expected tests
3. Error handling verified - tested both success and failure cases
4. Backward compatible - existing test groups unchanged, only enhanced
5. Improved organization - new groups follow existing naming conventions
6. Ready to merge - all validation checks passed

Original prompt

Reference: https://github.com/githubnext/gh-aw/actions/runs/21454914137/job/61800028191#step:7:1

Update ci.yml jobs to run the missing tests listed in the job report above.

---

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

[pelikhan]

@copilot Ensure failing tests cause the job to fail. Do not swallow any errors in the scripts. Review all scripts in CI YAML and confirm that no error codes are being swallowed.

[Copilot]

@copilot Ensure failing tests cause the job to fail. Do not swallow any errors in the scripts. Review all scripts in CI YAML and confirm that no error codes are being swallowed.

Fixed in commit 9985000. Changes made:

CI YAML (.github/workflows/ci.yml):

• Added set -o pipefail to unit and integration test steps to ensure go test exit codes propagate when using | tee
• Added set -euo pipefail to all canary_go inline bash scripts

Script (scripts/extract-executed-tests.sh):

• Removed || true that was swallowing errors
• Now explicitly fails with exit code 1 when no test records are found
• Uses temp file approach for better error detection

All test failures and script errors will now properly cause CI jobs to fail.