CI Failure Doctor🏥 CI Failure Investigation - Daily Perf Improver Run #25 - Safe Output Validation Regression

[github-actions]

🏥 CI Failure Investigation - Run #25

Summary

The "Daily Perf Improver" workflow failed due to a safe-output validation regression that incorrectly rejects legitimate output types (create_pull_request and create_discussion), only allowing create_issue and missing_tool.

Failure Details

• Run: 19218848108
• Commit: 8abd9cd - "Optimize SC2002 useless cat patterns in analysis workflows (Optimize SC2002 useless cat patterns in analysis workflows #3547)"
• Trigger: schedule (automated daily run at 2am UTC)
• Duration: 12m 50s
• Date: 2025-11-10 02:36 UTC

Root Cause Analysis

Primary Issue: Safe Output Validation Regression

The workflow configuration defines three safe-output types:

safe-outputs:
  create-discussion:
    title-prefix: "${{ github.workflow }}"
    category: "ideas"
    max: 5
  add-comment:
    discussion: true
    target: "*"
  create-pull-request:
    draft: true

However, the validation logic is rejecting these legitimate output types:

Validation errors found:
- Line 1: Unexpected output type 'create_pull_request'. Expected one of: create_issue, missing_tool
- Line 2: Unexpected output type 'create_discussion'. Expected one of: create_issue, missing_tool

Analysis: This indicates a validation allowlist regression where the validator has been restricted to only accept create_issue and missing_tool, blocking other valid safe-output types that the workflow legitimately uses.

Secondary Issue: JSON Parsing Error

Line 3452: Unexpected token 'C', "Could not "... is not valid JSON
(Pattern: Generic ERROR messages, Raw log: 2025-11-10T02:39:33.650Z [DEBUG] Unable to parse tool invocation as JSON...)

This appears to be a consequence of the validation failures disrupting the workflow execution.

Failed Jobs and Errors

Job Sequence

1. ✅ activation - 10s - succeeded
2. ❌ agent - 12m 28s - completed with 3 errors and 3 warnings
3. ✅ detection - 0s - succeeded (skipped due to errors)
4. ⏭️ create_discussion - skipped (validation failed)
5. ⏭️ create_pull_request - skipped (validation failed)
6. ⏭️ missing_tool - skipped
7. ⏭️ add_comment - skipped

Error Summary

• Critical Validation Errors: 2 (create_pull_request, create_discussion rejected)
• JSON Parsing Errors: 1
• Warnings: 3
• Jobs Skipped: 4 (due to validation failures)

Investigation Findings

Artifacts Produced

• agent-stdio.log (6.39 KB) - Agent execution logs with validation errors
• aw.patch (18 KB) - Generated patch file (not applied due to validation failure)
• aw_info.json (492 B) - Workflow metadata
• prompt.txt (6.42 KB) - Agent prompt
• safe_output.jsonl (3.54 KB) - Safe outputs data (rejected by validation)

Key Finding: The agent successfully generated outputs and patch file, but the safe-output validation layer rejected the output types, preventing downstream processing.

Recent Commits Context

The triggering commit was part of PR #3547 which optimized SC2002 shellcheck patterns. This commit itself is unlikely to have caused the validation regression - this appears to be a pre-existing bug that was exposed when this workflow ran.

Related Workflows

Other workflows using similar safe-output configurations may be affected:

• daily-test-improver.md (uses create-pull-request, create-discussion) - ALSO FAILING (CI Failure Doctor🏥 CI Failure Investigation - Daily Test Coverage Improver Run #17 #3553)
• daily-doc-updater.md (uses create-pull-request)
• daily-repo-chronicle.md (uses create-discussion)

Recommended Actions

🔴 CRITICAL - Fix Safe Output Validation Logic

Priority: URGENT - Blocking multiple daily workflows

1. Locate Validation Code: Find where safe-output types are validated

   cd /path/to/repo
   grep -r "Expected one of: create_issue, missing_tool" pkg/
   grep -r "Unexpected output type" pkg/

2. Identify Regression: Check recent changes to validation logic

   git log --all --grep="validation" --grep="safe-output" --since="2025-11-05"
   git log -p -- pkg/workflow/*validation* pkg/cli/*safe*

3. Restore Full Allowlist: Ensure validator accepts all documented safe-output types:

   • create-issue ✓
   • create-discussion ✗ (currently rejected)
   • create-pull-request ✗ (currently rejected)
   • add-comment ✓ (via create-issue logic?)
   • create-pull-request-review-comment
   • update-issue
   • missing-tool ✓

4. Test Fix:

   # Compile and validate affected workflow
   gh aw compile daily-perf-improver --verbose
   
   # Run local validation tests
   make test-unit

5. Regression Test: Add test cases to prevent future allowlist regressions

   func TestSafeOutputValidation_AllTypes(t *testing.T) {
       validTypes := []string{
           "create-issue",
           "create-discussion", 
           "create-pull-request",
           "add-comment",
           "create-pull-request-review-comment",
           "update-issue",
           "missing-tool",
       }
       for _, outputType := range validTypes {
           // Test that each type is accepted
       }
   }

🟡 MEDIUM - Update Documentation

If the validation restriction was intentional (unlikely), update documentation to reflect limited safe-output support. Otherwise, document all supported types clearly.

🟢 LOW - Monitor Affected Workflows

Track other workflows using create-discussion and create-pull-request to assess impact scope:

grep -r "create-discussion:" .github/workflows/*.md
grep -r "create-pull-request:" .github/workflows/*.md

Prevention Strategies

1. Validation Allowlist Management

• Centralized Constant: Define allowed safe-output types in a single constant/slice
• Version Compatibility: Track which output types are supported in which versions
• Clear Documentation: Document all supported safe-output types in instructions.md

2. Regression Testing

• Comprehensive Test Suite: Test all documented safe-output types are accepted
• Integration Tests: End-to-end tests for each safe-output type
• CI Validation: Run validation tests in CI before merging changes

3. Error Messaging

• List All Types: When validation fails, show the complete list of allowed types
• Helpful Suggestions: If a type is misspelled, suggest corrections
• Documentation Link: Include link to safe-outputs documentation in error messages

Historical Context

Pattern: This appears to be a new failure mode not seen in previous investigations. The validation logic may have been recently modified to restrict safe-output types, possibly as part of:

• Security hardening
• Refactoring/cleanup that inadvertently removed valid types
• Incomplete feature implementation

Impact Scope: Multiple daily automation workflows are affected, including:

• Daily Perf Improver (this issue)
• Daily Test Coverage Improver (CI Failure Doctor🏥 CI Failure Investigation - Daily Test Coverage Improver Run #17 #3553)
• Potentially others using create-discussion or create-pull-request

AI Team Self-Improvement

Add to .github/instructions/developer.instructions.md:

## Safe Output Type Validation

### Maintaining the Allowlist

When modifying safe-output validation logic:

1. **ALWAYS maintain the complete allowlist** of supported types:
   - `create-issue`
   - `create-discussion`
   - `create-pull-request`
   - `add-comment`
   - `create-pull-request-review-comment`
   - `update-issue`
   - `missing-tool`

2. **NEVER remove types** without explicit deprecation process:
   - Search for workflows using the type: `grep -r "type-name:" .github/workflows/`
   - Create migration plan for affected workflows
   - Add deprecation warnings before removal
   - Update documentation

3. **ALWAYS add tests** when adding new safe-output types:
   ```go
   func TestSafeOutputValidation_NewType(t *testing.T) {
       // Verify new type is accepted
   }

4. Document validation changes in CHANGELOG.md with migration guide

Error Message Best Practices

When validation fails:

• List ALL allowed types, not just a subset
• Provide helpful context about what went wrong
• Link to documentation for safe-outputs reference
• Suggest fixes if the error is likely a typo

Testing Safe Output Changes

Before committing changes to safe-output validation:

# Test all safe-output types still work
for type in create-issue create-discussion create-pull-request add-comment; do
  echo "Testing $type..."
  # Your test command here
done

# Run validation tests
make test-unit

# Compile affected workflows
gh aw compile daily-perf-improver
gh aw compile daily-test-improver

## Next Steps

1. ✅ **Immediate**: Locate and fix the validation allowlist regression
2. 🔄 **Short-term**: Add comprehensive test coverage for safe-output validation
3. 📅 **Long-term**: Implement deprecation process for future safe-output type changes

---

**Investigation Metadata:**
- **Investigator**: CI Failure Doctor (automated)
- **Investigation Run**: 19218848108  
- **Investigation Date**: 2025-11-10T02:51:29Z
- **Pattern**: Safe output validation regression - allowlist too restrictive
- **Related Issue**: #3553 (Daily Test Coverage Improver - different failure pattern, same commit)

---

**SECURITY NOTE**: This investigation processed workflow configuration and logs. No sensitive data or untrusted content was executed.




> AI generated by [CI Failure Doctor](https://github.com/githubnext/gh-aw/actions/runs/19219045441)
>
> To add this workflow in your repository, run `gh aw add githubnext/agentics/workflows/ci-doctor.md`. See [usage guide](https://githubnext.github.io/gh-aw/tools/cli/).