Repository Quality Improvement Report — Validation Error Consistency (2026-03-13)

[github-actions[bot]]

🎯 Repository Quality Improvement Report — Validation Error Consistency

Analysis Date: 2026-03-13
Focus Area: Validation Error Consistency and Actionability (Custom)
Strategy Type: Custom
Custom Area: Yes — gh-aw has an exceptionally rich validation system (90+ validation files, 340+ error messages in pkg/workflow/ alone). This focus area targets the user experience quality of those errors: are they consistent, noise-free, and actionable enough to guide users toward a fix?

---

Executive Summary

The gh-aw validation system is impressive in breadth and coverage, but has several consistency gaps that degrade the user experience when compilation fails. Three patterns stand out:

1. Timestamp noise: The structured WorkflowValidationError, OperationError, and ConfigurationError types each prepend an RFC3339 timestamp (e.g., [2026-03-13T10:30:00Z]) to their .Error() output. This is unusual for CLI tools and makes errors harder to read without adding value for end users.

2. Missing "what to use instead" in feature-incompatibility errors: When users configure max-turns or max-continuations on an engine that doesn't support them, the error says "engine X does not support this feature" — but never says which engine does. Only Claude supports max-turns; only Copilot supports max-continuations.

3. Hardcoded outdated URLs in strict_mode_validation.go: Seven error messages reference https://github.github.com/gh-aw/reference/… — a different domain/path format from the constants.DocsSandboxURL/DocsNetworkURL style used everywhere else. Missing constants (DocsSafeOutputsURL) force the workaround.

Fixing these will improve error clarity without touching any logic or changing any validation behavior.

Full Analysis Report

Focus Area: Validation Error Consistency and Actionability

Current State Assessment

Metrics Collected:

Metric	Value	Status
Total non-test fmt.Errorf in pkg/workflow/	340	ℹ️
Structured NewValidationError calls	39 (11.5%)	⚠️
Errors with actionable language (please/use/see/check/…)	~245 (72%)	✅
Errors lacking actionable language	~95 (28%)	⚠️
Hardcoded outdated URLs in strict_mode_validation.go	7	⚠️
Feature-incompatibility errors without alternative suggestion	2	⚠️
Structured errors that include RFC3339 timestamp in output	3 types	⚠️

Findings

Strengths

• mcp_config_validation.go, engine_validation.go, docker_validation.go — include rich inline YAML examples and See: doc links in almost every error
• The WorkflowValidationError / OperationError / ConfigurationError types have a clean Suggestion field architecture
• formatCompilerError / formatCompilerMessage provide IDE-navigable output with file/line context — widely used
• 72% of errors include some form of actionable language

Areas for Improvement

• [High] WorkflowValidationError.Error(), OperationError.Error(), ConfigurationError.Error() all include [2026-03-13T10:30:00Z] timestamp prefixes — unnecessary noise for user-facing CLI output
• [High] validateMaxTurnsSupport / validateMaxContinuationsSupport in agent_validation.go report unsupported features without naming the engine that does support them; also bypass formatCompilerError so the error lacks file context
• [Medium] strict_mode_validation.go uses 7 hardcoded https://github.github.com/gh-aw/reference/… URLs that don't match any existing constant — both pointing to a different domain and lacking the DocsSafeOutputsURL constant
• [Low] labels_validation.go and repo_memory_validation.go use raw fmt.Errorf without formatCompilerError, losing the consistent file-context formatting other validation files provide

Detailed Analysis

Timestamp noise in structured error types:

// Current output
[2026-03-13T10:30:00Z] Validation failed for field 'concurrency'
Value: invalid-value
Reason: concurrency group must be a non-empty string
Suggestion: Use a non-empty string like "$\{\{ github.ref }}"

The timestamp is useful in log aggregation but actively hurts readability in a terminal. No other validator in this repo includes timestamps in the error text.

Feature-incompatibility errors without alternatives (agent_validation.go):

// Current
return fmt.Errorf("max-turns not supported: engine '%s' does not support the max-turns feature", engine.GetID())

// Better
return formatCompilerError(markdownPath, "error",
    fmt.Sprintf("max-turns not supported: engine '%s' does not support the max-turns feature.\n\n"+
        "Engines that support max-turns: claude\n\n"+
        "See: %s", engine.GetID(), constants.DocsEnginesURL), nil)

Hardcoded URLs in strict_mode_validation.go:

// Current (7 occurrences)
"... See: https://github.github.com/gh-aw/reference/safe-outputs/"
"... See: https://github.github.com/gh-aw/reference/network/"
"... See: https://github.github.com/gh-aw/reference/sandbox/"

// Should use
constants.DocsSafeOutputsURL  // missing — needs to be added
constants.DocsNetworkURL      // exists
constants.DocsSandboxURL      // exists
constants.DocsEnginesURL      // exists

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot coding agent execution. Please split these into individual work items for the agent to process sequentially.

---

Task 1: Remove RFC3339 Timestamps from WorkflowValidationError, OperationError, and ConfigurationError

Priority: High
Estimated Effort: Small
Focus Area: Validation Error Consistency

Description:
The Error() methods on WorkflowValidationError, OperationError, and ConfigurationError in pkg/workflow/workflow_errors.go each prepend an RFC3339 timestamp (e.g., [2026-03-13T10:30:00Z]) to the error string. This is unusual for CLI tools and creates noisy output. The Timestamp field should remain (for programmatic use / logging) but should not appear in the user-facing .Error() string.

Acceptance Criteria:

• WorkflowValidationError.Error() no longer includes the [timestamp] prefix
• OperationError.Error() no longer includes the [timestamp] prefix
• ConfigurationError.Error() no longer includes the [timestamp] prefix
• The Timestamp field is retained on each struct
• All existing tests in pkg/workflow/ continue to pass (go test ./pkg/workflow/...)
• make fmt passes

Code Region: pkg/workflow/workflow_errors.go

In `pkg/workflow/workflow_errors.go`, update the three `Error()` methods to remove the timestamp prefix:

For `WorkflowValidationError.Error()`:
- Remove the line: `fmt.Fprintf(&b, "[%s] Validation failed for field '%s'", e.Timestamp.Format(time.RFC3339), e.Field)`
- Replace with: `fmt.Fprintf(&b, "Validation failed for field '%s'", e.Field)`

For `OperationError.Error()`:
- Remove the line: `fmt.Fprintf(&b, "[%s] Failed to %s %s", e.Timestamp.Format(time.RFC3339), e.Operation, e.EntityType)`
- Replace with: `fmt.Fprintf(&b, "Failed to %s %s", e.Operation, e.EntityType)`

For `ConfigurationError.Error()`:
- Remove the line: `fmt.Fprintf(&b, "[%s] Configuration error in '%s'", e.Timestamp.Format(time.RFC3339), e.ConfigKey)`
- Replace with: `fmt.Fprintf(&b, "Configuration error in '%s'", e.ConfigKey)`

Also remove the `"time"` import if it is no longer used after these changes (check — `time.Now()` is still called in constructors so the import stays).

Run `make fmt` and `go test ./pkg/workflow/...` to validate.

---

Task 2: Improve Feature-Incompatibility Errors in agent_validation.go

Priority: High
Estimated Effort: Small
Focus Area: Validation Error Actionability

Description:
validateMaxTurnsSupport and validateMaxContinuationsSupport in pkg/workflow/agent_validation.go produce "not supported" errors that:

1. Don't tell the user which engine does support the feature
2. Use raw fmt.Errorf instead of formatCompilerError, so they lack file context

Fix both issues:

• max-turns is supported only by claude engine
• max-continuations is supported only by copilot engine

Acceptance Criteria:

• validateMaxTurnsSupport error includes "Engines that support max-turns: claude" guidance
• validateMaxContinuationsSupport error includes "Engines that support max-continuations: copilot" guidance
• Both functions accept markdownPath string as a new parameter and use formatCompilerError for error formatting
• Call sites in the broader validation pipeline are updated to pass markdownPath
• Errors include a See: link to constants.DocsEnginesURL
• All existing tests pass (go test ./pkg/workflow/...)

Code Region: pkg/workflow/agent_validation.go

In `pkg/workflow/agent_validation.go`, update `validateMaxTurnsSupport` and `validateMaxContinuationsSupport`:

1. Add `markdownPath string` parameter to both functions.

2. Replace the `fmt.Errorf(...)` in `validateMaxTurnsSupport` with:
```go
return formatCompilerError(markdownPath, "error",
    fmt.Sprintf("max-turns not supported: engine '%s' does not support the max-turns feature.\n\n"+
        "Engines that support max-turns: claude\n\n"+
        "To use max-turns, change your workflow engine:\n"+
        "engine:\n"+
        "  id: claude\n"+
        "  max-turns: 10\n\n"+
        "See: %s", engine.GetID(), constants.DocsEnginesURL), nil)

3. Replace the fmt.Errorf(...) in validateMaxContinuationsSupport with:

return formatCompilerError(markdownPath, "error",
    fmt.Sprintf("max-continuations not supported: engine '%s' does not support the max-continuations feature.\n\n"+
        "Engines that support max-continuations: copilot\n\n"+
        "To use max-continuations, change your workflow engine:\n"+
        "engine: copilot\n\n"+
        "See: %s", engine.GetID(), constants.DocsEnginesURL), nil)

4. Add "github.com/github/gh-aw/pkg/constants" import if not present.

5. Find all call sites of these two functions in the codebase and add the markdownPath argument (search for validateMaxTurnsSupport( and validateMaxContinuationsSupport().

Run make fmt and go test ./pkg/workflow/... to validate.

---

#### Task 3: Add DocsSafeOutputsURL Constant and Replace Hardcoded URLs in strict_mode_validation.go

**Priority**: Medium
**Estimated Effort**: Small
**Focus Area**: Validation Error Consistency

**Description:**
`pkg/workflow/strict_mode_validation.go` contains 7 hardcoded `https://github.github.com/gh-aw/reference/…` URLs — a different domain/path format from constants in `pkg/constants/constants.go`. A `DocsSafeOutputsURL` constant is also missing. This task adds the missing constant and replaces all hardcoded URLs with the proper constant references.

The correct URLs should follow the pattern `https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/…` used by existing constants.

**Acceptance Criteria:**
- [ ] `DocsSafeOutputsURL` constant added to `pkg/constants/constants.go` with correct docs path
- [ ] All 7 hardcoded `github.github.com` URLs in `strict_mode_validation.go` replaced with constants
- [ ] `strict_mode_validation.go` imports `pkg/constants` and uses the constants
- [ ] `make fmt` and `go test ./pkg/workflow/...` pass

**Code Region:** `pkg/constants/constants.go`, `pkg/workflow/strict_mode_validation.go`

```markdown
Step 1: Add missing constant to `pkg/constants/constants.go`:

In the `DocURL` constants block (near `DocsSandboxURL`), add:
```go
// DocsSafeOutputsURL is the documentation URL for safe-outputs configuration
DocsSafeOutputsURL DocURL = "https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/safe-outputs.md"

Step 2: Update pkg/workflow/strict_mode_validation.go:

Add "github.com/github/gh-aw/pkg/constants" to imports if not already present.

Replace each hardcoded URL:

• https://github.github.com/gh-aw/reference/safe-outputs/ → constants.DocsSafeOutputsURL
• https://github.github.com/gh-aw/reference/network/#available-ecosystem-identifiers → constants.DocsNetworkURL
• https://github.github.com/gh-aw/reference/network/ → constants.DocsNetworkURL
• https://github.github.com/gh-aw/reference/sandbox/ → constants.DocsSandboxURL
• https://github.github.com/gh-aw/reference/engines/ → constants.DocsEnginesURL

Run make fmt and go test ./pkg/workflow/... to validate.

---

#### Task 4: Add formatCompilerError to labels_validation.go and repo_memory_validation.go

**Priority**: Low
**Estimated Effort**: Small
**Focus Area**: Validation Error Consistency

**Description:**
`pkg/workflow/labels_validation.go` and `pkg/workflow/repo_memory_validation.go` use raw `fmt.Errorf` for user-facing validation errors, losing the file-path context and consistent IDE-navigable formatting that `formatCompilerError` provides. These files should be updated to use `formatCompilerError` so that errors in these validators look the same as errors in `mcp_config_validation.go`, `engine_validation.go`, and others.

**Acceptance Criteria:**
- [ ] All user-facing validation error returns in `labels_validation.go` use `formatCompilerError(markdownPath, "error", ...)`
- [ ] All user-facing validation error returns in `repo_memory_validation.go` use `formatCompilerError(markdownPath, "error", ...)`
- [ ] `markdownPath` is plumbed through to these functions (check existing signatures; add parameter if not present)
- [ ] Internal/wrapping errors (e.g., `fmt.Errorf("failed to parse X: %w", err)`) remain as-is — only user-facing validation returns are changed
- [ ] `make fmt` and `go test ./pkg/workflow/...` pass

**Code Region:** `pkg/workflow/labels_validation.go`, `pkg/workflow/repo_memory_validation.go`

```markdown
In `pkg/workflow/labels_validation.go` and `pkg/workflow/repo_memory_validation.go`:

1. Inspect each `return fmt.Errorf(...)` in these files.
2. For errors that describe a user configuration problem (not an internal operation failure like "failed to read file"), replace:
   ```go
   return fmt.Errorf("labels: %s", message)

with:

return formatCompilerError(markdownPath, "error", fmt.Sprintf("labels: %s", message), nil)

3. Add markdownPath string to function signatures if not already present, and update all call sites.
4. Leave fmt.Errorf("failed to ...: %w", err) style wrapping errors unchanged.

Run make fmt and go test ./pkg/workflow/... to validate.

---

### 📊 Historical Context

<details>
<summary><b>Previous Focus Areas</b></summary>

| Date | Focus Area | Type | Custom | Key Outcomes |
|------|------------|------|--------|--------------|
| 2026-03-13 | Validation Error Consistency | Custom | Yes | 4 tasks generated covering timestamp noise, feature-incompatibility UX, URL constant hygiene, and error formatter consistency |

</details>

---

### 🎯 Recommendations

#### Immediate Actions (This Week)
1. **Task 1** — Remove timestamp from WorkflowValidationError output — Priority: High, effort ~15 min
2. **Task 2** — Add engine alternatives to feature-incompatibility errors — Priority: High, effort ~20 min

#### Short-term Actions (This Month)
1. **Task 3** — Add `DocsSafeOutputsURL` and fix hardcoded URLs in strict_mode_validation.go — Priority: Medium
2. **Task 4** — Migrate labels/repo_memory validation to `formatCompilerError` — Priority: Low

#### Long-term Actions (This Quarter)
1. Audit the remaining 95 non-actionable "failed to X: %w" errors and upgrade those that are user-facing (vs internal stack-wrapping) to include suggestions
2. Consider a linting rule (or CI check) to detect user-facing validation errors that don't include a `See:` doc URL

---

### 📈 Success Metrics

Track these metrics to measure improvement in **Validation Error Consistency**:

- **Structured error adoption**: 11.5% → 25%+ of validation returns use `NewValidationError` or `formatCompilerError`
- **Errors with actionable language**: 72% → 85%+
- **Hardcoded non-constant URLs**: 7 → 0
- **Feature-incompatibility errors with "use X instead"**: 0 → 2 (max-turns, max-continuations)

---

### Next Steps

1. Review and prioritize the 4 tasks above
2. Assign tasks to Copilot coding agent via planner agent
3. Re-evaluate this focus area after tasks are merged
4. Next analysis will select a different focus area (diversity algorithm)

---

**References:**
- [§23053171869](https://github.com/github/gh-aw/actions/runs/23053171869)


> AI generated by [Repository Quality Improvement Agent](https://github.com/github/gh-aw/actions/runs/23053171869) · [history](https://github.com/search?q=repo%3Agithub%2Fgh-aw+%22gh-aw-workflow-call-id%3A+github%2Fgh-aw%2Frepository-quality-improver%22&type=discussions)
> - [x] expires <!-- gh-aw-expires: 2026-03-14T13:39:03.121Z --> on Mar 14, 2026, 1:39 PM UTC

<!-- gh-aw-workflow-id: repository-quality-improver -->
<!-- gh-aw-workflow-call-id: github/gh-aw/repository-quality-improver -->

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-14T13:39:03.121Z.

Closed by Workflow