[syntax-error-quality] Daily Syntax Error Quality Check — 2026-03-19

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2026-03-19
Test Cases: 3
Average Score: 69.7/100
Status: ⚠️ Needs Improvement (average below threshold of 70; one test case scores in Poor range)

---

Executive Summary

Three workflows were tested with one syntax error each (YAML syntax, invalid engine name, and invalid schema value). Engine validation errors are excellent — with "Did you mean?" suggestions, examples, and documentation links. However, schema validation errors for frontmatter fields like timeout-minutes have critical DX issues: they always report line 1:1 (instead of the actual field line), include a confusing double-nested prefix, and provide no source-code context. YAML syntax errors are generally good but lack correct-usage examples.

Key Findings:

• Strengths: Engine validation messages are excellent — they list valid options, suggest the closest match, show a correct usage example, and link to docs
• Weaknesses: Schema validation for numeric/type constraints uses the wrong line number (1:1 instead of the actual field line), and YAML syntax errors don't show the corrected form
• Critical Issues: timeout-minutes: -10 reports test-3.md:1:1 — the error is not navigable via IDE click-through; the double prefix "workflow schema validation failed: GitHub Actions schema validation failed:" is confusing and redundant

---

Test Case Results

Test Case 1: YAML Syntax Error (Missing colon after key) — Score: 74/100 ⚠️

Test Configuration

Workflow: smoke-gemini.md (simple, 80 lines)
Error Type: Category A — Frontmatter YAML syntax error
Error Introduced: Removed colon from engine: key, changing engine: to engine (bare word, no colon)

Expected Compiler Output

test-1.md:15:1: error: Invalid YAML syntax: expected 'key: value' format (did you forget a colon after the key?)
  14 | name: Smoke Gemini
  15 | engine
> 16 |   id: gemini
     | ^
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 20/25 | Good |
| Actionability | 20/25 | Good |
| Context | 18/20 | Excellent |
| Examples | 3/15 | Poor |
| Consistency | 13/15 | Excellent |
| **Total** | **74/100** | **Good** |

#### Strengths
- ✅ File:line:column format enables IDE navigation
- ✅ Source context shown with `>` pointer to error location (Rust-style)
- ✅ Error message translated from cryptic YAML jargon to plain English
- ✅ Parenthetical hint "(did you forget a colon after the key?)" is genuinely helpful

#### Weaknesses
- ⚠️ No example of the **correct** syntax (e.g., `engine:\n  id: gemini`)
- ⚠️ No documentation link for workflow frontmatter syntax
- ⚠️ "expected 'key: value' format" is slightly technical; plain "add `:` after the key" would be simpler

#### Improvement Suggestions

1. **Add correct usage example**:
   ```
   Correct usage:
   engine:
     id: gemini
   ```

2. **Add a doc URL** to the workflow syntax reference, similar to how engine validation does it.

</details>

<details>
<summary><b>Test Case 2: Invalid Engine Name (typo)</b> — Score: 85/100 ✅</summary>

#### Test Configuration

**Workflow**: `workflow-generator.md` (medium, 112 lines)
**Error Type**: Category B — Configuration error (invalid engine name)
**Error Introduced**: Changed `engine: copilot` to `engine: copiilot` (double `i`)

#### Expected Compiler Output

```
test-2.md:13:1: error: invalid engine: copiilot. Valid engines are: copilot, claude, codex, custom.

Did you mean: copilot?

Example:
engine: copilot

See: https://github.com/github/gh-aw/blob/main/docs/src/content/docs/reference/engines.md
```

#### Evaluation Scores

| Dimension | Score | Rating |
|-----------|-------|--------|
| Clarity | 22/25 | Excellent |
| Actionability | 24/25 | Excellent |
| Context | 11/20 | Acceptable |
| Examples | 14/15 | Excellent |
| Consistency | 14/15 | Excellent |
| **Total** | **85/100** | **Excellent** |

#### Strengths
- ✅ Points to the exact line of the `engine:` field (not just line 1)
- ✅ Lists all valid engine names
- ✅ "Did you mean: copilot?" fuzzy-match suggestion
- ✅ Shows correct usage example
- ✅ Links to documentation
- ✅ Consistent format with other compiler errors

#### Weaknesses
- ⚠️ No source-code context lines (no `>` pointer showing the `engine: copiilot` line) — unlike YAML errors
- ⚠️ Minor: multi-paragraph format with blank lines may be harder to parse by tooling

#### Improvement Suggestions

1. **Add source-code context** showing the bad line with a pointer (same as YAML errors do):
   ```
   test-2.md:13:1: error: invalid engine: copiilot. ...
    12 | engine: copiilot
        |         ^^^^^^^^ unknown engine
   ```

</details>

<details>
<summary><b>Test Case 3: Negative timeout-minutes value</b> — Score: 50/100 ❌</summary>

#### Test Configuration

**Workflow**: `cli-version-checker.md` (complex, 351 lines)
**Error Type**: Category C — Semantic error (invalid value, constraint violation)
**Error Introduced**: Changed `timeout-minutes: 45` to `timeout-minutes: -10` (violates schema `minimum: 1`)

#### Expected Compiler Output

```
test-3.md:1:1: error: workflow schema validation failed: GitHub Actions schema validation failed: timeout-minutes: got number -10, must be >= 1. Example: timeout-minutes: 10

Evaluation Scores

Dimension	Score	Rating
Clarity	14/25	Acceptable
Actionability	12/25	Poor
Context	4/20	Critical
Examples	10/15	Good
Consistency	10/15	Acceptable
Total	50/100	Poor

Strengths

• ✅ Mentions timeout-minutes by name
• ✅ Includes "Example: timeout-minutes: 10" at the end of the message
• ✅ States the exact constraint violated (>= 1)

Weaknesses

• ❌ Line 1:1 is wrong — reports test-3.md:1:1: instead of the actual timeout-minutes: line (e.g., line 19). IDE click-through navigates to the wrong place.
• ❌ Redundant double prefix — "workflow schema validation failed: GitHub Actions schema validation failed:" — this is confusing and repetitive (says "failed" twice, names two different layers)
• ❌ No source context — no > pointer showing the problematic line
• ❌ Example buried at the end of a long message — easy to miss

Improvement Suggestions

1. Report the actual line number by using findFrontmatterFieldLine("timeout-minutes", ...) just like findFrontmatterFieldLine is used for engine errors. This requires the compiler to resolve the field's line in the source markdown before formatting the error:

   test-3.md:19:1: error: timeout-minutes: value -10 is below minimum of 1
   ```
   
   

2. Simplify the error prefix — remove the double-nesting:

   # Current (confusing):
   workflow schema validation failed: GitHub Actions schema validation failed: timeout-minutes: ...
   
   # Improved:
   timeout-minutes: value -10 is below minimum 1. Example: timeout-minutes: 10
   

3. Add source context lines (same as YAML errors) using the frontmatter source:

   test-3.md:19:1: error: timeout-minutes must be at least 1 (got -10)
    18 | imports:
    19 |   - shared/reporting.md
   > 20 | timeout-minutes: -10
        | ^^^^^^^^^^^^^^^^ must be >= 1
   

4. Promote the example to appear earlier in the message, not buried at the end.

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	69.7/100
Excellent (85+)	1 (Test 2: engine validation)
Good (70-84)	1 (Test 1: YAML syntax)
Poor (40-54)	1 (Test 3: schema validation)
Critical (<40)	0

Quality Assessment: ⚠️ Needs Improvement

• Average score 69.7 < threshold 70
• Test Case 3 score (50) < individual threshold 55
• Critical DX issue: schema constraint errors report wrong line number

---

Priority Improvement Recommendations

🔴 High Priority — Schema validation must point to the correct line

Problem: When validateGitHubActionsSchema fails, the error is formatted with formatCompilerError(markdownPath, "error", ...) which defaults to line 1:1. This means all schema validation errors (not just timeout-minutes) are mis-located in the file.

Root Cause: compiler.go:447 — the schema validator operates on the compiled YAML, not the source markdown, so line numbers from the compiled YAML don't map back to the source. The fix is to use findFrontmatterFieldLine for known fields after schema validation fails.

Solution (in compiler.go around line 445–453):

if err := c.validateGitHubActionsSchema(yamlContent); err != nil {
    // Extract field name from schema error for better line reporting
    fieldLine := 1
    if fieldName := extractSchemaErrorField(err); fieldName != "" {
        if line := findFrontmatterFieldLine(frontmatterLines, frontmatterStart, fieldName); line > 0 {
            fieldLine = line
        }
    }
    formattedErr := formatCompilerErrorWithPosition(markdownPath, fieldLine, 1, "error",
        fmt.Sprintf("schema validation failed: %v", err), err)
    ...
}

Impact: Makes schema validation errors immediately navigable via IDE click-through, same as engine errors.

🔴 High Priority — Remove redundant double-prefix in schema error message

Problem: The error message says "workflow schema validation failed: GitHub Actions schema validation failed:" — two nested "validation failed" phrases that confuse rather than clarify.

Solution (in compiler.go:447):

// Current:
fmt.Sprintf("workflow schema validation failed: %v", err)  // err already says "GitHub Actions schema validation failed: ..."

// Improved: extract the inner error message and present it directly
// Or simplify the outer wrapper to not repeat the word "failed":
fmt.Sprintf("invalid workflow: %v", err)
// Which produces: "invalid workflow: timeout-minutes: must be >= 1. Example: timeout-minutes: 10"

🟡 Medium Priority — Add correct-usage example to YAML syntax errors

Problem: YAML syntax errors show context and a helpful translation ("did you forget a colon?") but don't show what the corrected line would look like.

Affected: frontmatter_error.go — createFrontmatterError and yamlErrorTranslations.

Solution: When a YAML parse error is associated with a known top-level key, append the correction pattern:

// After translating the error message, look up a correction example
if strings.Contains(translatedMsg, "did you forget a colon") && fieldName != "" {
    hint = fmt.Sprintf("Add ':' after the key: %s: (value)", fieldName)
}

🟡 Medium Priority — Add source context to engine validation errors

Engine errors correctly point to the right line but don't show the source code snippet with a > pointer (unlike YAML errors). Minor improvement: populate CompilerError.Context in formatCompilerErrorWithPosition calls from engine validation.

---

Implementation Guide

1. Fix schema error line reporting

File: pkg/workflow/compiler.go (line ~445)

Add a helper to extract the field path from a *jsonschema.ValidationError and use findFrontmatterFieldLine to look it up in the source markdown. The field name is already available via ve.InstanceLocation in enhanceSchemaValidationError.

2. Simplify schema error message prefix

File: pkg/workflow/compiler.go (line 447)

Change "workflow schema validation failed: %v" to "invalid workflow: %v" or simply "%v" — the inner error already identifies the field and constraint.

3. Add doc URL to YAML syntax errors

File: pkg/workflow/frontmatter_error.go

After returning the VSCode-format line, append a reference to the workflow syntax docs (similar to engine errors). E.g., append "\n\nSee: " + constants.DocsWorkflowSyntaxURL.

---

Success Metrics

Track these to measure improvement:

1. All compiler errors point to the correct line/column — zero occurrences of file.md:1:1: for non-line-1 errors
2. Error score ≥ 70 average across next daily quality check
3. Test Case 3 score ≥ 70 after schema line-number fix

---

References:

• §23309439240

Generated by Daily Syntax Error Quality Check workflow
Next check: Runs daily (see workflow schedule)

Generated by Daily Syntax Error Quality Check · ◷

• expires on Mar 22, 2026, 6:14 PM UTC