[syntax-error-quality] Syntax error messages need improvements — YAML errors show misleading location and jargon

[github-actions]

📊 Error Message Quality Analysis

Analysis Date: 2026-03-13
Test Cases: 3
Average Score: 65.3/100
Status: ⚠️ Needs Improvement

---

Executive Summary

Daily quality analysis of compiler error messages found one critical failure (YAML syntax errors score 46/100) and two acceptable results (engine typo 73/100, negative timeout 77/100). The average of 65.3/100 falls below the 70-point quality threshold.

The root issue is that YAML parse errors are wrapped by formatCompilerError using a hardcoded line 1, column 1 fallback, even though the actual error location is embedded in the goccy [line:col]: formatted string. This produces a misleading IDE link (file.md:1:1) while showing the real location only as internal goccy text — confusing both humans and tooling. Engine validation errors have the same line:1:1 problem, though their message content is excellent.

Key Findings:

• ✅ Strengths: Engine validation errors have "Did you mean?" suggestions, valid-values lists, and documentation links. Schema constraint errors (Test 3) have precise file:line:col location and source context with visual underline.
• ⚠️ Weaknesses: YAML syntax errors expose raw goccy error format with YAML jargon. formatCompilerError always falls back to line:1:col:1, discarding positional info already available in the error text. No actionable fix suggestions for YAML errors.
• ❌ Critical Issue: YAML parse errors score 46/100 — below the critical threshold of 55. The combined file:1:1: error: failed to parse frontmatter:\n[16:1]: mapping values are not allowed in this context format is inconsistent and misleading.

---

Test Case Results

Test Case 1: Invalid YAML Syntax — Score: 46/100 ❌ POOR (below critical threshold 55)

Test Configuration

Workflow: smoke-gemini.md (79 lines — simple)
Error Type: Category A — Invalid YAML syntax
Error Introduced: Line 16: engine gemini (removed colon — engine: with indented id: gemini replaced by bare scalar engine gemini)

Compiler Output

/tmp/test-1.md:1:1: error: failed to parse frontmatter:
[16:1]: mapping values are not allowed in this context
   15 | name: Smoke Gemini
>  16 | engine gemini
   17 | strict: true
```

*(The `file:1:1:` IDE prefix is the `formatCompilerError` fallback; the real location `[16:1]` comes from `yaml.FormatError` embedded inside the message.)*

#### Code Path

1. `pkg/parser/frontmatter_content.go:64` — `yaml.Unmarshal` fails
2. `pkg/parser/yaml_error.go:29` — `yaml.FormatError(err, false, true)` formats with source context
3. Returns `"failed to parse frontmatter:\n[16:1]: mapping values are not allowed..."`
4. `pkg/workflow/compiler.go:84` — `formatCompilerError(markdownPath, "error", err.Error(), err)` wraps with `Line:1, Column:1`
5. `pkg/workflow/compiler_error_formatter.go:33` — `console.FormatError` produces `file:1:1: error: (multi-line YAML text)`

#### Evaluation Scores

| Dimension | Score | Notes |
|-----------|-------|-------|
| Clarity | 15/25 | "mapping values are not allowed in this context" is raw YAML jargon; `file:1:1` vs `[16:1]` creates confusion |
| Actionability | 10/25 | Source context is shown but no suggestion on what to change. No "Did you forget ':' after 'engine'?" |
| Context | 15/20 | Source lines with `>` pointer are present (via goccy), but IDE link points to wrong line (1 not 16) |
| Examples | 0/15 | No example of correct syntax |
| Consistency | 6/15 | Mixed formats: `file:1:1: error:` wrapping `[16:1]:` text. Other errors use `file:N:M:` correctly. |
| **Total** | **46/100** | **POOR — below critical threshold** |

#### Strengths
- ✅ Source code context is displayed with `>` pointer (thanks to `yaml.FormatError(..., true)`)
- ✅ The actual error line IS shown — just buried in the text

#### Weaknesses
- ❌ `formatCompilerError` uses `Line:1, Column:1` fallback — IDE jumps to the wrong line
- ❌ "mapping values are not allowed in this context" is YAML parser jargon, not plain English
- ❌ No hint that the fix is simply adding `:` after the key name
- ❌ No example of correct YAML syntax
- ❌ `failed to parse frontmatter:` + `[16:1]:` + `file:1:1:` is three different location indicators, all inconsistent

#### Improvement Suggestions

1. **Parse `[line:col]` from `FormatYAMLError` output and use `formatCompilerErrorWithPosition`**:
   ```go
   // In pkg/workflow/compiler.go (~line 77-84):
   // Instead of formatCompilerError (line:1, col:1), extract line/col from FormatYAMLError
   line, col := extractLineColFromYAMLError(err)
   return formatCompilerErrorWithPosition(markdownPath, line, col, "error", err.Error(), err)
   ```

2. **Translate common YAML error messages** in `pkg/parser/yaml_error.go`:
   ```go
   var yamlErrorTranslations = map[string]string{
       "mapping values are not allowed in this context": "Expected 'key: value' — did you forget the ':' after the key?",
       "did not find expected key": "Check indentation — unexpected content at this level",
   }
   ```

3. **Add "Correct usage" hint** for YAML syntax errors:
   ```
   /tmp/test-1.md:16:1: error: Expected 'key: value' format
   >  16 | engine gemini
            ^^^
   Hint: Did you forget ':' after 'engine'?
   Correct usage: engine: copilot
   ```

</details>

<details>
<summary><b>Test Case 2: Invalid Engine Name</b> — Score: 73/100 ✅ Good</summary>

#### Test Configuration

**Workflow**: `dead-code-remover.md` (236 lines — medium)
**Error Type**: Category B — Invalid configuration value
**Error Introduced**: Line 11: `engine: copiilot` (extra `i` — typo for `copilot`)

#### Compiler Output

```
/tmp/test-2.md:1:1: error: invalid engine: copiilot. Valid engines are: claude, codex, copilot, gemini.

Did you mean: copilot?

Example:
engine: copilot

See: https://docs.github.com/en/enterprise-cloud@latest/actions/using-github-copilot/using-copilot-in-github-actions
```

#### Code Path

1. `pkg/parser/schema_validation.go` — schema passes (engine string type is unconstrained in JSON schema)
2. `pkg/workflow/compiler_orchestrator_engine.go:254` — `c.engineCatalog.Resolve("copiilot", ...)` fails
3. `pkg/workflow/engine_definition.go:264` — builds multi-line error with "Did you mean?" suggestion
4. Propagates to `pkg/workflow/compiler.go:84` — wrapped by `formatCompilerError` with `Line:1, Column:1`

#### Evaluation Scores

| Dimension | Score | Notes |
|-----------|-------|-------|
| Clarity | 21/25 | "invalid engine: copiilot" is clear. File path shown. `1:1` is slightly misleading. |
| Actionability | 23/25 | "Did you mean: copilot?", valid engines, example, docs link — excellent |
| Context | 7/20 | File path present but line points to 1 (wrong). No source context showing the `engine:` line. |
| Examples | 14/15 | `engine: copilot` example + documentation link |
| Consistency | 8/15 | `file:1:1:` format (wrong line). Test 3 gets precise location; engine errors do not. |
| **Total** | **73/100** | **Good** |

#### Strengths
- ✅ "Did you mean: copilot?" typo correction is excellent DX
- ✅ Lists all valid engines
- ✅ Shows `engine: copilot` usage example
- ✅ Links to documentation
- ✅ File path is present

#### Weaknesses
- ⚠️ Line/column is `1:1` — IDE navigation points to top of file, not the `engine:` field
- ⚠️ No source context (the `engine: copiilot` line is not shown)

#### Improvement Suggestions

1. **Surface the `engine:` line number via schema-level validation** or by using `formatCompilerErrorWithPosition` with a field-lookup step before returning the engine error.
2. **Add source context** — even a single-line context showing `engine: copiilot` would significantly improve the error.

</details>

<details>
<summary><b>Test Case 3: Negative Timeout Value</b> — Score: 77/100 ✅ Good</summary>

#### Test Configuration

**Workflow**: `code-simplifier.md` (406 lines — complex)
**Error Type**: Category C — Semantic/range error
**Error Introduced**: Line 14: `timeout-minutes: -10` (negative value, schema `minimum: 1`)

#### Compiler Output

```
/tmp/test-3.md:14:18: error: must be at least 1 (got -10)
   12 |
   13 | tracker-id: code-simplifier
>  14 | timeout-minutes: -10
                        ~~~
   15 |
   16 | imports:
   17 |   - shared/reporting.md
```

#### Code Path

1. `pkg/parser/schema_validation.go` — `validateWithSchemaAndLocation` finds `minimum` constraint violation
2. `pkg/parser/schema_errors.go:26` — `translateSchemaConstraintMessage`: `"minimum: got -10, want 1"` → `"must be at least 1 (got -10)"`
3. `pkg/parser/schema_compiler.go:261` — `LocateJSONPathInYAMLWithAdditionalProperties` finds precise column
4. `pkg/parser/schema_compiler.go:305-318` — `console.FormatError` with precise `file:14:18`

#### Evaluation Scores

| Dimension | Score | Notes |
|-----------|-------|-------|
| Clarity | 22/25 | "must be at least 1 (got -10)" is very clear |
| Actionability | 17/25 | Clear what the constraint is, but no suggestion for a valid value |
| Context | 19/20 | Precise file:line:col, 7-line source window with `~~~` underline |
| Examples | 5/15 | No "e.g. timeout-minutes: 30" shown. Schema has `examples: [5, 10, 30]` but unused. |
| Consistency | 14/15 | Proper IDE-parseable format, consistent with other schema errors |
| **Total** | **77/100** | **Good** |

#### Strengths
- ✅ Precise `file:14:18:` — IDE navigation works correctly
- ✅ Source context with visual `~~~` underline at exact error column
- ✅ Plain-English constraint message ("must be at least 1")
- ✅ Shows the bad value in parentheses ("got -10")

#### Weaknesses
- ⚠️ No example of valid value (schema has `examples: [5, 10, 30]` but they're not surfaced)
- ⚠️ Message could add: `Hint: Valid values: 5, 10, 30 (see examples in schema)`

#### Improvement Suggestion

Surface schema `examples` in constraint errors:
```
/tmp/test-3.md:14:18: error: must be at least 1 (got -10)
>  14 | timeout-minutes: -10
                        ~~~
Example: timeout-minutes: 30

---

Overall Statistics

Metric	Value
Tests Run	3
Average Score	65.3/100
Excellent (85+)	0
Good (70-84)	2
Acceptable (55-69)	0
Poor (40-54)	1 ← critical
Critical (<40)	0

Quality Assessment: ⚠️ Needs Improvement — Average 65.3/100 is below the 70-point threshold, and Test Case 1 (YAML syntax errors) scores 46/100, below the critical threshold of 55.

---

Priority Improvement Recommendations

🔴 High Priority — Fix YAML parse error location (pkg/workflow/compiler.go)

The formatCompilerError fallback at line 84 always uses Line:1, Column:1. For YAML parse errors, the actual line/column is already available in the error text produced by FormatYAMLError. The fix is to extract it and use formatCompilerErrorWithPosition:

// pkg/workflow/compiler.go ~line 77-84
if err := ...; err != nil {
    if isAlreadyFormatted(err) {
        return err
    }
    // NEW: extract line/col from YAML error instead of defaulting to 1:1
    line, col := parser.ExtractLineColFromYAMLError(err)
    if line > 0 {
        return formatCompilerErrorWithPosition(markdownPath, line, col, "error", err.Error(), err)
    }
    return formatCompilerError(markdownPath, "error", err.Error(), err)
}

Impact: Fixes IDE navigation for ALL YAML syntax errors (not just this test case).

🔴 High Priority — Translate YAML error jargon (pkg/parser/yaml_error.go)

The goccy YAML parser produces technical messages like "mapping values are not allowed in this context". These should be translated to plain English:

// pkg/parser/yaml_error.go
var yamlJargonTranslations = map[string]string{
    "mapping values are not allowed in this context": "unexpected value — check that all keys use 'key: value' format",
    "did not find expected key":                      "unexpected content — check indentation",
    "found character that cannot start any token":    "invalid character in YAML — check for special characters",
    "could not find expected ':'":                    "missing ':' — all keys require a colon after them",
}

🟡 Medium Priority — Add source context to engine validation errors (pkg/workflow/engine_definition.go)

Engine validation errors currently show file:1:1: with no source context. Since the engine field location is known (it's a top-level YAML key), the error should include the engine: line:

// Currently (engine_definition.go:258):
errMsg := fmt.Sprintf("invalid engine: %s. Valid engines are: %s.\n\nDid you mean: %s?...", ...)
return nil, fmt.Errorf("%s", errMsg)

// Better: return a structured error that includes location info,
// OR catch this at schema validation time in validateEngineSpecificRules()
// where file location is already available
```

Alternatively, restore engine enum validation in the JSON schema (`engine_config` string variant) so it's caught by `validateWithSchemaAndLocation` which already provides precise location.

#### 🟢 Low Priority — Surface schema `examples` in constraint errors

For `minimum`/`maximum` violations, append schema examples from the field definition:

```
/tmp/test-3.md:14:18: error: must be at least 1 (got -10)
>  14 | timeout-minutes: -10
Example: timeout-minutes: 30

---

Implementation Guide

Key Files to Modify

File	Change
pkg/workflow/compiler.go	Extract YAML error line/col before calling formatCompilerError
pkg/parser/yaml_error.go	Add yamlJargonTranslations map, apply in FormatYAMLError
pkg/workflow/engine_definition.go	Propagate location info OR add engine enum to schema
pkg/parser/schema_errors.go	Surface schema field examples in constraint messages

Suggested Unit Tests

// pkg/workflow/compiler_test.go
func TestYAMLSyntaxErrorHasCorrectLineNumber(t *testing.T) {
    // engine gemini (missing colon) should produce error at line 16, not line 1
}

// pkg/workflow/engine_definition_test.go  
func TestEngineValidationErrorHasFileLocation(t *testing.T) {
    // invalid engine should show the engine: field line, not line 1
}

---

References:

• §23063728061

Generated by Daily Syntax Error Quality Check · ◷

• expires on Mar 16, 2026, 6:15 PM UTC