Translate YAML/schema jargon into actionable error messages

[Copilot]

• Translate raw YAML error messages ("non-map value is specified") to plain English in pkg/workflow/frontmatter_error.go
• Add cleanOneOfMessage function in pkg/parser/schema_errors.go to filter "got string, want object" branches and simplify oneOf errors
• Apply oneOf cleanup in pkg/parser/schema_compiler.go for both precise location and fallback error paths
• Add tests for YAML message translation in pkg/workflow/yaml_message_translation_test.go
• Add tests for oneOf message cleanup in pkg/parser/schema_errors_test.go
• Fix TestStepValidation integration test to check for improved "missing property" message
• All tests pass including integration tests

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[syntax-error-quality] Compiler error messages need clearer language and actionable guidance</issue_title>
<issue_description>### 📊 Error Message Quality Analysis

Analysis Date: 2026-02-19
Test Cases: 3
Average Score: 56.7/100
Status: ⚠️ Needs Improvement

---

Executive Summary

Three compiler error scenarios were tested across different error categories: a YAML syntax error (missing colon), an invalid engine name typo, and an invalid permission value. All three fell in the "Acceptable" range (55–58/100), averaging 56.7/100 — below the 70-point quality threshold.

The compiler's structural presentation (file:line:column format, source code context with > marker) is solid. The primary weaknesses are: raw parser/schema jargon leaking through to users ("non-map value is specified", "oneOf failed, none matched"), the absence of "did you mean?" suggestions for typos, no examples of correct syntax, and some errors pointing to parent fields rather than the specific offending line.

Key Findings:

• Strengths: Consistent file:line:col format; good source context with > marker and ^ pointer; valid-values lists for enum fields
• Weaknesses: Schema/YAML-parser jargon not translated to plain language; no "did you mean?" fuzzy matching; no corrected-syntax examples
• Critical Issues: Test 1's "non-map value is specified" message provides zero guidance on how to fix a missing colon, a very common beginner mistake

---

Test Case Results

Test Case 1: Missing colon in frontmatter key (workflow-generator.md) — Score: 55/100 ⚠️

Test Configuration

Workflow: workflow-generator.md (108 lines, simple)
Error Category: A — Frontmatter YAML Syntax
Error Introduced: Removed : from engine: copilot → engine copilot (line 15)

Compiler Output

✗ Compiled 1 workflow(s): 1 error(s), 0 warning(s)

✗ Failed workflows:
  ✗ test-1.md

/tmp/.../test-1.md:15:1: error: non-map value is specified
  12 |   contents: read
  13 |   issues: read
  14 |   pull-requests: read
> 15 | engine copilot
       ^
  16 | tools:
  17 |   github:
  18 |     lockdown: true

✗ compilation failed
```

#### Evaluation Scores

| Dimension | Score | Notes |
|-----------|-------|-------|
| Clarity | 14/25 | "non-map value is specified" is YAML parser jargon; not obvious it means "missing colon" |
| Actionability | 8/25 | No hint to add `:` after `engine`; no suggestion |
| Context | 18/20 | Excellent: 4 lines of context, `>` marker, `^` pointer |
| Examples | 2/15 | No example of correct syntax |
| Consistency | 13/15 | IDE-parseable `file:line:col` format ✓ |
| **Total** | **55/100** | **Acceptable** |

#### Strengths
- ✅ Source context with `>` line indicator and `^` caret pointer
- ✅ Consistent `file:line:column: error:` format for IDE integration
- ✅ Shows 4 lines before and after the error

#### Weaknesses
- ❌ "non-map value is specified" — raw YAML parser output, not translated to developer-friendly language
- ❌ `^` pointer placed at column 1 (start of line), not under the space where `:` is missing
- ❌ No suggestion ("Did you mean `engine: copilot`?")
- ❌ No correct-syntax example

#### Improvement Suggestions

1. **Translate raw YAML errors to plain language**
   - Current: `error: non-map value is specified`
   - Better: `error: invalid YAML syntax — expected 'key: value' format. Did you mean 'engine: copilot'?`

2. **Move `^` pointer to the gap where the colon is missing**
   ```
   > 15 | engine copilot
               ^ missing ':' here
   ```

3. **Add a "Correct usage" line**
   ```
   Correct: engine: copilot
   ```

</details>

<details>
<summary><b>Test Case 2: Typo in engine name (audit-workflows.md)</b> — Score: 57/100 ⚠️</summary>

#### Test Configuration

**Workflow**: `audit-workflows.md` (174 lines, medium)
**Error Category**: B — Configuration Error (invalid engine name)
**Error Introduced**: `engine: claude` → `engine: claud` (one character removed)

#### Compiler Output

```
✗ Compiled 1 workflow(s): 1 error(s), 0 warning(s)

✗ Failed workflows:
  ✗ test-2.md

/tmp/.../test-2.md:12:8: error: at '/engine': 'oneOf' failed, none matched
- at '/engine': value must be one of 'claude', 'codex', 'copilot'
- at '/engine': got string, want object
  9 |   issues: read
 10 |   pull-requests: read
 11 | tracker-id: audit-workflows-daily
 12 | engine: claud
             
 13 | tools:
 14 |   agentic-workflows:
 15 |   repo-memory:

✗ compilation failed
```

#### Evaluation Scores

| Dimension | Score | Notes |
|-----------|-------|-------|
| Clarity | 14/25 | "oneOf failed, none matched" is JSON schema jargon; second branch "got string, want object" is misleading since valid engines ARE strings |
| Actionability | 17/25 | Lists valid values (`claude`, `codex`, `copilot`) — helpful; but no "did you mean: claude?" |
| Context | 12/20 | Shows surro...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes github/gh-aw#16895

<!-- START COPILOT CODING AGENT TIPS -->
---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security)

[pelikhan]

Fix tests in https://github.com/github/gh-aw/actions/runs/22199884178/job/64211002481

[Copilot]

Pull request overview

This pull request improves compiler error message quality by translating raw YAML parser and JSON schema validation jargon into plain, actionable English. The goal is to increase error message quality scores from an average of 56.7/100 to above the 70-point threshold.

Changes:

• Added YAML syntax error translation that converts raw goccy/go-yaml messages (e.g., "non-map value is specified") into user-friendly guidance (e.g., "Invalid YAML syntax: expected 'key: value' format")
• Implemented oneOf error cleanup that removes confusing schema jargon like "'oneOf' failed, none matched" and filters out misleading type-mismatch lines from irrelevant schema branches
• Applied error message cleanup consistently across both precise-location and fallback error paths

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
pkg/workflow/yaml_message_translation_test.go	Comprehensive tests for YAML error message translation, covering various error patterns and edge cases
pkg/workflow/frontmatter_error.go	Implements YAML error translation table and applies it during frontmatter error creation
pkg/parser/schema_errors_test.go	Tests for oneOf cleanup logic, type conflict detection, and path prefix stripping
pkg/parser/schema_errors.go	Core implementation of oneOf message cleanup, type conflict detection, and path prefix stripping utilities
pkg/parser/schema_compiler.go	Integrates oneOf cleanup into the precise-location error path for schema validation errors

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[pelikhan]

@copilot Fix tests in https://github.com/github/gh-aw/actions/runs/22199884178/job/64211002481

[Copilot]

@copilot Fix tests in https://github.com/github/gh-aw/actions/runs/22199884178/job/64211002481

Fixed in ca6cd19. The cleanOneOfMessage() was improving the error for "step without uses or run" from 'oneOf' failed to '0': missing property 'uses'; '0': missing property 'run', but the test was still checking for the old "oneOf" string. Updated the test expectation to "missing property" to match the improved message.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• https://api.github.com/repos/actions/checkout/git/ref/tags/v4
  • Triggering command: /usr/bin/gh gh api /repos/actions/checkout/git/ref/tags/v4 --jq .object.sha (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)