[syntax-error-quality] Compiler error messages for invalid `engine:` types lack clarity and actionability

[github-actions]

📊 Error Message Quality Analysis

Date: 2026-04-11 | Tests: 2 | Average Score: 59/100 | Status: ⚠️ Needs Improvement

Summary: When a developer writes engine: 123 (wrong type), the compiler surfaces raw JSON Schema jargon ('oneOf' failed, none matched) with no guidance on valid values or correct usage. Type-error messages for the engine field are significantly worse than name-typo errors on the same field.

Test	Workflow	Error Type	Score	Rating
1	sub-issue-closer.md	Category A – engine: 123 (integer instead of string)	38/100	❌ Critical
2	deep-report.md	Category B – engine: copiilot (name typo)	80/100	✅ Good

---

Test 1 — engine: 123 (integer type)

Deduced error output (from pkg/parser/schema_compiler.go → cleanOneOfMessage → validateWithSchemaAndLocation):

sub-issue-closer.md:10:1: error: 'oneOf' failed, none matched:
  - at '/engine': got number, want string
  - at '/engine': got number, want object
```

**Dimension scores**:
- Clarity 8/25 — Pure JSON Schema jargon (`oneOf failed`). A first-time developer won't know what to do.
- Actionability 5/25 — `got number, want string` hints at the type but lists no valid string values.
- Context 16/20 — `file:line:col` with Rust-style code context lines (schema validation provides them).
- Examples 0/15 — No correct-usage example.
- Consistency 9/15 — `file:line:col` format is present but message style is completely different from the name-typo error path.

**Root cause** (`pkg/parser/schema_errors.go → cleanOneOfMessage`): when both `oneOf` branches produce only type-conflict lines (`got number, want string` / `got number, want object`), the function returns the original raw message unchanged, since `meaningful` is empty. No fallback exists to enrich the message for the `engine` field specifically.

---

### Test 2 — `engine: copiilot` (name typo)

**Deduced error output** (from `pkg/workflow/engine_definition.go → Resolve` → `formatCompilerErrorWithPosition`):

```
deep-report.md:18:1: error: invalid engine: copiilot. Valid engines are: claude, codex, copilot, gemini.

Did you mean: copilot?

Example:
engine: copilot

See: (redacted)

Dimension scores:

• Clarity 22/25 — Very clear: invalid value, valid alternatives, suggestion.
• Actionability 23/25 — "Did you mean?" + correct example is excellent.
• Context 10/20 — file:line:col is correct but formatCompilerErrorWithPosition does not populate Context []string, so no source-code snippet is rendered.
• Examples 13/15 — Shows engine: copilot and doc link.
• Consistency 12/15 — Follows standard format.

---

Weaknesses (top 3)

1. cleanOneOfMessage has no fallback for all-type-conflict oneOf failures on known fields.

   • Current: returns raw jargon ('oneOf' failed, none matched).
   • Suggested fix: in pkg/parser/schema_errors.go, add a post-processing step that, when meaningful is empty AND the path is /engine, synthesises a plain-English message: engine must be a string (e.g. 'copilot') or an object — got <type>. Valid engine names: claude, codex, copilot, gemini. See: <docs>.
   • Alternative: add an enum constraint to the string branch of engine_config.oneOf in main_workflow_schema.json so the jsonschema library emits value must be one of 'claude', 'codex', 'copilot', 'gemini' instead of a raw type-mismatch (test in schema_errors_test.go already handles this case).

2. formatCompilerErrorWithPosition called for invalid-engine-name errors does not include source-code context lines.

   • Location: pkg/workflow/compiler_orchestrator_workflow.go lines 48-52.
   • Current: Context []string is empty → no Rust-style snippet rendered.
   • Suggested fix: read 3–5 lines around engineLine from the source file and pass them as Context, mirroring what validateWithSchemaAndLocation already does.

3. No unified error quality for engine type vs name errors. The schema-validation path and the registry-resolution path produce visually different diagnostics for errors on the same field. Developers who see the name-typo error once are trained to expect rich output, and are confused when a type error produces raw JSON Schema text.

---

References

• pkg/parser/schema_errors.go → cleanOneOfMessage
• pkg/workflow/engine_definition.go → Resolve
• pkg/workflow/compiler_orchestrator_workflow.go:48-52
• Workflow run §24280559180

Generated by Daily Syntax Error Quality Check · ● 12.9M · ◷

• expires on Apr 14, 2026, 10:50 AM UTC