Repository Quality Improvement: Agentic Engine Feature Parity (2026-03-18)

[github-actions[bot]]

🎯 Repository Quality Improvement Report — Agentic Engine Feature Parity

Analysis Date: 2026-03-18
Focus Area: Agentic Engine Feature Parity
Strategy Type: Custom (repository-specific)
Custom Area: Yes — gh-aw's core value proposition is multi-engine support (copilot, claude, codex, gemini). Inconsistencies in how engines are handled in the compiler can silently degrade behavior for newer engines like Gemini, making this a high-value area to audit.

Executive Summary

gh-aw supports four built-in AI engines: copilot, claude, codex, and gemini. While copilot, claude, and codex have been in the project longer, gemini was added more recently. Analysis of the compiler and engine-handling code reveals several parity gaps where gemini falls through to a default branch that uses incorrect constants or returns empty values, while the other three engines have explicit, correct handling.

Two bugs were identified that affect Gemini workflows in production:

1. compiler_yaml.go uses GH_AW_MODEL_AGENT_CUSTOM instead of GH_AW_MODEL_AGENT_GEMINI for the model info env var when a model is not explicitly configured.
2. compiler_yaml_helpers.go returns an empty string for the installation version instead of DefaultGeminiVersion.

Additionally, plugin installation has TODOs for Claude and Codex syntax validation, and Gemini has no plugin install handling at all.

Full Analysis Report

Focus Area: Agentic Engine Feature Parity

Current State Assessment

Metrics Collected:

Metric	Value	Status
Engines supported	4 (copilot, claude, codex, gemini)	✅
Engine-specific .go files	copilot ✅, claude ✅, codex ✅, gemini ✅	✅
compiler_yaml.go model switch coverage	copilot ✅, claude ✅, codex ✅, gemini ❌	❌
compiler_yaml_helpers.go version switch coverage	copilot ✅, claude ✅, codex ✅, gemini ❌	❌
Plugin install switch coverage (gemini)	Not handled (falls to generic)	⚠️
Plugin install TODO items	2 (claude, codex syntax unvalidated)	⚠️
domains.go engine switch coverage	All 4 engines ✅	✅

Findings

Strengths

• domains.go correctly handles all four engines in its switch statement
• Each engine has a dedicated *_engine.go file with engine-specific logic
• Constants package has EnvVarModelAgentGemini and DefaultGeminiVersion properly defined
• Gemini engine correctly uses its native GEMINI_MODEL env var for explicit model configuration

Areas for Improvement

[HIGH] compiler_yaml.go — Missing Gemini model env var case

In pkg/workflow/compiler_yaml.go lines 558–567, the modelEnvVar switch does not have a "gemini" case. Gemini falls to default: constants.EnvVarModelAgentCustom, so the compiled workflow sets:

GH_AW_INFO_MODEL: $\{\{ vars.GH_AW_MODEL_AGENT_CUSTOM || '' }}

…instead of the correct:

GH_AW_INFO_MODEL: $\{\{ vars.GH_AW_MODEL_AGENT_GEMINI || '' }}

This means users who configure vars.GH_AW_MODEL_AGENT_GEMINI in their repository to override the default Gemini model will have their setting silently ignored.

[HIGH] compiler_yaml_helpers.go — Missing Gemini version in getInstallationVersion

In pkg/workflow/compiler_yaml_helpers.go, getInstallationVersion() does not have a "gemini" case. Gemini falls to default: return "", so GH_AW_INFO_AGENT_VERSION will be an empty string in compiled Gemini workflows instead of "latest" (the value of constants.DefaultGeminiVersion).

[MEDIUM] plugin_installation.go — Unvalidated plugin CLI syntax for Claude and Codex

Two TODO comments in generatePluginInstallStep() note that the claude plugin install and codex plugin install command syntax has not been validated. If those CLIs use different subcommand structures, the generated steps would fail silently.

[MEDIUM] plugin_installation.go — No explicit Gemini case

Gemini falls to the generic default handler which uses fmt.Sprintf("%s plugin install %s", engineID, plugin). This may be intentional, but should be documented or have an explicit case.

[LOW] compiler_yaml.go — Gemini case missing in "custom" engine fallthrough comment

The "custom" engine has a dedicated case that clearly signals intent, but Gemini (a first-class engine) hits the default branch. Adding an explicit case for Gemini improves readability and avoids future regression.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot coding agent execution. Each is self-contained and independently actionable.

Improvement Tasks

---

Task 1: Fix Gemini model env var in compiler_yaml.go

Priority: High
Estimated Effort: Small
Focus Area: Agentic Engine Feature Parity

Description:
In pkg/workflow/compiler_yaml.go, the modelEnvVar switch (around line 558) does not include a "gemini" case. Gemini falls to default and incorrectly uses constants.EnvVarModelAgentCustom. This causes the compiled YAML to reference vars.GH_AW_MODEL_AGENT_CUSTOM instead of vars.GH_AW_MODEL_AGENT_GEMINI for Gemini workflows.

Acceptance Criteria:

• Add case "gemini": modelEnvVar = constants.EnvVarModelAgentGemini to the switch in compiler_yaml.go
• Existing tests pass and a new test case validates Gemini model env var output
• make fmt && make test-unit passes

Code Region: pkg/workflow/compiler_yaml.go (lines ~558–567)

In `pkg/workflow/compiler_yaml.go`, find the switch statement that sets `modelEnvVar` (around line 558):

```go
switch engineID {
case "copilot":
    modelEnvVar = constants.EnvVarModelAgentCopilot
case "claude":
    modelEnvVar = constants.EnvVarModelAgentClaude
case "codex":
    modelEnvVar = constants.EnvVarModelAgentCodex
case "custom":
    modelEnvVar = constants.EnvVarModelAgentCustom
default:
    modelEnvVar = constants.EnvVarModelAgentCustom
}
```

Add a `"gemini"` case before the `"custom"` case:

```go
case "gemini":
    modelEnvVar = constants.EnvVarModelAgentGemini
```

After the fix, run: `make fmt && go test -v -run "TestGemini\|TestCompileGemini" ./pkg/workflow/`

Also add or update a test in `pkg/workflow/compiler_yaml_test.go` (or the appropriate test file) to assert that when a Gemini workflow is compiled without an explicit model, the generated YAML contains `GH_AW_INFO_MODEL: $\{\{ vars.GH_AW_MODEL_AGENT_GEMINI || '' }}` rather than `GH_AW_MODEL_AGENT_CUSTOM`.

---

Task 2: Fix Gemini installation version in compiler_yaml_helpers.go

Priority: High
Estimated Effort: Small
Focus Area: Agentic Engine Feature Parity

Description:
In pkg/workflow/compiler_yaml_helpers.go, getInstallationVersion() does not include a "gemini" case. Gemini falls to default: return "", causing GH_AW_INFO_AGENT_VERSION to be empty in compiled Gemini workflows instead of "latest" (constants.DefaultGeminiVersion).

Acceptance Criteria:

• Add case "gemini": return string(constants.DefaultGeminiVersion) to getInstallationVersion() in compiler_yaml_helpers.go
• GH_AW_INFO_AGENT_VERSION is non-empty in Gemini compiled output
• make fmt && make test-unit passes

Code Region: pkg/workflow/compiler_yaml_helpers.go — getInstallationVersion() function

In `pkg/workflow/compiler_yaml_helpers.go`, find `getInstallationVersion()`:

```go
switch engineID {
case "copilot":
    return string(constants.DefaultCopilotVersion)
case "claude":
    return string(constants.DefaultClaudeCodeVersion)
case "codex":
    return string(constants.DefaultCodexVersion)
default:
    compilerYamlHelpersLog.Printf("No default version for custom engine: %s", engineID)
    return ""
}
```

Add a `"gemini"` case:

```go
case "gemini":
    return string(constants.DefaultGeminiVersion)
```

After the fix, run: `make fmt && go test -v -run "TestGemini\|TestInstallation" ./pkg/workflow/`

Also add or update a test in `compiler_yaml_helpers_test.go` verifying that `getInstallationVersion` returns `"latest"` for the Gemini engine.

---

Task 3: Validate and document Claude & Codex plugin CLI syntax

Priority: Medium
Estimated Effort: Medium
Focus Area: Agentic Engine Feature Parity

Description:
pkg/workflow/plugin_installation.go has two TODO comments indicating that the claude plugin install and codex plugin install command syntax has not been validated. The Claude CLI and Codex CLI may use different subcommand names or flags. The generated steps will fail silently if the syntax is wrong.

Acceptance Criteria:

• Research and verify the correct plugin installation command syntax for Claude CLI and Codex CLI
• Update the case "claude" and case "codex" branches with the validated command syntax (or a verified placeholder if plugins aren't supported yet)
• Remove the TODO comments once resolved, or replace them with a // NOTE: explaining any limitations
• Add a unit test asserting the generated step contains the correct command string for each engine
• make fmt && make test-unit passes

Code Region: pkg/workflow/plugin_installation.go — generatePluginInstallStep() function

In `pkg/workflow/plugin_installation.go`, review the `generatePluginInstallStep()` function:

```go
case "claude":
    // TODO: validate the correct claude CLI plugin install command syntax
    command = "claude plugin install " + plugin
case "codex":
    // TODO: validate the correct codex CLI plugin install command syntax
    command = "codex plugin install " + plugin
```

Research the actual plugin install commands for:
1. **Claude CLI** — check (docs.anthropic.com/redacted) or the claude CLI `--help` output
2. **Codex CLI** — check the OpenAI Codex CLI documentation or `codex --help`

Update the command strings with the verified syntax. If the CLIs don't support plugins at all yet, set the command to something that will fail gracefully and add a clear comment explaining the limitation.

Also add an explicit `"gemini"` case documenting that Gemini uses the generic fallback (if that is intentional), or add the correct syntax if Gemini CLI supports plugins.

Add unit tests in `pkg/workflow/` (or update existing plugin_installation tests) asserting the correct command string per engine.

---

Task 4: Add engine parity integration test

Priority: Medium
Estimated Effort: Medium
Focus Area: Agentic Engine Feature Parity

Description:
There is no systematic test that verifies all four engines produce consistent metadata fields in compiled output. A table-driven parity test would catch future regressions when new engines are added or engine-specific switch statements are extended for some engines but not others.

Acceptance Criteria:

• Create pkg/workflow/compiler_engine_parity_test.go with a table-driven test covering all four engines
• Test verifies that GH_AW_INFO_MODEL references the correct engine-specific vars variable (not GH_AW_MODEL_AGENT_CUSTOM for non-custom engines)
• Test verifies that GH_AW_INFO_AGENT_VERSION is non-empty for all four built-in engines
• Build tag //go:build !integration at top of file
• make fmt && make lint && go test -v -run "TestEngineParity" ./pkg/workflow/ passes

Code Region: pkg/workflow/ (new file: compiler_engine_parity_test.go)

Create a new test file `pkg/workflow/compiler_engine_parity_test.go` with build tag `//go:build !integration`.

Write a table-driven test `TestEngineParity_InfoVariables` that compiles a minimal workflow (one that sets no explicit model or version) for each of the four engines: copilot, claude, codex, gemini.

For each engine, assert:
1. The compiled YAML contains `GH_AW_INFO_MODEL: $\{\{ vars.GH_AW_MODEL_AGENT_(ENGINE) || '' }}` (not `GH_AW_MODEL_AGENT_CUSTOM` for non-custom engines)
2. The compiled YAML contains a non-empty `GH_AW_INFO_AGENT_VERSION` value
3. `GH_AW_INFO_ENGINE_ID` matches the engine ID

Use the existing test helper patterns from `compiler_yaml_test.go` for compilation.

Expected model env var per engine:
- copilot → `GH_AW_MODEL_AGENT_COPILOT`
- claude  → `GH_AW_MODEL_AGENT_CLAUDE`
- codex   → `GH_AW_MODEL_AGENT_CODEX`
- gemini  → `GH_AW_MODEL_AGENT_GEMINI`

Expected version per engine (when no version configured):
- copilot → non-empty (e.g. "latest")
- claude  → non-empty (e.g. "latest")
- codex   → non-empty (e.g. "latest")
- gemini  → "latest"

---

Task 5: Document engine feature matrix in AGENTS.md or docs

Priority: Low
Estimated Effort: Small
Focus Area: Agentic Engine Feature Parity

Description:
There is no single reference showing which features are supported per engine and which constants/env vars apply. An engine feature matrix would help contributors avoid introducing parity gaps when adding new engine-specific logic.

Acceptance Criteria:

• Add a brief "Engine Feature Matrix" section to AGENTS.md or DEVGUIDE.md (whichever is more appropriate for contributors)
• Table lists each engine and its: model agent env var, model detection env var, CLI model env var (if different), default version constant, plugin support status
• Section includes a note: "When adding a new switch case for one engine, always check if the other engines need matching cases"

Code Region: AGENTS.md or DEVGUIDE.md

Add an "Engine Feature Matrix" section to `DEVGUIDE.md` (under a heading like "### Engine Constants Reference") with a table:

| Engine  | Agent Model Env Var            | Detection Model Env Var           | CLI Model Env Var | Default Version Constant    | Plugin Support |
|---------|-------------------------------|----------------------------------|-------------------|-----------------------------|----------------|
| copilot | `GH_AW_MODEL_AGENT_COPILOT`   | `GH_AW_MODEL_DETECTION_COPILOT`  | (native)          | `DefaultCopilotVersion`     | ✅ validated   |
| claude  | `GH_AW_MODEL_AGENT_CLAUDE`    | `GH_AW_MODEL_DETECTION_CLAUDE`   | (native)          | `DefaultClaudeCodeVersion`  | ⚠️ TODO        |
| codex   | `GH_AW_MODEL_AGENT_CODEX`     | `GH_AW_MODEL_DETECTION_CODEX`    | (native)          | `DefaultCodexVersion`       | ⚠️ TODO        |
| gemini  | `GH_AW_MODEL_AGENT_GEMINI`    | `GH_AW_MODEL_DETECTION_GEMINI`   | `GEMINI_MODEL`    | `DefaultGeminiVersion`      | ⚠️ generic     |

And add a note: "When adding a new switch case that covers copilot/claude/codex, always add a matching case for gemini (and vice versa). Check compiler_yaml.go, compiler_yaml_helpers.go, and plugin_installation.go."

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-03-18	Agentic Engine Feature Parity	Custom	Yes	Found 2 Gemini bugs, 2 plugin TODOs, proposed parity test

---

🎯 Recommendations

Immediate Actions (This Week)

1. Fix Gemini model env var (compiler_yaml.go) — Priority: High — 15-minute fix
2. Fix Gemini installation version (compiler_yaml_helpers.go) — Priority: High — 15-minute fix

Short-term Actions (This Month)

1. Validate Claude/Codex plugin syntax — Priority: Medium — requires CLI research
2. Add engine parity integration test — Priority: Medium — prevents future regressions

Long-term Actions (This Quarter)

1. Add engine feature matrix to contributor docs — Priority: Low — improves contributor onboarding

---

📈 Success Metrics

Track these metrics to measure improvement in Agentic Engine Feature Parity:

• Switch case coverage: All engine switch statements in compiler_yaml.go, compiler_yaml_helpers.go, plugin_installation.go cover all 4 engines → Target: 100%
• Parity test existence: Engine parity test exists and covers all 4 engines → Target: exists
• TODO count in engine files: Currently 2 (claude, codex plugin syntax) → Target: 0

---

Next Steps

1. Review and prioritize the 5 tasks above
2. Assign Tasks 1 & 2 to the Copilot coding agent immediately (quick wins, high impact)
3. Schedule Tasks 3 & 4 for this week
4. Re-evaluate engine parity after Gemini feature additions

---

Generated by Repository Quality Improvement Agent
Next analysis: 2026-03-19 — Focus area will be selected based on diversity algorithm

References: §23247183495

AI generated by Repository Quality Improvement Agent · history

• expires on Mar 19, 2026, 1:42 PM UTC

[github-actions[bot]]

This discussion has been marked as outdated by Repository Quality Improvement Agent.

A newer discussion is available at Discussion #21807.