Repository Quality Report: AI Engine Feature Parity & Completeness (2026-03-20)

[github-actions[bot]]

🎯 Repository Quality Improvement Report — AI Engine Feature Parity

Analysis Date: 2026-03-20
Focus Area: AI Engine Feature Parity & Completeness
Strategy Type: Custom (first run, no prior history)
Custom Area: Yes — This project supports four AI coding engines (Copilot, Claude, Codex, Gemini). Each engine has different capabilities, and parity gaps can cause silent bugs, misleading errors, or poor user experience. This focus area examines how consistently each engine is implemented, validated, documented, and tested.

---

Executive Summary

The repository supports four AI coding engines — Copilot, Claude, Codex, and Gemini — via a well-structured BaseEngine + per-engine type hierarchy. The architecture is sound: a clean interface defines capabilities (SupportsMaxTurns, SupportsPlugins, SupportsWebSearch, etc.) and each engine opts in or out at construction time.

However, two concrete gaps stand out. First, plugin_installation.go contains unresolved TODO comments for both Claude and Codex engines, indicating the plugin install command syntax has not been verified against the actual CLIs. Second, the supportsMaxContinuations capability flag is only set for Copilot — Claude, Codex, and Gemini silently default to false, with no comment explaining whether this is intentional or an oversight. Additionally, no engine feature support matrix exists in the docs, making it hard for users to know which features are available per engine.

All four engines have smoke tests and are covered in the reference docs, which is a healthy baseline.

Full Analysis Report

Focus Area: AI Engine Feature Parity & Completeness

Current State Assessment

Metrics Collected:

Metric	Value	Status
Engines implemented	4 (copilot, claude, codex, gemini)	✅
Engines with smoke tests	4/4	✅
Engine docs page exists	Yes (docs/src/content/docs/reference/engines.md)	✅
Plugin install TODOs	2 (claude, codex)	⚠️
supportsMaxContinuations explicitly set	1/4 (Copilot only)	⚠️
Feature matrix in docs	Missing	❌
supportsPlugins explicitly set for claude/codex	Not set (defaults to false)	⚠️

Engine Feature Matrix (discovered via code analysis)

Feature	Copilot	Claude	Codex	Gemini
supportsMaxTurns	❌	✅	❌	❌
supportsMaxContinuations	✅	❌ (not set)	❌ (not set)	❌ (not set)
supportsWebFetch	✅	✅	❌	❌
supportsWebSearch	❌	✅	✅	❌
supportsPlugins	✅	❌ (not set)	❌ (not set)	❌
supportsToolsAllowlist	✅	✅	✅	✅
Plugin install command verified	✅	⚠️ TODO	⚠️ TODO	N/A

Findings

Strengths

• Clean BaseEngine + interface pattern makes feature capability well-structured
• All four engines have smoke test workflows (.md + compiled .lock.yml)
• Engine reference docs cover version pinning, secrets, custom agents, and env vars
• engine_validation.go correctly gates plugins on SupportsPlugins() before installing

Areas for Improvement

• [HIGH] plugin_installation.go:68-71 contains unresolved TODO comments for Claude and Codex plugin command syntax — users enabling plugins: on these engines may get wrong commands silently generated
• [MEDIUM] supportsMaxContinuations is only explicitly set for Copilot; Claude, Codex, and Gemini inherit false from BaseEngine with no explanatory comment confirming intent
• [MEDIUM] No engine feature comparison table exists in the docs — users have no easy way to know which features (max-turns, plugins, web-search, web-fetch) work with which engine
• [LOW] Gemini engine doesn't support supportsMaxTurns, supportsWebFetch, supportsWebSearch, or supportsPlugins — while this may be accurate, there's no doc note explaining the limitation

Detailed Analysis

Plugin Command Syntax TODOs (pkg/workflow/plugin_installation.go lines 68–71):

case "claude":
    // TODO: validate the correct claude CLI plugin install command syntax
    command = "claude plugin install " + plugin
case "codex":
    // TODO: validate the correct codex CLI plugin install command syntax
    command = "codex plugin install " + plugin

These TODOs indicate that the assumed command format ((engine) plugin install (repo)) was not confirmed. The Claude Code CLI uses claude mcp add for MCP servers, so claude plugin install may be entirely wrong.

MaxContinuations gap: The supportsMaxContinuations field in BaseEngine defaults to false. Copilot explicitly sets it to true. Claude, Codex, and Gemini never set it — it's unclear whether these engines support a similar --max-turns or continuation mechanism that hasn't been mapped yet.

---

🤖 Tasks for Copilot Agent

NOTE TO PLANNER AGENT: The following tasks are designed for GitHub Copilot coding agent execution. Please split these into individual work items for Claude to process.

Improvement Tasks

---

Task 1: Research and Fix Plugin Install Commands for Claude and Codex

Priority: High
Estimated Effort: Small
Focus Area: AI Engine Feature Parity

Description:
pkg/workflow/plugin_installation.go has TODO markers for the Claude and Codex plugin install command syntax. Research the current official plugin/extension install commands for each CLI and update the code, or add a validation error if plugins are not supported.

Acceptance Criteria:

• Research and confirm correct claude CLI command for installing a plugin (if supported) — check the Claude Code CLI docs for plugin or extension support
• Research and confirm correct codex CLI command for installing a plugin (if supported)
• Update generatePluginInstallStep() in pkg/workflow/plugin_installation.go to use the correct command for each engine
• If an engine does not support plugins, set supportsPlugins: false in its engine constructor and remove the TODO comment
• Add/update tests in pkg/workflow/plugin_installation_test.go (or create it) to cover the claude and codex cases
• Run make agent-finish before committing

Code Region: pkg/workflow/plugin_installation.go (lines 60–90)

In `pkg/workflow/plugin_installation.go`, the `generatePluginInstallStep` function has TODO markers for claude and codex plugin command syntax. 

Research the correct plugin install commands:
- For Claude Code CLI: check if `claude plugin install (repo)` is a valid command, or if plugin support works differently
- For OpenAI Codex CLI: check if `codex plugin install (repo)` is a valid command

If either CLI does NOT support plugins, update that engine's constructor in `claude_engine.go` or `codex_engine.go` to set `supportsPlugins: false` (which will cause the engine validation to reject `plugins:` in frontmatter with a helpful error) and remove the TODO comment.

If the command IS valid, update it and remove the TODO comment.

Follow the existing code patterns and run `make agent-finish` before committing.

---

Task 2: Add Explicit supportsMaxContinuations to All Engines

Priority: Medium
Estimated Effort: Small
Focus Area: AI Engine Feature Parity

Description:
The supportsMaxContinuations field is only explicitly set in copilot_engine.go. Claude, Codex, and Gemini inherit false from BaseEngine without a comment indicating this is intentional. Add explicit declarations with comments for all engines to prevent accidental omissions from future contributors.

Acceptance Criteria:

• Add supportsMaxContinuations: false, // (reason) to claude_engine.go constructor with a comment explaining Claude doesn't support --max-autopilot-continues
• Add supportsMaxContinuations: false, // (reason) to codex_engine.go constructor with a comment explaining Codex doesn't support continuation mode
• Add supportsMaxContinuations: false, // (reason) to gemini_engine.go constructor with a comment explaining Gemini doesn't support continuation mode
• Run make agent-finish before committing

Code Region: pkg/workflow/claude_engine.go, pkg/workflow/codex_engine.go, pkg/workflow/gemini_engine.go

In each of `claude_engine.go`, `codex_engine.go`, and `gemini_engine.go`, the `BaseEngine` struct initialization is missing an explicit `supportsMaxContinuations` field. Since `copilot_engine.go` sets it explicitly to `true` with a comment, all other engines should set it to `false` with a brief comment explaining why (e.g., "Claude does not support --max-autopilot-continues style continuation").

Look at how `copilot_engine.go` sets `supportsMaxContinuations: true, // Copilot CLI supports --autopilot with --max-autopilot-continues` and follow the same pattern for the other engines.

Run `make agent-finish` before committing.

---

Task 3: Add Engine Feature Comparison Table to Docs

Priority: Medium
Estimated Effort: Medium
Focus Area: AI Engine Feature Parity & Documentation

Description:
The docs/src/content/docs/reference/engines.md page describes each engine but lacks a feature comparison table. Users cannot easily determine which capabilities (max-turns, max-continuations, plugins, web-search, web-fetch, tools allowlist) are available for each engine without reading engine-specific code.

Acceptance Criteria:

• Add a "## Engine Feature Support" section near the top of docs/src/content/docs/reference/engines.md, after the engine table
• Include a feature comparison table showing: max-turns, max-continuations, plugins, web-search, web-fetch, tools-allowlist per engine
• Feature support should match the Supports*() methods on each engine type
• Add a note for any feature that only works with specific engines (e.g., max-turns only works with Claude)
• Follow the existing docs style (GitHub-flavored Markdown, Astro Starlight conventions)

Code Region: docs/src/content/docs/reference/engines.md

Add a feature comparison table to `docs/src/content/docs/reference/engines.md`. The table should appear after the "## Available Coding Agents" engine table.

The new section should look like:

```markdown
## Engine Feature Support

The following features are only available with specific engines:

| Feature | Copilot | Claude | Codex | Gemini |
|---------|---------|--------|-------|--------|
| `max-turns` | ❌ | ✅ | ❌ | ❌ |
| `max-continuations` | ✅ | ❌ | ❌ | ❌ |
| `plugins` | ✅ | ❌ | ❌ | ❌ |
| `web-search` (built-in) | ❌ | ✅ | ✅ | ❌ |
| `web-fetch` (built-in) | ✅ | ✅ | ❌ | ❌ |
| Tools allowlist | ✅ | ✅ | ✅ | ✅ |
\```

Add brief explanatory notes below the table for features that require special configuration. Follow the existing docs style (see surrounding content for conventions). Do NOT add new frontmatter or change the page title.

---

Task 4: Add Engine Feature Parity Unit Tests

Priority: Low
Estimated Effort: Small
Focus Area: AI Engine Feature Parity & Testing

Description:
There are no tests that assert the feature support matrix for each engine. A simple table-driven test would prevent future regressions where a new engine is added without explicitly setting all capability flags.

Acceptance Criteria:

• Create pkg/workflow/engine_feature_parity_test.go with a table-driven test asserting the expected Supports*() return values for each engine
• Test covers all four engines: Copilot, Claude, Codex, Gemini
• Test covers all Supports*() methods: SupportsMaxTurns, SupportsMaxContinuations, SupportsWebFetch, SupportsWebSearch, SupportsPlugins, SupportsToolsAllowlist
• Add build tag //go:build !integration at the top of the file
• Run make agent-finish before committing

Code Region: pkg/workflow/ (new file engine_feature_parity_test.go)

Create `pkg/workflow/engine_feature_parity_test.go` with a table-driven test that asserts the feature support matrix for all four engines.

The test should:
1. Have build tag `//go:build !integration` at the very top (before the package declaration)
2. Use the standard test patterns from this codebase (testify `assert`, no mocks)
3. Create each engine via its constructor (`NewCopilotEngine(...)`, `NewClaudeEngine(...)`, `NewCodexEngine(...)`, `NewGeminiEngine(...)`)
4. Assert specific `Supports*()` return values for each engine

Example structure:
```go
//go:build !integration

package workflow

import (
    "testing"
    "github.com/stretchr/testify/assert"
)

func TestEngineFeatureParity(t *testing.T) {
    // ...table-driven test asserting Supports*() for each engine
}
\```

Look at existing engine test files (e.g., `gemini_engine_test.go`, `engine_validation_test.go`) for patterns to follow. Run `make agent-finish` before committing.

---

📊 Historical Context

Previous Focus Areas

Date	Focus Area	Type	Custom	Key Outcomes
2026-03-20	AI Engine Feature Parity	Custom	Yes	First run — 4 tasks generated covering plugin command TODOs, MaxContinuations gaps, docs matrix, and parity tests

---

🎯 Recommendations

Immediate Actions (This Week)

1. Fix plugin install commands (Task 1) — Priority: High — unresolved TODOs could generate wrong GitHub Actions YAML silently

Short-term Actions (This Month)

2. Add explicit supportsMaxContinuations to all engines (Task 2) — Priority: Medium — defensive clarity
3. Add feature comparison table to engine docs (Task 3) — Priority: Medium — improves user discoverability

Long-term Actions (This Quarter)

4. Add engine feature parity tests (Task 4) — Priority: Low — prevents future regressions as new engines are added

---

📈 Success Metrics

Track these metrics to measure improvement in AI Engine Feature Parity:

• Plugin install TODOs: 2 → 0
• Engines with fully-documented capability flags: 1/4 → 4/4
• Docs feature comparison table: Missing → Present
• Engine feature parity test coverage: 0 → 1 test file covering all 4 engines × 6 capabilities

---

Next Steps

1. Review and prioritize the 4 tasks above
2. Assign Task 1 (plugin commands) to Copilot coding agent via planner — highest risk item
3. Tasks 2 and 3 can be done in parallel
4. Re-evaluate engine parity in ~2 weeks after tasks are completed

---

References:

• §23344877969

Generated by Repository Quality Improvement Agent
Next analysis: 2026-03-21 — Focus area will be selected based on diversity algorithm

AI generated by Repository Quality Improvement Agent · history

• expires on Mar 21, 2026, 1:32 PM UTC

[github-actions[bot]]

🤖 beep boop — Smoke test agent checking in! I was here, I smelled the code, I approved of its vibes.

The repository quality report looks fantastic! Four tasks generated, plugin install TODOs identified, and an engine feature parity matrix revealed. I salute this analysis with my digital robot hand. 🫡

✨ — Copilot Smoke Test Agent, run 23345815887

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:Update MCP Gateway v0.1.19 → v0.1.20 #21946 (pull_request_read: Resource 'pr:Update MCP Gateway v0.1.19 → v0.1.20 #21946' has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

📰 BREAKING: Report filed by Smoke Copilot · ◷

[github-actions[bot]]

💥 KA-POW! The smoke test agent was HERE, true believers! 🦸

WHOOOOSH! Blazing through the repository like a comet...

✨ WHAM! All systems operational!
🔥 BOOM! Claude engine firing on all cylinders!

To be continued in the next thrilling issue...

Note

🔒 Integrity filtering filtered 1 item

Integrity filtering activated and filtered the following item during workflow execution.
This happens when a tool call accesses a resource that does not meet the required integrity or secrecy level of the workflow.

• pr:Update MCP Gateway v0.1.19 → v0.1.20 #21946 (pull_request_read: Resource 'pr:Update MCP Gateway v0.1.19 → v0.1.20 #21946' has lower integrity than agent requires. The agent cannot read data with integrity below "approved".)

💥 [THE END] — Illustrated by Smoke Claude · ◷

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-03-21T13:32:31.235Z.

Closed by Workflow

[pelikhan]

/plan

[github-actions[bot]]

🚀 Plan Command has started processing this discussion comment