Add evaluation pipeline for MCP quality

[ashragrawal]

Summary

Adds a365 develop-mcp evaluate — a new subcommand that measures the quality
of an MCP server's tool schemas and produces an interactive HTML report with
prioritized action items.

Poor tool names, descriptions, or parameter schemas cause AI agents to select
the wrong tool or pass incorrect arguments. This command makes that quality
visible and actionable before the server is shipped to agents.

What the command does

1. Discovers tools from the MCP server over Streamable HTTP (JSON-RPC 2.0
   handshake).
2. Generates a checklist combining deterministic checks
   (structural/objective, evaluated inline) and semantic checks (require
   language judgment).
3. Evaluates semantic checks via a locally installed coding agent —
   GitHub Copilot or Claude Code (both default to Haiku).
4. Analyzes results to compute a 0-100 score, a 0-4 maturity level, and a
   ranked set of action items.
5. Writes a machine-readable JSON report and a self-contained HTML report
   that opens in the browser.

Usage

# Fully automatic (requires GitHub Copilot or Claude Code CLI)
a365 develop-mcp evaluate --server-url https://my-mcp-server.com/mcp

# With auth, specific engine, custom output
a365 develop-mcp evaluate \
  --server-url https://my-server/mcp \
  --auth-token $TOKEN \
  --eval-engine claude-code \
  --output-dir ./reports

Bring-your-own-LLM workflow

If no coding agent is installed (or the user wants to use a different LLM),
the command writes the checklist and a self-contained
semantic_eval_prompt.txt to the output directory, then stops. The user scores
each score: null item with their own tool (ChatGPT, Gemini, IDE assistant,
etc.) and re-runs the exact same command — the pipeline detects the existing
scored checklist, skips discovery, and generates the report. No special flag
required for the resume.

Key design decisions

• Thin command, fat service. DevelopMcpCommand only defines args/options and
  delegates to IEvaluationPipelineService, which orchestrates the 5 steps.
  Keeps the command focused and the pipeline unit-testable.
• Resume-by-file. The checklist JSON on disk is the source of truth between
  runs. This is what makes the BYOL workflow round-trip without special flags.
• Strict completion check. The pipeline refuses to generate a report if any
  semantic check is still score: null, because Scorer treats all-null
  categories as 100 — proceeding would produce an inflated, misleading score.
• Required option, not positional arg. --server-url is a required option for
  consistency with other develop-mcp subcommands and the Azure CLI regression
  tests.
• Neutral terminology. The 18 schema-quality problems are called "issues"
  across types, JSON fields, prompts, and the HTML report.
• Same HTTP pattern as the rest of the repo. SchemaDiscoveryService uses
  HttpClientFactory.CreateAuthenticatedClient() — same as GraphApiService,
  ArmApiService, etc. — so user output isn't polluted by the default
  LoggingHttpMessageHandler's 4-lines-per-request noise.

Test plan

• dotnet test tests.proj — 1544 passing, 0 failing
• End-to-end with --eval-engine github-copilot against learn.microsoft.com (3
  tools, 48 semantic checks)
• End-to-end with --eval-engine claude-code
• End-to-end with --eval-engine none on fresh run (stops with guidance)
• End-to-end with --eval-engine auto and no agent on PATH (stops with install

• BYOL guidance)

• BYOL resume: score the checklist manually (simulated via
  scripts/simulate_llm_scoring.py), re-run same command, confirm report
  generates without re-discovery
• Engine fallthrough: Copilot timeout falls through to Claude Code
• Clean logs (no Start/Sending/Received/End HTTP noise)
• Help text mentions coding-agent dependency + BYOL alternative

[github-actions]

⚠️ Deprecation Warning: The deny-licenses option is deprecated for possible removal in the next major release. For more information, see issue 997.

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds the new a365 evaluate command and supporting evaluation pipeline to discover MCP tools, run deterministic + semantic checks (via GitHub Copilot / Claude Code), score + compute maturity, and generate JSON/HTML reports.

Changes:

• Introduces evaluation pipeline services (schema discovery, checklist generation/evaluation, scoring, analysis, reporting) and wires them into the CLI.
• Adds models for checklists/results, smell taxonomy + maturity scoring, and prompt templates for coding-agent semantic evaluation.
• Adds extensive xUnit coverage for the new evaluation components and CLI command.

Reviewed changes

Copilot reviewed 46 out of 46 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
CHANGELOG.md	Documents the new a365 evaluate command.
src/Microsoft.Agents.A365.DevTools.Cli/Commands/EvaluateCommand.cs	Adds the evaluate CLI command orchestrating the 5-step pipeline.
src/Microsoft.Agents.A365.DevTools.Cli/Constants/ErrorCodes.cs	Adds evaluation-specific error codes.
src/Microsoft.Agents.A365.DevTools.Cli/Exceptions/EvaluationException.cs	Introduces a dedicated exception type for evaluation failures.
src/Microsoft.Agents.A365.DevTools.Cli/Microsoft.Agents.A365.DevTools.Cli.csproj	Adds HTTP client factory package + embeds HTML report template.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/ActionItem.cs	Adds report model for prioritized remediations.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/ChecklistItem.cs	Adds model for deterministic/semantic checklist items.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/EvalReportData.cs	Adds HTML-template data model (result + impact map + ladder).
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/EvaluateEnums.cs	Adds enums for categories, priorities, engines, etc.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/EvaluationChecklist.cs	Adds checklist root + metadata model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/MaturityLevel.cs	Adds maturity level model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/SchemaEvalResult.cs	Adds top-level evaluation result model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/SmellDefinition.cs	Adds smell taxonomy definition model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/ToolChecklist.cs	Adds per-tool checklist + grouped checks model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/ToolEvalResult.cs	Adds per-tool evaluation result model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/ToolSchema.cs	Adds discovered tool schema model.
src/Microsoft.Agents.A365.DevTools.Cli/Models/Evaluate/ToolsetEvalResult.cs	Adds toolset-level evaluation result model.
src/Microsoft.Agents.A365.DevTools.Cli/Program.cs	Registers evaluate command + DI services (HttpClient + pipeline services).
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ActionItemGenerator.cs	Generates action items from failed checks with score impact + smell mapping.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ChecklistEvaluator.cs	Runs extract-evaluate-merge with coding agents and merges results back.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/CodingAgentRunner.cs	Detects/runs Copilot/Claude CLIs with timeouts and cleanup.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/EvaluationAnalyzer.cs	Computes scores, maturity, smell summary, and action items aggregation.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/IChecklistEvaluator.cs	Interface + result type for semantic evaluation step.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/IChecklistGenerator.cs	Interface for checklist generation step.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/IEvaluationAnalyzer.cs	Interface for analysis/scoring step.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/IReportGenerator.cs	Interface for report generation step.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ISchemaDiscoveryService.cs	Interface for MCP tool discovery step.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/MaturityCalculator.cs	Implements maturity level calculation + ladder rendering data.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ReportGenerator.cs	Writes JSON/HTML report and optionally opens browser.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SchemaDiscoveryService.cs	Implements MCP initialize + tools/list discovery over JSON-RPC + SSE handling.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/Scorer.cs	Implements category/tool/overall scoring + averages.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SemanticCheckDefinitions.cs	Defines semantic check metadata for tools/params/toolset.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SemanticCheckPrompts.cs	Builds prompts/commands for coding-agent semantic evaluation.
src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SmellTaxonomy.cs	Adds 18-smell taxonomy and impact map for HTML report.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Commands/EvaluateCommandTests.cs	Tests CLI structure + engine parsing + server-name derivation.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/ActionItemGeneratorTests.cs	Tests action item generation + score impact + sorting.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/DeterministicChecksTests.cs	Tests deterministic checks behavior (naming/description/schema/toolset).
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/EvaluationAnalyzerTests.cs	Tests analyzer aggregation, scoring, maturity, smells, priorities.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/MaturityCalculatorTests.cs	Tests maturity thresholds/caps + ladder output.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/ReportGeneratorTests.cs	Tests JSON/HTML report generation + sanitization + arg validation.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/ScorerTests.cs	Tests scoring rules and weight math.
src/Tests/Microsoft.Agents.A365.DevTools.Cli.Tests/Services/Evaluate/SemanticCheckDefinitionsTests.cs	Tests semantic check definitions are consistent and stable.

Comments suppressed due to low confidence (5)

src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ReportGenerator.cs:1

• htmlPath may contain spaces/shell-special characters, and passing it via the (fileName, arguments) constructor can break argument parsing (especially on Linux/macOS). Prefer ProcessStartInfo.ArgumentList (single arg) or explicitly quote/escape the path to ensure the report opens reliably.
  src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ReportGenerator.cs:1
• The HTML template receives raw JSON substituted into the document. If any server-supplied fields (tool names/descriptions, reasons, etc.) contain sequences like </script> or other HTML-breaking content, this can break the page or enable script injection when the report is opened. Consider embedding the JSON in a safer form (e.g., HTML-escaped JSON, base64-encoded payload decoded at runtime, or a <script type="application/json"> block with escaping of <, >, &, and </script>).
  src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SemanticCheckPrompts.cs:1
• The JSON structure example suggests the parameters object is keyed by "param_name", but in the actual model it’s a dictionary keyed by the actual parameter name (e.g., "userId": { "param_name": [...], ... }). This mismatch can mislead the coding agent and reduce evaluation success; update the example to reflect the real shape.
  src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SchemaDiscoveryService.cs:1
• SerializerOptions is declared but not used anywhere in this file. Remove it to avoid dead code, or use it consistently for any deserialization that needs case-insensitive behavior.
  src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SchemaDiscoveryService.cs:1
• For SSE responses, ReadAsStringAsync reads the entire response body before parsing, which can be problematic if the server keeps the event stream open (the call may not complete promptly). Consider switching to streaming processing (e.g., SendAsync(..., ResponseHeadersRead) and reading the content stream line-by-line until a JSON data: message is found) to avoid hangs and reduce memory usage.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 48 changed files in this pull request and generated 1 comment.

[ashragrawal]

@microsoft-github-policy-service agree company="Microsoft"

[Copilot]

Pull request overview

Copilot reviewed 48 out of 48 changed files in this pull request and generated 9 comments.

[Copilot]

Pull request overview

Copilot reviewed 48 out of 48 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/ReportGenerator.cs:1

• Injecting raw JSON into an HTML template via string replacement can be unsafe if the template places it inside a <script> tag: tool descriptions could contain </script> and prematurely terminate the script, enabling HTML/script injection in the generated report. Escape JSON for safe embedding (e.g., replace </ with <\\/), or embed the JSON as HTML-encoded text within a <script type=\"application/json\"> element and parse it at runtime.
  src/Microsoft.Agents.A365.DevTools.Cli/Services/Evaluate/SchemaDiscoveryService.cs:1
• SerializerOptions is declared but never used anywhere in this file. Remove it to reduce noise, or apply it consistently where deserialization occurs (if the intent is to deserialize into typed models rather than using JsonDocument/manual parsing).

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 49 out of 49 changed files in this pull request and generated 4 comments.