feat(sdk): improve AI tool definitions for LLM accuracy (25% → 95% pass rate)

[tupizz]

TL;DR

LLMs using our SDK tools failed 75% of the time because parameter schemas had no descriptions. This PR adds descriptions to 90% of parameters and improves the codegen pipeline. Execution test pass rate went from 25% → 95%.

---

Problem

Our 9 grouped SDK tools expose ~115 parameters to LLMs via JSON Schema. Most had no description field — models saw names and types but no guidance on format, valid values, or when params are required.

Common failures:

• Model passed at: {kind: "end"} — schema requires "documentEnd" but nothing said so
• Model used value instead of text for replacement content
• Model omitted changeMode (required) because nothing marked it as mandatory
• Model passed target: {ref: "..."} instead of using the ref param directly
• Model tried {type: "heading"} instead of {type: "node", nodeType: "heading"}
• List creation burned 5 iterations guessing kind, mode, at, target requirements

Solution

Three layers of improvements:

1. Descriptions on schemas (schemas.ts, operation-params.ts)

Added descriptions with format examples to all major operations:

// Before: model guesses wrong format
at: { oneOf: [...] }

// After: model knows exactly what to pass  
at: { oneOf: [...], description: "Position: {kind:'documentEnd'} to append, {kind:'before'|'after', target:{kind:'block', nodeType:'...', nodeId:'...'}} for relative placement." }

2. Smarter codegen (generate-intent-tools.mjs)

• Auto-annotates conditional requirements: adds "Required for action 'create'" when a param is required by some actions but not all
• Checks contract-level required arrays: previously only checked CLI params, missing requirements from if/then/else schemas
• Removes empty {} oneOf branches: bold: oneOf:[{type:"boolean"}, {}] → bold: {type:"boolean"} (42 inline properties simplified)
• Deduplicates same-type oneOf branches: collapses ref: oneOf:[{type:"string",...},{type:"string",...}] into single {type:"string"}
• Fallback descriptions: auto-adds descriptions to target, ref, content, inline when the schema doesn't provide one

3. System prompt guidance (system-prompt.md)

• "Always take action" — stop asking clarifying questions
• "Placing content near specific text" — search by text, not by node type
• ref vs target clarification — use ref for inline formatting
• List creation workflow — two modes explained with examples
• select.type must be "text" or "node" — prevents {type: "heading"} error

Results

Execution tests (same 20 tests, same model):

Stage	Pass rate	Key fix
Baseline	25%	No descriptions
+create at desc	60%	Model stops guessing {kind:"end"}
+changeMode, select, target	85%	Required params and format examples
+mutations changeMode: "Required"	95%	Last major failure fixed

Description coverage: 60% → 90% of parameters

Schema token cost: ~11,175 → ~11,001 tokens (empty branch removal)

The 1 remaining failure is a table creation test where the tool surface doesn't support the operation yet.

Files

Area	Files	What
Schema descriptions	schemas.ts	Descriptions on 30+ properties across all tools
CLI params	operation-params.ts, types.ts, export-sdk-contract.ts	Descriptions on envelope + flat params, removed dead agentRequired
Codegen	generate-intent-tools.mjs	Auto-annotation, dedup, empty branch removal, fallbacks
System prompt	system-prompt.md	Targeting guide, list workflow, action-first instruction
Eval tests	tool-quality.yaml, execution.yaml, checks.cjs	4-group structure, trace assertions, partial credit scoring
Reference docs	apps/docs/**/*.mdx	Regenerated with new descriptions

Test plan

• pnpm run generate:all passes
• tools.openai.json has descriptions on 104/115 params
• Execution tests: 19/20 (95%)
• Tool quality tests: 172/174 (99%)
• Reference docs regenerated
• Cross-provider validation (Anthropic, Google)

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7fd1ad12eb

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[andrii-harbour]

well done!

minor comments here

[harbournick]

LGTM - really nice!

[superdoc-bot]

🎉 This PR is included in superdoc-cli v0.3.0-next.38

The release is available on GitHub release

[superdoc-bot]

🎉 This PR is included in superdoc-cli v0.3.0

The release is available on GitHub release