Summary
Add a unified response_format option to LLMRequest that enforces structured JSON output across all providers.
Motivation
From the frontend team (Oracle project): they need structured JSON output (a files[] array) and currently have no way to enforce this through the package. Each upstream provider handles this differently:
- OpenAI:
response_format: { type: "json_object" }
- Anthropic: prefilled assistant responses / tool use
- Cloudflare Workers AI: no native support
- Groq:
response_format: { type: "json_object" }
- Cerebras: varies by model
Proposed API
const response = await llm.chat({
provider: 'openai',
model: 'gpt-4o-mini',
messages: [{ role: 'user', content: 'List the files as JSON' }],
response_format: { type: 'json_object' },
});
Implementation approach
Each provider adapter translates the unified response_format to its native mechanism:
| Provider |
Native mechanism |
| OpenAI |
response_format: { type: "json_object" } (pass-through) |
| Groq |
response_format: { type: "json_object" } (pass-through) |
| Anthropic |
Prefilled assistant turn { + system prompt instruction |
| Cerebras |
System prompt instruction + validation |
| Cloudflare |
System prompt instruction + validation |
For providers without native support, inject a system prompt suffix and optionally validate the response parses as JSON.
Acceptance criteria
Summary
Add a unified
response_formatoption toLLMRequestthat enforces structured JSON output across all providers.Motivation
From the frontend team (Oracle project): they need structured JSON output (a
files[]array) and currently have no way to enforce this through the package. Each upstream provider handles this differently:response_format: { type: "json_object" }response_format: { type: "json_object" }Proposed API
Implementation approach
Each provider adapter translates the unified
response_formatto its native mechanism:response_format: { type: "json_object" }(pass-through)response_format: { type: "json_object" }(pass-through){+ system prompt instructionFor providers without native support, inject a system prompt suffix and optionally validate the response parses as JSON.
Acceptance criteria
response_formatfield added toLLMRequesttype