Skip to content

feat: unified response_format / structured JSON output support #8

@stackbilt-admin

Description

@stackbilt-admin

Summary

Add a unified response_format option to LLMRequest that enforces structured JSON output across all providers.

Motivation

From the frontend team (Oracle project): they need structured JSON output (a files[] array) and currently have no way to enforce this through the package. Each upstream provider handles this differently:

  • OpenAI: response_format: { type: "json_object" }
  • Anthropic: prefilled assistant responses / tool use
  • Cloudflare Workers AI: no native support
  • Groq: response_format: { type: "json_object" }
  • Cerebras: varies by model

Proposed API

const response = await llm.chat({
  provider: 'openai',
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'List the files as JSON' }],
  response_format: { type: 'json_object' },
});

Implementation approach

Each provider adapter translates the unified response_format to its native mechanism:

Provider Native mechanism
OpenAI response_format: { type: "json_object" } (pass-through)
Groq response_format: { type: "json_object" } (pass-through)
Anthropic Prefilled assistant turn { + system prompt instruction
Cerebras System prompt instruction + validation
Cloudflare System prompt instruction + validation

For providers without native support, inject a system prompt suffix and optionally validate the response parses as JSON.

Acceptance criteria

  • response_format field added to LLMRequest type
  • OpenAI and Groq pass through natively
  • Anthropic uses prefilled assistant response approach
  • Cloudflare and Cerebras use system prompt injection fallback
  • Optional JSON parse validation on response
  • Unit tests for each provider's translation
  • Documented in README

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions