Add cache-aware routing and provider prompt-cache support

[stackbilt-admin]

Goal

Add first-class cache-aware routing to the inference router so we can take advantage of provider-side prompt/prefix caching while keeping the public API provider-agnostic.

Why

Providers increasingly expose prompt/prefix caching that can materially reduce latency and/or cost for long repeated prefixes, agent loops, RAG, coding workflows, stable tool definitions, and stable system prompts.

The router should understand cache hints, normalize cache usage, and eventually account for cached-token pricing in routing decisions.

Providers to cover

• OpenAI: automatic prompt caching, prompt_cache_key, prompt_cache_retention, cached token usage.
• Anthropic: explicit cache_control breakpoints, 5m/1h ephemeral TTLs, cache read/write token usage.
• Groq: automatic prompt caching on supported models, cached token usage.
• Cerebras: automatic prompt caching on supported models, 128-token blocks, guaranteed 5m TTL with up to 1h max, usage.prompt_tokens_details.cached_tokens.
• Cloudflare Workers AI: prefix caching on supported models, x-session-affinity, cached token usage.
• Cloudflare AI Gateway: existing cf-aig-cache-key / cf-aig-cache-ttl support should be reconciled with the new abstraction.

Proposed shape

Add request-level cache hints, for example:

cache?: {
  strategy?: 'off' | 'provider-prefix' | 'response' | 'both';
  key?: string;
  ttl?: '5m' | '1h' | '24h' | number;
  sessionId?: string;
  cacheablePrefix?: 'auto' | 'system' | 'tools' | 'messages';
}

Normalize response usage:

usage: {
  inputTokens: number;
  outputTokens: number;
  totalTokens: number;
  cost: number;
  cachedInputTokens?: number;
  cacheReadInputTokens?: number;
  cacheCreationInputTokens?: number;
  cacheWriteInputTokens?: number;
}

Implementation notes

• Add cache capability metadata to the model catalog.
• Translate generic cache hints inside each provider adapter.
• Parse provider-specific cached token fields into normalized usage.
• Update cost tracking to account for cache read/write pricing where providers discount cached tokens.
• Keep response/output caching separate and disabled by default.
• Add tests per provider for request translation and usage parsing.

Non-goals

• Do not promise identical cache semantics across providers.
• Do not cache arbitrary chat responses by default.
• Do not persist sensitive prompt content unless an explicit response-cache store is configured.

Acceptance criteria

• LLMRequest supports provider-agnostic cache hints without breaking existing callers.
• TokenUsage exposes normalized cached-token fields.
• OpenAI, Anthropic, Groq, Cerebras, and Cloudflare adapters either translate cache hints or explicitly no-op with documented behavior.
• Cost tracking and ledger paths preserve existing behavior and include cached-token metadata where available.
• Unit tests cover cache request translation and usage parsing for each provider.

[stackbilt-admin]

Auditor note: the issue has the right shape, but I would recommend sequencing and scoping it carefully because this will affect routing, usage accounting, and public API compatibility.

Suggested implementation order:

1. Extend types first with optional cache hints and optional cached-token fields on TokenUsage, with no behavior change.
2. Parse provider usage fields next, even before request translation. That gives cost/ledger code real data to preserve and avoids losing provider metadata.
3. Add request translation one provider at a time. Anthropic explicit cache breakpoints and Cloudflare Gateway response cache headers should not be treated as identical semantics.
4. Only after usage parsing exists, update CostTracker/CreditLedger to account for cached-token discounts.

Two risk areas to capture in acceptance criteria:

• Cache keys/session IDs may contain tenant or prompt-derived identifiers; docs should say whether callers must hash or sanitize them before passing them.
• gatewayMetadata.cacheKey/cacheTtl already exists for Cloudflare AI Gateway response caching. The new abstraction should explicitly distinguish provider prompt/prefix caching from Gateway response caching so consumers do not accidentally persist full responses when they intended only prefix-cache hints.

This is worth doing, but I would avoid one large all-provider PR. The blast radius is high enough that provider-by-provider slices with shared type/tests first will be safer for OSS review.

[stackbilt-admin]

Implemented in three slices:

1. Types — CacheHints interface, cache field on LLMRequest, cachedInputTokens / cacheReadInputTokens / cacheCreationInputTokens on TokenUsage, supportsPromptCache on ModelCapabilities.

2. Usage parsing — All providers extract cached token counts from provider-specific fields: Anthropic (cache_read_input_tokens, cache_creation_input_tokens), Groq/Cerebras/OpenAI (prompt_tokens_details.cached_tokens).

3. Request translation — Anthropic: when cache.strategy is 'provider-prefix' or 'both', the system prompt is wrapped as a content-block array with cache_control: { type: 'ephemeral' }, and the last tool definition (if cacheablePrefix is 'auto' or 'tools') gets a breakpoint too. OpenAI/Groq/Cerebras use automatic caching with no request-side translation needed. Cloudflare uses platform-level prefix caching via the Workers AI binding.

Tests in cache-hints.test.ts cover all translation paths and the cached token extraction.