refactor(3/3): switch dynamo-protocols to upstream async-openai types

[ishandhanani]

Summary

Replace the entire forked async-openai codebase in dynamo-protocols with upstream async-openai v0.34 re-exports + locally-defined inference-serving extensions.

19,108 deletions and 1,089 additions in the current PR diff (net -18,019 lines). dynamo-protocols now contains a minimal crate root, slimmed error types, and protocol type definitions -- no HTTP client, no API handlers, and no forked copies of upstream types.

Approach

The fork originally vendored the entire async-openai crate (HTTP client, API handlers, type definitions) and added NVIDIA-specific fields directly onto base OpenAI types. This PR reverses that by:

1. Adding upstream async-openai as a types-only dependency (default-features = false with chat-completion-types, response-types, completion-types, embedding-types, image-types)

2. Re-exporting upstream types where identical -- ~50 chat types (CompletionUsage, FunctionCall, Role, Prompt, ChatCompletionMessageToolCall, etc.), all embedding types, all response types, all image types. Consumers still import from dynamo_protocols::types::* -- the re-exports are transparent.

3. Keeping local definitions for types we extend -- when a type has inference-serving fields that upstream doesn't have, we define it locally and document the delta. For example:

   /// Chat completion response message with multimodal content and reasoning.
   ///
   /// Extends upstream `ChatCompletionResponseMessage` with:
   /// - `content`: `Option<ChatCompletionMessageContent>` (multimodal) instead of `Option<String>`
   /// - `reasoning_content`: model reasoning output (DeepSeek-R1, QwQ)
   pub struct ChatCompletionResponseMessage {
       pub content: Option<ChatCompletionMessageContent>,  // our extension
       pub reasoning_content: Option<String>,               // our extension
       // ... standard fields (refusal, tool_calls, role, etc.)
   }

   Types form a containment tree -- if a child type is extended, all parent containers must also be locally defined. So ChatChoice is local because it contains our extended ChatCompletionResponseMessage, CreateChatCompletionResponse is local because it contains ChatChoice, etc.

4. Adapting consumers for upstream API changes that differ from the fork:

   • ChatCompletionMessageToolCall no longer has a type field (always "function" in the spec)
   • ChatCompletionToolType renamed to FunctionType in upstream
   • Stop renamed to StopConfiguration in upstream
   • ImagesResponse gained new fields (background, output_format, quality, size, usage)
   • ImageModel gained new variants (GptImage1, GptImage1dot5, GptImage1Mini)

5. Deleting everything else -- 31 API handler modules, HTTP client, config, 24 forked type modules that were either identical to upstream or unused. Removed client-only dependencies (reqwest, backoff, secrecy, async-openai-macros, etc.).

Types retained locally (inference-serving extensions)

Type	What we added	Why
ChatCompletionResponseMessage	content: Option<ChatCompletionMessageContent> (enum: Text or multimodal Parts), reasoning_content: Option<String>	Backends return multimodal content (images, video, audio) and reasoning tokens
ChatCompletionStreamResponseDelta	Same content + reasoning_content fields	Streaming variant of the above
ChatChoice / ChatChoiceStream	stop_reason: Option<StopReason>	Backends report which stop condition triggered (matched string or token ID)
CreateChatCompletionRequest	mm_processor_kwargs: Option<Value>	vLLM multimodal processor configuration
ChatCompletionStreamOptions	continuous_usage_stats: bool	Per-chunk usage stats (not just final chunk)
ChatCompletionRequestAssistantMessage	reasoning_content: Option<ReasoningContent>	Interleaved reasoning segments for KV cache-correct context reconstruction
ChatCompletionRequestUserMessageContentPart	VideoUrl(...), AudioUrl(...) variants	Multimodal input beyond images
ImageUrl	url: url::Url (not String), uuid: Option<Uuid>	Strongly-typed URLs + asset tracking through pipeline
CreateCompletionRequest	prompt_embeds: Option<String>, strict echo validation	Pre-computed embeddings, reject non-bool echo values
ReasoningContent	Fully custom enum (Text or Segments)	No upstream equivalent
StopReason	Fully custom enum (String or Int)	No upstream equivalent
ChatCompletionMessageContent	Fully custom enum (Text or Parts)	No upstream equivalent
All anthropic/ types	Fully custom (~870 lines)	Anthropic Messages API

What remains in dynamo-protocols

lib/protocols/src/
├── lib.rs              # Crate root
├── error.rs            # Slimmed error types
└── types/
    ├── mod.rs          # Re-exports upstream + local extensions
    ├── anthropic.rs    # Full Anthropic Messages API
    ├── chat.rs         # Upstream chat re-exports + local multimodal/reasoning extensions
    ├── completion.rs   # Upstream completion re-export + local CreateCompletionRequest
    ├── impls.rs        # From/Default impls for local types
    └── responses/
        └── mod.rs      # Upstream response re-exports + aliases

Stack: 3/3 -- depends on #7565, see RFC #7563

Test plan

• cargo test -p dynamo-protocols -- pass
• cargo test -p dynamo-parsers -- 354 pass
• cargo test -p dynamo-llm --lib -- 759 pass
• cargo check -- clean across all crates

---

Summary by CodeRabbit

• New Features

  • Added multimodal support (video and audio inputs) for chat completions.
  • Extended response handling with reasoning content and streaming options.
  • Improved tool calling with updated type definitions.

• Breaking Changes

  • Removed HTTP client public exports; simplified HTTP service interface.
  • Removed support for assistants, threads, runs, and file operations.
  • Removed realtime and BYOT feature support.
  • Updated protocol types to align with upstream async-openai library.

• Refactor

  • Consolidated protocol types and dependencies.

[grahamking]

Great work.

[MatejKosec]

Review: Type Migration Risk Assessment

Thorough analysis of the type conversion changes. Overall the refactor is clean and well-structured — the -18K line reduction is a big maintenance win. A few things that might be overlooked:

1. stream field serializes null (Medium Risk)

CreateChatCompletionRequest.stream lost its #[serde(skip_serializing_if = "Option::is_none")] attribute and now only has #[serde(default)]. This means stream: None will serialize as "stream": null instead of being omitted. Some backends reject explicit null on this field. Fix: add skip_serializing_if back alongside default.

2. v1/responses has near-zero test coverage (High Risk)

The PR rewrites the type conversion layer for Responses (chat_completion_to_response, ResponseStreamConverter, InputContent handling), but there are almost no E2E tests exercising the /v1/responses endpoint. The only usage is in test_sglang.py — no coverage for vLLM, TRT-LLM, streaming responses, structured output through responses, or error paths. The 36/36 E2E pass is mostly chat completions.

Specifically untested after this PR:

• convert_content() in responses/mod.rs (InputText/InputImage/InputFile conversion)
• chat_completion_to_response() (the core translation layer)
• ResponseStreamConverter (streaming delta → response event)
• Error propagation through the responses path

3. InputVideo/InputAudio handling silently removed

InputContent conversion drops InputVideo, InputAudio, OutputText, Refusal variants. Correct today (upstream v0.34 doesn't have them), but when upstream adds them, the code will silently skip them. A TODO comment would help.

4. store behavioral change

store: true is now accepted (previously rejected) in Responses API requests, enabling audit gating. This is intentional but worth noting in the PR description since it's a behavioral change, not just a refactor.

5. convert_image_detail_str does JSON round-trip per image

Minor perf concern: serializes to serde_json::Value then matches string for every image. A direct enum match or From impl would avoid the allocation.

Recommendation: Items 1 and 2 should be addressed before merge. The rest are non-blocking follow-ups.

[MatejKosec]

Follow-up: ImageModel Wildcard Bug (New Finding)

The wildcard match at lib/llm/src/http/service/openai.rs:1944:

_ => format!("{:?}", m).to_lowercase()

produces wrong model names for upstream ImageModel variants because Debug doesn't respect #[serde(rename)]:

Variant	Debug output (lowercased)	Correct serde name
GptImage1	gptimage1	gpt-image-1
GptImage1dot5	gptimage1dot5	gpt-image-1.5
GptImage1Mini	gptimage1mini	gpt-image-1-mini

This would cause state.manager().get_images_engine(&model) to fail to find the engine for GPT-Image models. Latent bug — won't fire until someone uses these models through Dynamo, but worth fixing.

Fix: Use serde_json::to_value(&m) to get the correctly-renamed string, or match explicitly on known variants.

[MatejKosec]

Follow-up: v1/responses Wire-Format Changes & Test Gap

Deep analysis of the v1/responses path reveals visible wire-format changes with no regression test coverage.

Fields removed from Response JSON

Field	Old Value	After PR
frequency_penalty	0.0	absent
presence_penalty	0.0	absent
store	false/true	absent
max_tool_calls	null	absent

Fields that flip from null to omitted

error, incomplete_details, prompt, prompt_cache_key, safety_identifier, conversation, billing — all previously serialized as "field": null, now omitted entirely (upstream uses skip_serializing_if).

Concrete client breakage scenario

response = requests.post("/v1/responses", json={"input": "hello"}).json()
fp = response["frequency_penalty"]  # Was 0.0, now KeyError
store = response.get("store", True)  # Was False, now defaults to True (flipped!)

The gap

No test anywhere serializes a Response to JSON and validates the shape. The Rust unit tests check struct field values but never serialize. The Python E2E tests check object, id, status, output[0].type but none of the fields that changed.

Recommendation

Add a ~30-line wire-format snapshot test:

let json = serde_json::to_value(&response).unwrap();
assert!(json.get("frequency_penalty").is_none());
assert!(json.get("store").is_none());
assert!(json.get("presence_penalty").is_none());
assert_eq!(json["metadata"], serde_json::json!({}));
assert!(json["output"][0].get("id").is_some());
assert!(json["output"][0].get("status").is_some());

Also consider running the OpenResponses compliance suite (bun run test:compliance) which the test file itself references.