Python: FoundryChatClient/OpenAIResponsesClient: hosted_file content roundtrips as input_file in assistant messages, breaking multi-agent workflows after RC5 → 1.0 migration

[moonbox3]

Migrating an example from RC5 to 1.0.1, a user hit a 400 from the Responses API whenever a SequentialBuilder/GroupChatBuilder flow forwarded history from an agent that used file_search. They worked around it with a ChatMiddleware that strips problematic content items, but the underlying bug is in the SDK and is worth fixing properly.

The trigger: a FoundryChatClient agent runs with file_search (vector store), the model emits text with file_citation annotations, those citations land in the assistant Message as HostedFileContent items, and on the next call (e.g., another participant in the GroupChat, or the next agent in a Sequential workflow) the SDK serializes that history back to Responses API input. The serializer maps hosted_file → input_file for any role:

# packages/openai/agent_framework_openai/_chat_client.py, around line 1524
case "hosted_file":
    return {
        "type": "input_file",
        "file_id": content.file_id,
    }

The result is an assistant message whose content array contains an input_file item. input_file is an input-only content type in the Responses API schema, so it's rejected. Reproduces with no network:

msg = Message(role="assistant", contents=[
    Content.from_text("According to the docs, the answer is X."),
    Content.from_hosted_file(file_id="file_abc123"),
])
client._prepare_message_for_openai(msg)
# → [{
#     "type": "message", "role": "assistant",
#     "content": [
#       {"type": "output_text", "text": "...", "annotations": []},
#       {"type": "input_file", "file_id": "file_abc123"}   ← invalid for assistant
#     ]
#   }]

There's a related asymmetry that makes this worse on the streaming path. In non-streaming, file_citation annotations are attached as Annotation objects on the surrounding text:

# _chat_client.py, around line 1696
case "file_citation":
    text_content.annotations.append(Annotation(type="citation", file_id=annotation.file_id, ...))

But on the streaming path (response.output_text.annotation.added, around line 2517), each citation is appended as a separate Content.from_hosted_file(...) item rather than an annotation on the text:

elif ann_type == "file_citation":
    if ann_file_id:
        contents.append(
            Content.from_hosted_file(file_id=str(ann_file_id), ...)
        )

So streaming users always get standalone HostedFileContent items in assistant messages, which then trip the outbound hosted_file → input_file mapping. This is why the bug shows up reliably with serve(...) and with multi-agent forwarding patterns.

There's also a third related corner: even when the non-streaming path does preserve citations as text annotations, the outbound output_text serializer hardcodes \"annotations\": []:

# _chat_client.py, around line 1374
if role == \"assistant\":
    return {
        \"type\": \"output_text\",
        \"text\": content.text,
        \"annotations\": [],   # citations dropped on roundtrip
    }

so file citations are silently lost on roundtrip even when the message would be valid. Lossy, but at least not erroring.

Why this didn't bite in RC5: RC5 used Chat Completions, where citations were just text annotations and there was no input/output content-type schema split. The 1.0 move to Responses API tightened the input schema and the assistant-history roundtrip case was missed.

The user's reported workaround was a ChatMiddleware that filters c.get(\"type\") == \"input_file\" from msg.to_dict() contents — worth noting that this filter is actually a no-op as written (the dicts come out as type: \"hosted_file\"), so either something else in their flow is the actual fix or the filter should target \"hosted_file\". Either way, it shouldn't be required.

Suggested fixes, in increasing order of correctness:

• Minimal: in _prepare_content_for_openai, when role == \"assistant\" and content.type == \"hosted_file\", return {} (drop the unreplayable item). Stops the 400 immediately.
• Better: make the streaming response.output_text.annotation.added handler attach file_citation to the in-progress text content's annotations (matching the non-streaming path), instead of creating standalone HostedFileContent items.
• Complete: also have the outbound output_text serializer preserve Annotation objects (file_citation, url_citation, container_file_citation) on roundtrip, instead of \"annotations\": [].

Affected files:

• python/packages/openai/agent_framework_openai/_chat_client.py (lines ~1374, ~1524, ~2517) — bug lives here, inherited by FoundryChatClient via RawOpenAIChatClient.