DurableAgent: Support LanguageModelV3ToolResultOutput for multimodal tool results (images, files)

[Dinesh563]

Summary

Currently, DurableAgent.executeTool always wraps tool results as a text type by JSON stringifying them, even when the tool returns a properly typed LanguageModelV3ToolResultOutput with multimodal content (e.g., images, files).

This prevents tools from returning images or files that the LLM can "see" via vision capabilities.

Current Behavior

In durable-agent.js, the executeTool function unconditionally stringifies tool results:

const toolResult = await execute(parsedInput, { toolCallId, messages, experimental_context });
return {
    type: 'tool-result',
    toolCallId: toolCall.toolCallId,
    toolName: toolCall.toolName,
    output: {
        type: 'text',
        value: JSON.stringify(toolResult) ?? '',  // <-- Always stringified!
    },
};

This means if a tool returns:

return {
  type: 'content',
  value: [
    { type: 'text', text: 'Here is the image' },
    { type: 'file-data', data: base64ImageData, mediaType: 'image/jpeg' },
  ],
};

return { type: 'content', value: [ { type: 'text', text: 'Here is the image' }, { type: 'file-data', data: base64ImageData, mediaType: 'image/jpeg' }, ],};
It gets stringified to a text value, and the LLM never "sees" the image.

Expected Behavior

executeTool should detect if the tool result is already a valid LanguageModelV3ToolResultOutput and pass it through without modification.
Proposed Solution

const toolResult = await execute(parsedInput, { toolCallId, messages, experimental_context });

// Check if tool result is already a LanguageModelV3ToolResultOutput
if (isToolResultOutput(toolResult)) {
  return {
    type: 'tool-result',
    toolCallId: toolCall.toolCallId,
    toolName: toolCall.toolName,
    output: toolResult,  // Pass through as-is
  };
}

// Otherwise, wrap as text (current behavior)
return {
    type: 'tool-result',
    ...
};

function isToolResultOutput(result: unknown): result is LanguageModelV3ToolResultOutput {
  if (typeof result !== 'object' || result === null) return false;
  const r = result as { type?: string };
  return ['text', 'json', 'content', 'error-text', 'error-json', 'execution-denied'].includes(r.type ?? '');
}

Use Case
Building AI agents that can:
Generate images via tools (e.g., image generation APIs)
Fetch and display images to the LLM for vision analysis
Return file attachments (PDFs, etc.) in tool results

[pranaygp]

Just shipped an update here: #1362 including matching the right type

A TODO is to still add tests to ensure multimodal tool results are wired through correctly