Skip to content

Add image support to readFile via toModelOutput#22

Open
tomas-b wants to merge 1 commit intovercel-labs:mainfrom
tomas-b:readfile-image-support
Open

Add image support to readFile via toModelOutput#22
tomas-b wants to merge 1 commit intovercel-labs:mainfrom
tomas-b:readfile-image-support

Conversation

@tomas-b
Copy link
Copy Markdown

@tomas-b tomas-b commented Apr 9, 2026

Add image support to readFile via toModelOutput

Summary

readFile currently treats image files like text, which means models receive
garbled output instead of usable visual input.

This change adds image support to readFile by detecting common image types,
reading them as binary when the sandbox supports readFileBuffer, and returning
them through toModelOutput as image-data content parts.

Changes

  • Add optional readFileBuffer? to the Sandbox interface
  • Wire readFileBuffer through the just-bash and Vercel sandbox adapters
  • Detect supported image extensions in readFile
  • Return image results as base64 plus mediaType
  • Convert image tool results to image-data via toModelOutput
  • Add unit coverage for image output, model output conversion, and fallback behavior
  • Update README and AGENTS docs to match the new behavior

Supported image types

  • png
  • jpg
  • jpeg
  • gif
  • webp

Compatibility

  • Text file behavior is unchanged
  • readFileBuffer is optional, so existing custom sandboxes continue to work
  • If readFileBuffer is not implemented, image files fall back to the existing text path
  • No new dependencies are added

Testing

  • Added unit tests for image reads returning { data, mediaType }
  • Added unit tests for toModelOutput image conversion
  • Added fallback coverage for sandboxes without readFileBuffer
  • pnpm validate passes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant