Add codeExecution support to google providers and more safely handle provider and user tools#6354
Add codeExecution support to google providers and more safely handle provider and user tools#6354Und3rf10w wants to merge 5 commits intovercel:mainfrom
codeExecution support to google providers and more safely handle provider and user tools#6354Conversation
…n combination with user defined tools
|
We have switched development to the v5 branch. Can you make the change against v5? |
… url doesn't support it
|
@Und3rf10w i attempted a backport but ran into merge conflicts. if you could reimplement against v5 that would be excellent |
|
Looking at provider-defined-tool, would it make more sense to define the google provided tools as these? If so, I just don't have a good understanding of how to convert the response to a tool result in the middle of the generation or stream. |
|
code execution is more complex, i wanted to have a cleaner way in v5 but haven't yet gotten around to do it. you need to stream back tool results from the provider afaik |
Okay thanks. I'll opt for handling it as file and text parts for now. I'll get you that PR soonish |
|
Closed in lieu of #6365 |
## Background This change was necessary to add support for the Code Execution feature available in Google's Gemini models. This allows the model to generate and execute Python code to perform calculations and solve problems, enhancing its capabilities. ## Summary - Added `codeExecution` tool: - A new tool, `google.tools.codeExecution()`, is now available. This enables models that support it to run Python code. ## Verification I manually verified the changes by setting up the new examples for Google Vertex AI. I ran both the `generate-text/google-vertex-code-execution.ts` and `stream-text/google-vertex-code-execution.ts` examples against the `gemini-2.5-pro model`. In both cases, the model successfully used the code execution tool to first calculate the 20th Fibonacci number and then find the nearest palindrome. The console output correctly displayed the tool calls and their corresponding results, followed by the final generated text, confirming that the end-to-end flow is working as expected. <img width="924" height="1013" alt="image" src="https://github.com/user-attachments/assets/d70eea58-1fce-4bcf-873e-3b7f5981ac6c" /> ## Related Issues Closes #3205 Closes #6354 Closes #6365
3 participants
Background
The google and google-vertex providers would silently drop user defined tools when used in combination with
useSearchGrounding. UsinguseSearchGroundingin combination with user defined tools is not supported. This change throws an error if that happens.In addition, this change also adds support for the Code Execution feature in both the Google Generative AI and Vertex providers.
Summary
useSearchGroundinganduseCodeExecutionon the Google Generative AI and thus Vertex providerscodeExecutiontool, and extract those as text parts.codeExecutionfeature.Verification
I added examples for both the google and google-vertex providers and have tested the both of them.
I originally added tests for the code execution logic, but removed them in b75484b because apparently the test url doesn't support the 2.5 models.
Tasks
pnpm changesetin the project root)pnpm prettier-fixin the project root)Future Work
I would more likely prefer that the
executableCodepart be treated as a tool-call, and thecodeExecutionResultPartSchemabe treated as the tool-result of that call, but i could not get that to function properly.This is particularly difficult in a stream. The shape of both are the same regardless of whether it's being returned in a stream or in a generation.
Unit tests likely need to be added.
Related Issues
Fixes #6302
Fixes #6219