feat(providers/google): Add reasoning token output support#6261
feat(providers/google): Add reasoning token output support#6261lgrammel merged 5 commits intovercel:mainfrom
Conversation
chore(docs/providers): Update google provider documentation to add logic around reasoning token support
|
Added example in 80eb2c3 to demonstrate: 
|
| ### Reasoning (Thinking Tokens) | ||
|
|
||
| Certain Google Gemini models support emitting "thinking" tokens, which represent the model's reasoning process before generating the final response. The AI SDK exposes these as reasoning information. | ||
|
|
||
| To enable thinking tokens, set `includeThoughts: true` in the `thinkingConfig` provider option: | ||
|
|
||
| ```ts | ||
| import { google } from '@ai-sdk/google'; | ||
| import { GoogleGenerativeAIProviderOptions } from '@ai-sdk/google'; | ||
| import { generateText, streamText } from 'ai'; | ||
|
|
||
| // For generateText: | ||
| const { text, reasoning, reasoningDetails } = await generateText({ | ||
| model: google('gemini-2.5-flash-preview-04-17'), // Or other supported model | ||
| providerOptions: { | ||
| google: { | ||
| thinkingConfig: { | ||
| includeThoughts: true, | ||
| // thinkingBudget: 2048, // Optional | ||
| }, | ||
| } satisfies GoogleGenerativeAIProviderOptions, | ||
| }, | ||
| prompt: 'Explain quantum computing in simple terms.', | ||
| }); | ||
|
|
||
| console.log('Reasoning:', reasoning); | ||
| console.log('Reasoning Details:', reasoningDetails); | ||
| console.log('Final Text:', text); | ||
|
|
||
| // For streamText: | ||
| const result = streamText({ | ||
| model: google('gemini-2.5-flash-preview-04-17'), // Or other supported model | ||
| providerOptions: { | ||
| google: { | ||
| thinkingConfig: { | ||
| includeThoughts: true, | ||
| // thinkingBudget: 2048, // Optional | ||
| }, | ||
| } satisfies GoogleGenerativeAIProviderOptions, | ||
| }, | ||
| prompt: 'Explain quantum computing in simple terms.', | ||
| }); | ||
|
|
||
| for await (const part of result.fullStream) { | ||
| if (part.type === 'reasoning') { | ||
| process.stdout.write(`THOUGHT: ${part.textDelta}\n`); | ||
| } else if (part.type === 'text-delta') { | ||
| process.stdout.write(part.textDelta); | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| When `includeThoughts` is true, parts of the API response marked with `thought: true` will be processed as reasoning. | ||
|
|
||
| - In `generateText`, these contribute to the `reasoning` (string) and `reasoningDetails` (array) fields. | ||
| - In `streamText`, these are emitted as `reasoning` stream parts. | ||
|
|
||
| <Note> | ||
| Refer to the [Google Generative AI | ||
| documentation](https://ai.google.dev/gemini-api/docs/thinking) for a list of | ||
| models that support thinking tokens and for more details on `thinkingBudget`. | ||
| </Note> | ||
|
|
There was a problem hiding this comment.
@Und3rf10w have you tested this with the gemini api? couldn't find it in their docs
There was a problem hiding this comment.
@lgrammel, I have not directly, but I'm making the assumption it works for the gemini API based off of this Google Provided Notebook: https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_thinking.ipynb
I HAVE successfully tested this with the Vertex API.
There was a problem hiding this comment.
I tried a variation of https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Get_started_thinking.ipynb with a valid API key, and it turns out the GEMINI api doesn't yet OFFICIALLY support (read: documented) includeThoughts.
Looking at the proto definitions for the python client, include_thoughts isn't yet supported: (documented) https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform_v1.types.GenerationConfig.ThinkingConfig
Compare this to the vertex api documentation, where includeThoughts IS supported: https://cloud.google.com/vertex-ai/generative-ai/docs/reference/rest/v1/GenerationConfig#ThinkingConfig
However, trying this in the notebook, we can see when we set a thinkingBudget and includeThoughts, the request IS valid, but it doesn't return thought candidates, despite using thinking tokens:
So I suppose, in the Google Generative API, it WON'T provide the thoughts, but the request will still work likely it will eventually be supported. We should probably just remove the 15-google-generative-ai.mdx file edits for now?
Here's more photos playing around with the ThinkingConfig:
TL;DR: While the parameter include_thoughts works on the Google Generative AI platform, it doesn't currently return the thought tokens from the response. It does work as expected in Vertex AI. Likely way forward is to remove the edits to 15-google-generative-ai.mdx from this PR.
Maybe also:
- Update changeset to be
@ai-sdk/vertexinstead of@ai-sdk/google - Throw a warning when
includeThoughtsis specified with an@ai-sdk/googleprovider for now? To be removed if/when that's specified?
There was a problem hiding this comment.
Updated with 6586ef2.
- Now add a warning when
includeThoughtsis used with thegoogleprovider - added a test for above
- removed
includeThoughtsaddition from Google provider. - Added a
generateTextexample.
… thinking config chore(providers/google): Remove `includeThoughts` from google provider docs chore(providers/google): Add unit test for reasoning warning when google provider is used with `includeThoughts`
|
Thanks @Und3rf10w for implementing this. I'm surprised that Gemini APIs do not return thought tokens, because https://aistudio.google.com/ does support this in the UI as well as generated code - for all displayed programming languages: 
|
@nileshtrivedi, I agree. I am sure it's supported but undocumented as of right now in the Gemini AI studio logic, and that the Vertex API is likely also how it's going to work.
|
|
I am seeing thinking tokens working only with 2.5-flash, not 2.5-pro (via Vertex). I get this error message: |
I can confirm, I am experiencing this now, but it did work. The vertex documentation still states support for the 2.5 pro models, so maybe it's a new model version they deployed or something? |
Ports the changes from this PR: vercel#6261 to the v5 branch. Adds support for Gemini's thinking messages and the includeThoughts flag in the google and vertex configs.
|
How to disable thinking on vertex provider altogether for 'gemini-2.5-pro-preview', I'm getting this error: Oddly for 'gemini-2.5-flash-preview' it seems to work to disable it by setting the thinkingBudget to 0 (includeThoughts is not related to fully disabling it right, it's only about if thinking IS enabled to make the model send thinking tokens if my understanding is correct?). Did they make thinking mandatory on 'gemini-2.5-pro-preview'? Is anybody else having difficulties? Such a wierd move if it is required, given they don't let us see the thinking. This is really keeping me back from using 2.5-pro for anything on the api as the thinking is a black box and is messing up a lot of prompt outputs because it gets confused by its own thinking lol... |
I've created an issue in google's issue tracker, please comment if you are having issues too: |
Did you try At one point of the 2.5 pro exp model, you were able to configure the thinking, but now that's only limited to 2.5 flash |
Both or either result in the same error that thinking config is not allowed :( |
I'm using gemini-2.5-flash-preview-05-20, but every time the response includes reasoning or thoughts, even though I've clearly asked in the prompt not to include them. I also set |
## Background [Vertex now supports extraction of thinking tokens in certain Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking). When the configuration is passed via `providerOptions`, the sdk: 1. Did not extract reasoning tokens 2. Did not pass `include_thoughts` to the provider ## Summary Added extraction logic to google-generative-ai package to parse reasoning tokens. Added a `includeThoughts` switch to the `thinkingConfig` for vertex models. ## Verification I verified it manually. Testable via examples/ai-core/src/stream-text/google-vertex-reasoning.ts. Easily copiable to google provider. ## Related Issues Fixes vercel#6259
[Vertex now supports extraction of thinking tokens in certain Gemini models](https://cloud.google.com/vertex-ai/generative-ai/docs/thinking). When the configuration is passed via `providerOptions`, the sdk: 1. Did not extract reasoning tokens 2. Did not pass `include_thoughts` to the provider Added extraction logic to google-generative-ai package to parse reasoning tokens. Added a `includeThoughts` switch to the `thinkingConfig` for vertex models. I verified it manually. Testable via examples/ai-core/src/stream-text/google-vertex-reasoning.ts. Easily copiable to google provider. Fixes vercel#6259
6 participants
Background
Vertex now supports extraction of thinking tokens in certain Gemini models.
When the configuration is passed via
providerOptions, the sdk:include_thoughtsto the providerSummary
Added extraction logic to google-generative-ai package to parse reasoning tokens.
Added a
includeThoughtsswitch to thethinkingConfigfor vertex models.Verification
I verified it manually. Testable via examples/ai-core/src/stream-text/google-vertex-reasoning.ts. Easily copiable to google provider.
Tests have been added.
Tasks
pnpm changesetin the project root)pnpm prettier-fixin the project root)Related Issues
Fixes #6259