⚡ [Performance] Run tool calls concurrently in acpPeer#85
⚡ [Performance] Run tool calls concurrently in acpPeer#85TrueAlpha-spiral wants to merge 1 commit intomainfrom
Conversation
Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
|
👋 Jules, reporting for duty! I'm here to lend a hand with this pull request. When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down. I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job! For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with New to Jules? Learn more at jules.google/docs. For security, I will only act on instructions from the user who triggered this task. |
|
What’s up with this? This is not the Sentient Lock. |
💡 What:
Replaced the sequential
for...ofloop running tool calls with a concurrentPromise.allapproach inGeminiAgent.resolveUserMessage.🎯 Why:
The tool calls generated by the model in a single prompt execution are independent. Executing them sequentially via
await this.#runTool(...)introduced unnecessary latency, as each tool had to wait for the previous one to finish executing before it could start. By executing them concurrently viaPromise.all, we significantly reduce the time needed to process batched tool responses.📊 Measured Improvement:
Using a custom benchmark simulating sequential vs parallel execution overhead with a synthetic 100ms delay tool, execution of 3 parallel tool invocations decreased from
~300msdown to~100ms, representing a 3x speedup on operations dependent on I/O. For high latency/batched tool executions, this drastically reduces time-to-first-token in the CLI.Passed all workspace tests via
npm run test --workspaces. Formatting is verified via prettier.PR created automatically by Jules for task 7303597159464572492 started by @TrueAlpha-spiral