Wire up tool calling execution in streaming responses

[anurag629]

Summary

Tools (e.g., LeadCaptureTool) are defined and passed to AI providers, but tool-use responses from the providers are never detected or executed. The providers currently only yield text deltas — when the AI decides to call a tool, the tool_use / function_call response blocks are silently ignored. This means lead capture (a headline feature) does not actually work.

This issue covers wiring up the full tool-calling loop: detect tool-use in provider responses → execute the tool → feed the result back to the provider → stream the final text response to the client.

Current Behavior

1. Server passes tools to the provider via ProviderChatParams.tools (packages/server/src/handler.ts:101)
2. Each provider converts tool definitions to the provider-specific format:
   • Claude: toClaudeTools() in packages/core/src/providers/base.ts:35-45
   • OpenAI: toOpenAITools() in packages/core/src/providers/base.ts:47-60
   • Gemini: toGeminiTools() in packages/core/src/providers/base.ts:68-86
3. Provider sends tools to the API — but streaming only yields text_delta content:
   • Claude (packages/core/src/providers/claude.ts:33-37): only handles content_block_delta with text_delta type
   • OpenAI (packages/core/src/providers/openai.ts:36-41): only reads delta.content
   • Gemini (packages/core/src/providers/gemini.ts:43-47): only reads chunk.text()
4. When the AI wants to call a tool (e.g., capture_lead), the tool-use block is silently skipped — no tool is executed, no result sent back
5. ChatTool.execute() (packages/core/src/tools/base.ts:8) is never called from anywhere in the codebase

Expected Behavior

1. Provider detects a tool-use response from the AI
2. Server looks up the tool by name from the configured tools array
3. Server calls tool.execute(input) with the parsed arguments
4. Tool result is sent back to the provider as a tool-result message
5. Provider continues generating text based on the tool result
6. Final text is streamed to the widget
7. Widget receives optional metadata (e.g., leadCaptured: true) via the SSE stream

Implementation Guide

Phase 1: Update Provider Interface

File: packages/core/src/providers/base.ts

The AIProvider.chat() method currently returns AsyncGenerator<string> — it can only yield text. Change the yield type to support both text and tool-use events:

// New yield type for the chat generator
export interface ChatYield {
  type: 'text' | 'tool_use';
  text?: string;
  toolUse?: {
    id: string;
    name: string;
    input: Record<string, unknown>;
  };
}

export interface AIProvider {
  name: string;
  chat(params: ProviderChatParams): AsyncGenerator<ChatYield>;
  chatSync(params: ProviderChatParams): Promise<string>;
}

Phase 2: Update Each Provider to Detect Tool Use

Claude (packages/core/src/providers/claude.ts)

Claude's streaming API emits content_block_start with type: 'tool_use' followed by input_json_delta events. The current code at line 33-37 only handles text_delta:

// Current (lines 33-37) — only handles text
for await (const event of stream) {
  if (event.type === 'content_block_delta' && event.delta.type === 'text_delta') {
    yield event.delta.text;
  }
}

Update to also handle tool_use blocks:

• Track content_block_start events where content_block.type === 'tool_use' to capture the tool id and name
• Accumulate input_json_delta chunks to build the full tool input
• On content_block_stop, yield a ChatYield with type: 'tool_use'

OpenAI (packages/core/src/providers/openai.ts)

OpenAI's streaming chunks include delta.tool_calls when the model wants to call a function. The current code at line 36-41 only reads delta.content:

// Current (lines 36-41) — only handles content
for await (const chunk of stream) {
  const delta = chunk.choices[0]?.delta?.content;
  if (delta) yield delta;
}

Update to also detect chunk.choices[0]?.delta?.tool_calls and accumulate the function name + arguments across chunks.

Gemini (packages/core/src/providers/gemini.ts)

Gemini returns functionCall parts in chunk.candidates[0].content.parts. The current code at line 43-47 only reads chunk.text():

Update to check for part.functionCall in the response parts.

Phase 3: Tool Execution Loop in Server Handler

File: packages/server/src/handler.ts

The handler at lines 96-110 currently streams provider output directly to the client. Add a tool execution loop:

// Pseudocode for the updated handler loop
const toolMap = new Map(config.tools?.map(t => [t.name, t]) ?? []);

let continueLoop = true;
while (continueLoop) {
  continueLoop = false;
  
  for await (const chunk of provider.chat({ messages, systemPrompt, tools: toolDefs })) {
    if (chunk.type === 'text' && chunk.text) {
      fullResponse += chunk.text;
      yield JSON.stringify({ content: chunk.text });
    }
    
    if (chunk.type === 'tool_use' && chunk.toolUse) {
      const tool = toolMap.get(chunk.toolUse.name);
      if (tool) {
        const result = await tool.execute(chunk.toolUse.input);
        
        // Add tool-use + tool-result messages to conversation
        // This lets the provider see what happened
        messages.push(/* assistant tool_use message */);
        messages.push(/* tool result message */);
        
        // Signal tool execution to widget
        if (chunk.toolUse.name === 'capture_lead' && result.success) {
          yield JSON.stringify({ leadCaptured: true, leadData: result.data });
        }
        
        continueLoop = true; // Re-call provider with tool result
      }
    }
  }
}

Key considerations:

• Add a max tool-call iterations guard (e.g., 5) to prevent infinite loops
• The conversation messages array needs to include tool-use and tool-result messages in the format each provider expects
• ChatMessage type in packages/core/src/types.ts may need a toolUse or toolResult field

Phase 4: Wire Up Webhook Dispatch

File: packages/server/src/handler.ts

The WebhookDispatcher is instantiated at line 18 but webhooks.dispatch() is never called. After a tool executes successfully, dispatch the event:

if (chunk.toolUse.name === 'capture_lead' && result.success) {
  webhooks?.dispatch('lead:captured', { lead: result.data, conversationId: req.conversationId });
}

Phase 5: Update Widget to Handle Tool Events

File: packages/widget/src/widget.ts

The widget's handleSend() method at line 253-284 already checks for chunk.leadCaptured implicitly via the WidgetChatChunk type (packages/widget/src/api/types.ts:17), but doesn't act on it. Add handling:

if (chunk.leadCaptured) {
  this.config.onLeadCaptured?.(chunk.leadData);
  this.emit('leadCaptured', chunk.leadData);
}

Add onLeadCaptured to WidgetConfig interface.

Files to Modify

File	Change
packages/core/src/providers/base.ts	New ChatYield type, update AIProvider interface
packages/core/src/providers/claude.ts	Handle tool_use content blocks in streaming
packages/core/src/providers/openai.ts	Handle tool_calls in streaming deltas
packages/core/src/providers/gemini.ts	Handle functionCall parts in streaming
packages/core/src/types.ts	Extend ChatMessage for tool-use/tool-result roles
packages/server/src/handler.ts	Add tool execution loop, webhook dispatch
packages/widget/src/api/types.ts	Add leadData to WidgetChatChunk
packages/widget/src/widget.ts	Handle leadCaptured event, add config callback

Tests to Add

• Unit: Each provider correctly yields ChatYield with type: 'tool_use' when mock API returns tool-use response
• Unit: Handler executes tool and re-calls provider with result
• Unit: Handler respects max iterations guard
• Unit: Widget emits leadCaptured event when chunk contains leadCaptured: true
• Integration: Full flow — user message → provider requests tool → tool executes → provider responds with text

Acceptance Criteria

• When the AI decides to call capture_lead, the tool actually executes and the callback fires
• The AI receives the tool result and generates a natural follow-up message
• The widget receives leadCaptured event with lead data
• Works with all 3 providers (Claude, OpenAI, Gemini)
• Max iterations guard prevents infinite tool-calling loops
• Existing text-only conversations (no tools configured) work unchanged
• All existing tests pass, new tests added for tool execution flow

[m13v]

the key issue with tool calling in streaming responses is that you need to accumulate the tool_use content blocks across multiple SSE chunks before you can actually execute the tool. the model sends the tool name first, then streams the JSON arguments incrementally. you can't execute until the arguments are complete. our approach: buffer tool_use blocks in a map keyed by tool_use_id, execute when the block is closed, then inject the result as a toolResult in the next turn

[m13v]

Our streaming tool execution implementation for Claude API: https://github.com/m13v/fazm/blob/main/Desktop/Sources/Providers/ChatToolExecutor.swift

Handles the full cycle: accumulate tool_use blocks from streaming, validate arguments, execute, inject toolResult, continue the conversation. The tricky part is error handling when a tool execution fails mid-stream.

[anurag629]

Thanks @m13v! The buffer-by-tool_use_id → execute-on-close → reinject pattern is exactly the approach I'm planning for the accumulation layer. That's the right mental model for all three providers (Claude's input_json_delta, OpenAI's chunked tool_calls arguments, and Gemini's functionCall parts).

Took a look at the ChatToolExecutor — the dispatch-and-return pattern is solid, though since this is TypeScript with AsyncGenerators rather than Swift static callbacks, the implementation will look quite different structurally. The error handling point is a good one though — I need to decide whether a failed tool execution should surface as an error SSE event to the client or get fed back to the AI as a failed tool result so it can respond gracefully. Leaning toward the latter (let the AI explain the failure to the user).

If you're interested in contributing, the trickiest part is going to be the provider-specific message formatting for tool results — Claude, OpenAI, and Gemini each expect different shapes for tool result messages. That and parallel tool call support (both Claude and OpenAI can request multiple tools in one turn).