feat: add streaming with tools support and resilience

[nazq]

Summary

• Add StreamChunk enum and chat_stream_with_tools() method for streaming responses while handling tool calls in real-time
• Implement for Anthropic (custom SSE parser) and OpenAI-compatible providers (OpenAI, Mistral, Groq, Cohere, OpenRouter, HuggingFace)
• Add resilience (retry with exponential backoff) to chat_stream_struct() and chat_stream_with_tools() in ResilientLLM

Test plan

• 24 unit tests for SSE parsing (Anthropic + OpenAI-compatible)
• Integration tests for streaming with tools (Anthropic)
• Integration tests for resilient streaming (test_anthropic_resilient_chat_stream_with_tools, test_anthropic_resilient_chat_stream_struct)
• cargo clippy --lib --features full passes
• cargo fmt applied

API

// New enum for streaming with tools
pub enum StreamChunk {
    Text(String),
    ToolUseStart { index: usize, id: String, name: String },
    ToolUseInputDelta { index: usize, partial_json: String },
    ToolUseComplete { index: usize, tool_call: ToolCall },
    Done { stop_reason: String },
}

// New trait method
async fn chat_stream_with_tools(
    &self,
    messages: &[ChatMessage],
    tools: Option<&[Tool]>,
) -> Result<Pin<Box<dyn Stream<Item = Result<StreamChunk, LLMError>> + Send>>, LLMError>;

Usage

let llm = LLMBuilder::new()
    .backend(LLMBackend::Anthropic)
    .api_key(api_key)
    .model("claude-3-5-haiku-20241022")
    .resilient(true)
    .function(/* ... */)
    .build()?;

let mut stream = llm.chat_stream_with_tools(&messages, llm.tools()).await?;
while let Some(chunk) = stream.next().await {
    match chunk? {
        StreamChunk::Text(t) => print!("{}", t),
        StreamChunk::ToolUseComplete { tool_call, .. } => {
            // Handle tool call
        }
        StreamChunk::Done { stop_reason } => break,
        _ => {}
    }
}

[graniet]

Thank you for your work ! @nazq

[Copilot]

Pull request overview

This PR adds streaming support for LLM responses with tool calls, enabling real-time processing of both text and tool invocations. It implements the new chat_stream_with_tools() method across Anthropic and OpenAI-compatible providers (OpenAI, Mistral, Groq, Cohere, OpenRouter, HuggingFace), and extends resilience capabilities to streaming operations.

Key changes:

• Introduces StreamChunk enum to represent different streaming events (text, tool use start/delta/complete, done)
• Implements custom SSE parsers for Anthropic and OpenAI-compatible streaming formats that handle tool calls
• Adds retry logic with exponential backoff to chat_stream_struct() and chat_stream_with_tools() in ResilientLLM

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/chat/mod.rs	Defines StreamChunk enum and adds chat_stream_with_tools() trait method with comprehensive documentation
src/backends/anthropic.rs	Implements Anthropic-specific streaming with tools, including SSE parser and 18 unit tests covering various streaming scenarios
src/backends/openai.rs	Delegates streaming with tools to the generic OpenAICompatibleProvider implementation
src/providers/openai_compatible.rs	Implements generic OpenAI-compatible streaming with tools, SSE parser, and 12 unit tests covering OpenAI and vLLM formats
src/resilient_llm.rs	Adds retry logic with exponential backoff to both chat_stream_struct() and chat_stream_with_tools() methods
tests/test_backends.rs	Adds 4 integration tests covering streaming with tools, text-only responses, and resilient streaming for both tools and struct modes

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

The message preparation logic (converting ChatMessage to AnthropicMessage) is duplicated between chat_with_tools() (lines 380-477) and chat_stream_with_tools() (lines 709-806). This is approximately 100 lines of identical code that should be extracted into a private helper method to improve maintainability and reduce the risk of inconsistencies.

[Copilot]

The tool preparation and tool_choice mapping logic is duplicated between chat_with_tools() (starting around line 479) and chat_stream_with_tools() (lines 808-839). Extract this into a private helper method to avoid duplication and ensure consistency.

Suggested change

	let maybe_tool_slice: Option<&[Tool]> = tools.or(self.tools.as_deref());
	let anthropic_tools = maybe_tool_slice.map(|slice| {
	slice
	.iter()
	.map(|tool| AnthropicTool {
	name: &tool.function.name,
	description: &tool.function.description,
	schema: &tool.function.parameters,
	})
	.collect::<Vec<_>>()
	});
	let tool_choice = match self.tool_choice {
	Some(ToolChoice::Auto) => {
	Some(HashMap::from([("type".to_string(), "auto".to_string())]))
	}
	Some(ToolChoice::Any) => Some(HashMap::from([("type".to_string(), "any".to_string())])),
	Some(ToolChoice::Tool(ref tool_name)) => Some(HashMap::from([
	("type".to_string(), "tool".to_string()),
	("name".to_string(), tool_name.clone()),
	])),
	Some(ToolChoice::None) => {
	Some(HashMap::from([("type".to_string(), "none".to_string())]))
	}
	None => None,
	};
	let final_tool_choice = if anthropic_tools.is_some() {
	tool_choice.clone()
	} else {
	None
	};
	let (anthropic_tools, final_tool_choice) = Self::prepare_anthropic_tools_and_choice(
	tools,
	self.tools.as_deref(),
	&self.tool_choice,
	);

[nazq]

Fair review, want an update to refactor ?

[nazq]

Addressed the copilot review above