feat: add LiteLLM as AI gateway provider

[RheagalFire]

Summary

• Adds LiteLLM class in llm.py extending the existing LLM class
• Gives users access to 100+ LLM providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Groq, Cohere, Ollama, etc.) through a single unified interface
• Users switch by setting CORECODER_PROVIDER=litellm and using any LiteLLM model string
• The existing LLM class (OpenAI SDK) remains the default, unchanged

---

• What kind of change does this PR introduce? Feature, adds LiteLLM as an alternative LLM backend

• Why was this change needed? CoreCoder currently supports OpenAI-compatible endpoints only. LiteLLM extends this to providers that are NOT OpenAI-compatible (AWS Bedrock, Google Vertex, Cohere, etc.)
  through a single completion() call. Users switch providers by changing the model string.

• Other information: See sections below.

---

What Changed

File	Change
corecoder/llm.py	Added LiteLLM class extending LLM, overriding __init__, chat(), and _call_with_retry()
corecoder/config.py	Added provider field (default "openai", option "litellm") + CORECODER_PROVIDER env var
corecoder/cli.py	Updated LLM instantiation to pick class based on config.provider
pyproject.toml	Added litellm>=1.60.0,<2.0.0 as optional dependency (pip install corecoder[litellm])
tests/test_litellm.py	26 unit tests

---

Usage

Quick Start

# Install with litellm support
pip install corecoder[litellm]

# Set the provider and model                                                                                                                                                                                       
export CORECODER_PROVIDER=litellm
export CORECODER_MODEL=anthropic/claude-3-haiku                                                                                                                                                                    
export ANTHROPIC_API_KEY=sk-ant-...
                                                                                                                                                                                                                   
# Run as usual
corecoder                                                                                                                                                                                                          
                
Switching Providers: Just Change the Model String                                                                                                                                                                  
 
# OpenAI (works without litellm too, via OPENAI_BASE_URL)                                                                                                                                                          
export CORECODER_PROVIDER=litellm                                                                                                                                                                                  
export CORECODER_MODEL=openai/gpt-4o
export OPENAI_API_KEY=sk-...                                                                                                                                                                                       
                
# Anthropic                                                                                                                                                                                                        
export CORECODER_MODEL=anthropic/claude-3-haiku
export ANTHROPIC_API_KEY=sk-ant-...                                                                                                                                                                                
                
# AWS Bedrock
export CORECODER_MODEL=bedrock/anthropic.claude-v2
export AWS_ACCESS_KEY_ID=...                                                                                                                                                                                       
export AWS_SECRET_ACCESS_KEY=...
                                                                                                                                                                                                                   
# Google Vertex AI
export CORECODER_MODEL=vertex_ai/gemini-pro
export VERTEXAI_PROJECT=your-project                                                                                                                                                                               
 
# Groq                                                                                                                                                                                                             
export CORECODER_MODEL=groq/llama3-70b-8192
export GROQ_API_KEY=gsk_...                                                                                                                                                                                        
 
# Local Ollama                                                                                                                                                                                                     
export CORECODER_MODEL=ollama/llama3

Any of the https://docs.litellm.ai/docs/providers works.

Programmatic Usage

  from corecoder.llm import LiteLLM

  llm = LiteLLM(
      model="anthropic/claude-3-haiku",
      # api_key is optional, litellm reads env vars automatically                                                                                                                                                    
  )
                                                                                                                                                                                                                     
  result = llm.chat(                                                                                                                                                                                                 
      messages=[{"role": "user", "content": "What is 2+2?"}],
      on_token=lambda t: print(t, end="", flush=True),                                                                                                                                                               
  )               
  print(result.content)                                                                                                                                                                                              

---

Testing

Unit Tests (26/26 Passing)

tests/test_litellm.py::TestLiteLLMClass::test_extends_llm PASSED
tests/test_litellm.py::TestLiteLLMClass::test_init_does_not_create_openai_client PASSED                                                                                                                            
tests/test_litellm.py::TestLiteLLMClass::test_init_stores_model PASSED                                                                                                                                             
tests/test_litellm.py::TestLiteLLMClass::test_init_stores_api_key PASSED
tests/test_litellm.py::TestLiteLLMClass::test_init_stores_base_url PASSED                                                                                                                                          
tests/test_litellm.py::TestLiteLLMClass::test_init_stores_extra_kwargs PASSED                                                                                                                                      
tests/test_litellm.py::TestLiteLLMClass::test_token_counters_start_at_zero PASSED
tests/test_litellm.py::TestCallWithRetry::test_passes_drop_params PASSED                                                                                                                                           
tests/test_litellm.py::TestCallWithRetry::test_forwards_api_key PASSED
tests/test_litellm.py::TestCallWithRetry::test_omits_api_key_when_none PASSED                                                                                                                                      
tests/test_litellm.py::TestCallWithRetry::test_forwards_api_base PASSED
tests/test_litellm.py::TestCallWithRetry::test_omits_api_base_when_none PASSED                                                                                                                                     
tests/test_litellm.py::TestChat::test_returns_llm_response PASSED
tests/test_litellm.py::TestChat::test_tracks_token_usage PASSED                                                                                                                                                    
tests/test_litellm.py::TestChat::test_on_token_callback PASSED
tests/test_litellm.py::TestChat::test_model_forwarded PASSED                                                                                                                                                       
tests/test_litellm.py::TestConfigProvider::test_default_provider_is_openai PASSED                                                                                                                                  
tests/test_litellm.py::TestConfigProvider::test_provider_from_env PASSED
tests/test_litellm.py::TestConfigProvider::test_cli_picks_litellm_class PASSED                                                                                                                                     
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[openai/gpt-4o] PASSED                                                                                                                        
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[anthropic/claude-3-haiku] PASSED
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[bedrock/anthropic.claude-v2] PASSED                                                                                                          
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[vertex_ai/gemini-pro] PASSED                                                                                                                 
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[groq/llama3-70b-8192] PASSED                                                                                                                 
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[ollama/llama3] PASSED                                                                                                                        
tests/test_litellm.py::TestMultiProvider::test_model_string_forwarded[azure/gpt-4o] PASSED
============================== 26 passed in 0.23s ==============================                                                                                                                                   

E2E Tests(Azure AI Foundry, Claude Sonnet via LiteLLM)

Model: azure_ai/claude-sonnet-4-6                                                                                                                                                                                  
                
--- E2E 1: Basic streaming chat ---                                                                                                                                                                                
Prompt: "What is 2+2? Reply with just the number."
Response: "4"                                                                                                                                                                                                      
Streamed tokens: ["4"]

--- E2E 2: Multi-turn conversation ---                                                                                                                                                                             
Messages:
  system: "You are a math tutor. Be very brief."                                                                                                                                                                   
  user: "What is 10 * 5?"
  assistant: "50"
  user: "Divide that by 2"                                                                                                                                                                                         
Response: "25"
                                                                                                                                                                                                                   
--- E2E 3: Config provider selection ---
Config.provider=litellm selects LiteLLM class: True
                                                                                                                                                                                                                   
--- E2E 4: Error handling (invalid model) ---
Model: nonexistent/bad-model                                                                                                                                                                                       
Caught expected error: BadRequestError
"LLM Provider NOT provided. Pass in the LLM provider you are trying to call."

---

Risk / Compatibility

• Additive only, existing LLM class unchanged
• LiteLLM is an optional dependency (pip install corecoder[litellm]), not required for default usage
• drop_params=True set by default so provider-unsupported kwargs are silently dropped
• Default provider remains "openai", LiteLLM only activates when CORECODER_PROVIDER=litellm

[RheagalFire]

cc @he-yufeng