Skip to content

feat: add timeout to Agent.generate_response() #2

@rosspeili

Description

@rosspeili

Problem Statement

If a local model (e.g., Ollama) or an external API provider hangs or becomes unresponsive, the session is currently blocked indefinitely. There is no timeout mechanism in the Agent.generate_response() call, which can lead to a "frozen" interface.

Proposed Solution

  1. Add a timeout: int = Field(default=30) field to the AgentConfig model in config.py.
  2. Pass this timeout parameter to the litellm.completion() call within agent.py.
  3. Wrap the completion call in a try/except block to catch litellm.Timeout errors.
  4. Return a graceful error message (or trigger a retry) instead of allowing the process to hang.

Alternatives Considered

None. Robustness against infrastructure failure is a core requirement.

Priority

High 🔴

Additional Context

Essential for reliability in local environments where local model serving might be unstable or slow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingenhancementNew feature or request

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions