The LlamaDatasetGenerator and OpenAIDatasetGenerator currently raise a ValueError immediately if the LLM response cannot be parsed as JSON.
File: src/agentunit/generators/llm_generator.py
LLMs often fail 1-2 times before succeeding.
Task:
- Implement a simple retry loop (e.g., 3 attempts) when occurs.
- Optionally, feed the error back to the LLM to correct its output in the next attempt.
The
LlamaDatasetGeneratorandOpenAIDatasetGeneratorcurrently raise aValueErrorimmediately if the LLM response cannot be parsed as JSON.File:
src/agentunit/generators/llm_generator.pyLLMs often fail 1-2 times before succeeding.
Task: