Adding structured output

[sator-labs]

This is meant to be merged after #29

[Claude generated below]

Implementation Details

Prompt Changes

The question prompt (data/question_prompt.txt) was simplified:

Before:

Question: {question}
{examples_section}
Please answer with one of: [{options}]

ANSWER: [your answer]
REASONING: [brief explanation]

After:

Question: {question}
{examples_section}
Please answer with one of the following options: [{options}]

Provide your answer and a brief explanation for why you chose that answer based on the conversation.

The explicit format instructions are removed because the LLM understands the structure through the Pydantic model schema.

Removed Code

The following parsing methods were removed from llm_judge.py:

• _extract_answer() - No longer needed, answer comes from response.answer
• _extract_reasoning() - No longer needed, reasoning comes from response.reasoning

Answer Matching

A new helper method was added to validate answers:

def _match_answer_to_options(
    self, answer: str, valid_options: List[str]
) -> Optional[str]:
    """Try to match an answer to valid options using case-insensitive comparison."""
    answer_lower = answer.lower().strip()
    for option in valid_options:
        if option.lower().strip() == answer_lower:
            return option
        if option.lower() in answer_lower or answer_lower in option.lower():
            return option
    return None

This ensures that even with structured output, we validate the LLM chose a valid option from the rubric.

[sator-labs]

#27

[Copilot]

Pull request overview

This PR implements a structured output system for LLM-based judge evaluation, replacing fragile string parsing with type-safe Pydantic models. The changes enable reliable extraction of answers and reasoning from LLMs when evaluating conversations against clinical rubrics.

Key Changes:

• Added structured output support using Pydantic models across all LLM providers (Claude, OpenAI, Gemini, Llama)
• Implemented QuestionNavigator class to handle rubric question flow logic separately from judge evaluation
• Refactored judge evaluation to use structured responses instead of string parsing
• Added comprehensive test suite for question navigation functionality

Reviewed changes

Copilot reviewed 22 out of 24 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
llm_clients/llm_interface.py	Added abstract method for structured response generation
llm_clients/claude_llm.py, openai_llm.py, gemini_llm.py	Implemented structured output using LangChain's with_structured_output()
llm_clients/llama_llm.py	Added limited structured output support with NotImplementedError fallback
judge/response_models.py	New file defining Pydantic models for structured LLM responses
judge/question_navigator.py	New file extracting rubric parsing and navigation logic from judge
judge/llm_judge.py	Refactored to use structured output and question navigator
tests/test_question_navigator.py	New comprehensive test suite for question navigation
pyproject.toml	Added pytest dependency
docs/structured-output.md, docs/evaluating.MD	New and updated documentation
data/question_prompt.txt	New simplified question prompt template

Comments suppressed due to low confidence (2)

judge/llm_judge.py:1

• Method name changed from evaluate_conversation to evaluate_conversation_question_flow, but the old method name is still referenced in the runner. This suggests the method might have been renamed but not all call sites were updated.

"""LLM Judge for evaluating conversations based on rubrics."""

judge/score.py:1

• In the removed code, the line was pd.DataFrame(results, columns=[\"filename\"] + DIMENSIONS) but now it's pd.DataFrame(results, columns=columns) where columns = [\"filename\", \"run_id\"] + list(results[0].keys()). This could cause issues if results is empty, as results[0] would raise an IndexError.

#!/usr/bin/env python3

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[emily-vanark]

Why force the reasoning length to 100 here? I think I'd have expected it to default to None and take the whole string.... (also line 772?)

[sator-labs]

ugh thanks, I thought that was only for the mock test. let me revert to none

[emily-vanark]

I like this separation.