add an evaluator for skills

[peterj]

Problem or motivation

We want to have a dedicated skills evaluator (probably similar to rubric_based_tool_use_quality_v1) that would be used to check whether specific skills have been called and perhaps the order of them too.

Proposed solution

We want to have a dedicated skills evaluator (probably similar to rubric_based_tool_use_quality_v1) that would be used to check whether specific skills have been called and perhaps the order of them too.

Alternatives considered

No response

Additional context

No response

Human confirmation

• I am a human (not a bot, agent, or AI) filing this issue.

[frivas-at-navteca]

Hello @peterj I was wondering if this evaluator should work like the rubric_based_tool_use_quality_v1 in terms of using the LLM as a judge, the rubric can be configured etc. I would like to understand more to have a better idea as I got a draft but I realized I need to understand a few things first. If I may kindly ask you to elaborate just a bit more. Would it be ok for this first version have a deterministic set of rules or we should work on an LLM judged based one?

I am trying to understand the goal. I have been taking a look at the rubric_based_tool_use_quality_v1 and for skills it seems the approach might be different. I am wondering if the skills evaluator should provide a more reproducible results. I mean introducing an LLM judge introduces some non-reproducible (or stable?) results. My understanding, sorry if I am completely wrong and misunderstood the goal here, is to have a way to evaluate skills in a way that one can explain why they got that score, if the tools called was correct either in order or not, like what supposed to happen happened and not leave that to the interpretation of an LLM. Am I making sense?

Thank you very much again!

[peterj]

Hi @frivas-at-navteca ! we need to look into how skill creator is doing evals for skills (https://github.com/anthropics/skills/blob/main/skills/skill-creator/SKILL.md). Also, @krisztianfekete has an example of how the skill evals look like for the agentevals skill here: https://github.com/agentevals-dev/agentevals/blob/main/.claude/skills/eval/evals/evals.json.

[frivas-at-navteca]

Awesome @peterj I will review those and see if I can come up with something. I believe this is a hard one and will need a lot of review and back and forth which I am up for it. This is very interesting. Thank you very much for the links!

[frivas-at-navteca]

Hello @peterj thanks for links again!

I can see that Anthropic and Krisztian's proposals are more like evidence-based(?) evaluations of a skill. If I understood correctly, Anthropic's is like a test-driven with assertions (like Krisztian's with a list of assertions) a bit of behavior which I understand is to improve the skill based the results of those tests. What you have mentioned sounds more like a tracking or trajectory evaluation, as you said checking if specific tools were called or certain actions happened and also if they happened in a certain order.

Would it make sense for v1 to have a more deterministic evaluator like the one your are proposing and another one like Anthropic and Krisztian's? I am asking this because I am not sure if mixing different approaches might provide at the end a good score and they measure different things in different ways.

With the trajectory one (not sure if this can be a proper name) you can do smoke tests and works good in CI/CD or even to check if a workflow has regression. The evidence one (again, not sure if that name would be correct) is probably richer based on assertions and behavior. I like this way of having two because you don't have to run the whole thing if you are not interested on what the evidence-based one shows or viceversa. The trajectory one can be offered as an out-of-the-box evaluator while the evidence one might need the assertions to be configured.

One more thing I just thought is the UI, how will all this be visualized? I understand this issue is only for the backend and a new one for the UI will be opened later.

Sorry for the rambling I am just thinking about this to find a way to work it out and help.

Thank you very much, again, for your time and comments and for the very welcoming environment.

[krisztianfekete]

Thanks @frivas-at-navteca looking into this!

I think we'd generally want something closer "trajectory" based, or at least I am not yet sure how to do the Anthropic approach in a consistent way with our existing evals.

For this, we have some gaps today, as skills are not first class citizens in OTel GenAI semconv, see open-telemetry/semantic-conventions#3540 for further details.

We can either wait for this, or try to do some best effort estimations, e.g. infer skill A as invoked, if certain tools are called in a certain order, or via certain parameters (just thinking out loud here). We can build some kind of abstraction around this, similarly to tool trajectory metrics, but that got me thinking: does that worth the extra effort compared to some of our existing evals, e.g. tool_sequence_match, or even contains, or some of the OpenAI Graders?

What do you all think?

Let me move this discussion to the evaluator repository!

[frivas-at-navteca]

Thanks @krisztianfekete and @peterj

It makes sense to wait for that implementation definitely, though which we do not know how long it will take, because it will change how the evaluator will be implemented in aevals. Now I am thinking about what is the priority/demand for these metrics. What is the roadmap for this?

If the roadmap allows it, it sounds reasonable to wait for skills to become a first class citizen in OTel. If not then we can do a minimal trajectory-based evaluator for skills as they are now that also works with aevals. However in any case to me it still makes sense to have a split in trajectory and evidence based evaluators as mentioned earlier.

I just found this and thought it is interesting to also keep on our radar.

Thank you very much again.

[krisztianfekete]

I think it will take quite a while before we have the semantic conventions in place (and even more for actual SDK support), so IMO if you can take a look at how we can add support for the trajectory-based approach and flesh out some implementations/pseudo-code, I am happy to brainstorm and test them!

[frivas-at-navteca]

Hello @krisztianfekete sure. I would be glad. I have opened a PR with a stand alone one as a first attempt. Was that ok? Hopefully yes. Thank you very much for taking the time to check it out and suggest improvements/corrections.