When running upskill eval with -m for model selection (e.g. -m anthropic.claude-sonnet-4-20250514), the underlying FastAgent session receives the model string as the user prompt. This causes the evaluator to respond “you shared a model identifier…” instead of using the test case input.
Root cause:
FastAgent parses CLI args by default and treats -m/--message as a user message. Upskill’s -m model flag collides with that, so the model string gets injected as the message.
Fix:
Disable FastAgent CLI parsing when embedding it in upskill (parse_cli_args=False), and normalize provider-prefixed models when --provider is provided.
Repro:
upskill eval ./skills/<skill> --provider anthropic -m claude-sonnet-4-20250514 -t /tmp/tests.json --runs 1 -v
Observed user content == "anthropic.claude-sonnet-4-20250514" (model string)
Expected user content == case["input"]
When running
upskill evalwith-mfor model selection (e.g.-m anthropic.claude-sonnet-4-20250514), the underlying FastAgent session receives the model string as the user prompt. This causes the evaluator to respond “you shared a model identifier…” instead of using the test case input.Root cause:
FastAgent parses CLI args by default and treats
-m/--messageas a user message. Upskill’s-mmodel flag collides with that, so the model string gets injected as the message.Fix:
Disable FastAgent CLI parsing when embedding it in upskill (
parse_cli_args=False), and normalize provider-prefixed models when--provideris provided.Repro:
upskill eval ./skills/<skill> --provider anthropic -m claude-sonnet-4-20250514 -t /tmp/tests.json --runs 1 -vObserved user content == "anthropic.claude-sonnet-4-20250514" (model string)
Expected user content == case["input"]