Remove allows_unlisted_models — always allow custom model entry

[codefromthecrypt]

Please explain the motivation behind the feature request.

We keep merging one-liner PRs that flip allows_unlisted_models to true per provider:

• fix: enable custom model entry for OpenRouter provider #6761 — OpenRouter
• feat(gcp-vertex): add model list with org policy filtering #6393 — GCP Vertex AI
• fix: allow unlisted models for CLI providers #7090 — claude-code, codex, gemini-cli, cursor-agent, chatgpt-codex

The underlying problem is that our known_models lists don't match what the APIs actually return. I ran fetch_supported_models() against the real APIs:

• OpenAI: Our known_models has 11 entries but the API returns 115 models. Many valid chat models (gpt-5, gpt-5-nano, gpt-5-pro, gpt-5.2-codex) aren't in our list, and each also has a dated variant (gpt-5-nano-2025-08-07). Without unlisted enabled, users can't select any of them.
• Ollama: API returns names with tags (qwen3:latest, qwen3:4b) but known_models lists bare names (qwen3). Zero of our 4 known models match the fetched list due to the :tag mismatch.
• Claude Code: fetch_supported_models() returns only ["sonnet", "opus"] — hardcoded, no API. You can't enter claude-sonnet-4-5-20250929 without unlisted enabled.

The lists serve two purposes: (a) recommending known-good text+tools models, and (b) blocking users from entering arbitrary names. But (b) doesn't hold up — GOOSE_MODEL env var bypasses the UI entirely, and the canonical registry already resolves aliases to real names.

Describe the solution you'd like

Remove the allows_unlisted_models field from ProviderMetadata and the with_unlisted_models() builder method entirely. The configure UI would always show "Enter a model not listed..." alongside the curated list. If a user enters a model name the provider doesn't support, they get an API error immediately — no need to gate this in the UI.

Describe alternatives you've considered

Default allows_unlisted_models to true instead of removing it. But if no provider has a reason to set it false, the flag is dead code. The simpler option is to delete it.

Additional context

#7065 attempted a broader fix ("use dynamic model listing for all providers") but was closed. #5520 and #4834 also tried to address the same root issue from different angles.

• I have verified this does not duplicate an existing feature request

[michaelneale]

cc @katzdave who I think was looking at model name normalisation across providers (not sure if that helps with this).

Another option is a .json on github somewhere which is always live reloaded from any goose client so it is kept fresh outside of any releases as models change fast (if we wanted to have a hand made list... which I am not sure is the aim or not).

[codefromthecrypt]

I think what is making this issue harder than normal LLM abstractions is that we mask valid models preventing their validation against normal raw list (iirc).

There are probably at least a dozen agent or llm abstractions to swap notes on. Some have quirks but generally if a model is in the list you can use it and visa versa the same.

I would love validating tests that our preferred are always selectable and/or a build warn or fail if we are out of date (e.g. gpt-xxx-codex pattern instead of maintenance causing allow list of specific versions)

P.s. The exceptions about ollama and claude we could fix at the provider layer.. e.g. backfill plain name for any :latest and also add manual sonnet and opus alias entries to the claude list.

[codefromthecrypt]

Ps on the hosted json thing for recommendations.

I recall some agent framework that hosted json for pricing and it would be unavailable for whatever reason and break apps. So we would need to be careful about that. One way could be to use the exact same json in the release and override when available from the remote source. In that case the fallback would be old limited to last time you updated

[codefromthecrypt]

this PR documents why unlisted is always needed in claude, as only aliases are returned, at the moment "default" "sonnet" and "haiku" where detailed text of default says it is opus #7120

[katzdave]

So the plan on my end is:

canonical_models.json is the point of truth. Models.dev is currently the main source, but will eventually mix more data in for better coverage on local models.

This gets updated + bundled every release. I think its a high enough cadence since we can patch on major model releases, but maybe we'll eventually add a way to force it to refresh for your client.

KNOWN_MODELS is another abstraction I'd like to delete. Generally thinking that when providers return a massive list of models, we match them against canonical models and filter ones which match + support text. This has worked well so far for openai, anthropic, xai, google etc, but might still need an allow_unlisted for other providers.

[codefromthecrypt]

@katzdave sounds good. Small twist for consideration.

Right now our Provider abstraction is synonymous with "LLM provider" but this is already starting to diverge with a class of "Agentic Provider", starting with the CLI ones like codex and claude but continuing with ACP.

Many of these Agentic Providers are already curated lists of models. In some cases you could imagine even LLM providers returning a profile-specific list, but anyway def the agentic ones wont return embedding models as they cant be used.

I like a built in model list that an integration test can verify matches up, and fails where it doesn't. These can use registries of choice even ollamas for expressions instead of specific model ids.

So as you imply latest goose always has a an offline fallback if we want to filter the raw API. However, Agentic providers and some LLM providers may want to use their raw lists and avoid drift on already curated model lists.

Hope this helps