Model family

[borgr]

There are going to be generations of new models, people doing A\B testings etc. as well as different runners calling it by different names. We need to better define what we mean by a model family, it is going to be extended a lot, what should be our philosophy there?

class Family(Enum):
Llama = 'Llama'
Mistral = 'Mistral'
OLMo = 'OLMo'
gemma = 'gemma'
gpt = 'gpt'
palm = 'palm'
claude = 'claude'
falcon = 'falcon'
Qwen = 'Qwen'
NoneType_None = None

[damian1996]

Do we even need info about Family name? It seems to be not the best choice. It would be easier and better to keep one or many of the following informations:

• model name e.g. Qwen3-Coder-480B-A35B-Instruct, o4-mini
• model provider name e.g. HuggingFace (or https://huggingface.co/Qwen?), Azure OpenAI
• model series e.g. Qwen3, o series
• organization/company e.g. Qwen, OpenAI

All this data is definitely less ambigious than model family name.

A follow-up question: should we require from users to fill in certain information, or should we only rely on the data that can be extracted from evaluation logs generated by eval frameworks? For example, the organization name is rarely included in evaluation logs, so it would depend on the user to provide it. (A good option might be to leave the choice to the user and mark these fields as Optional).