Issue encountered
Some newest LLM models can only supported by the new version VLLM. Currently, lighteval only support VLLM >= 0.10.0 and < 0.10.2
Solution/Feature
I think the main reason to this issue is VLLM >= 0.10.2 use different interface for LLM inference (more specifically, llm.generate) . The VLLM >= 0.10.2 doesn't support the keyword argument prompt_token_ids in case1 and case2
We can see there are multiple interfaces of llm.generate in VLLM==0.10.1.1 in this page: Link, where also indicates prompt_token_ids will be deprecated.
However, in the VLLM==0.10.2, there is only one single interface in the official document: Link, which accept the inputs in PromptType. To my best knowledge, the inputs in _generate ref. function are assumed to be tokenized, so all we need to do is wrap the original inputs with TokensPrompt ref. which contains the attribute prompt_token_ids: list[int]
The solution
- Insert on line code before L418
inputs = [TokensPrompt(prompt_token_ids=input_ids) for input_ids in inputs]
-
Remove the keyword argument prompt_token_ids at L440 and L458
-
Update pyproject.toml, to remove <0.10.2 in vllm installation
Done.
Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.
Issue encountered
Some newest LLM models can only supported by the new version VLLM. Currently, lighteval only support VLLM >= 0.10.0 and < 0.10.2
Solution/Feature
I think the main reason to this issue is VLLM >= 0.10.2 use different interface for LLM inference (more specifically,
llm.generate) . The VLLM >= 0.10.2 doesn't support the keyword argumentprompt_token_idsin case1 and case2We can see there are multiple interfaces of
llm.generatein VLLM==0.10.1.1 in this page: Link, where also indicatesprompt_token_idswill be deprecated.However, in the VLLM==0.10.2, there is only one single interface in the official document: Link, which accept the inputs in
PromptType. To my best knowledge, theinputsin_generateref. function are assumed to be tokenized, so all we need to do is wrap the originalinputswithTokensPromptref. which contains the attributeprompt_token_ids: list[int]The solution
Remove the keyword argument
prompt_token_idsat L440 and L458Update pyproject.toml, to remove
<0.10.2in vllm installationDone.
Possible alternatives
A clear and concise description of any alternative solutions or features you've considered.