Search-O1: remove hard-coded model length

[andreaschari]

No description provided.

[cmacdonald]

what problem does this solve? Also doesnt removing the length limit prevent some VLLM optimisation?

[andreaschari]

I found that there were cases where search-O1 would crash with:

ValueError: The decoder prompt (length 17848) is longer than the maximum model length of 16384. Make sure that max_model_len is no smaller than the number of text tokens.

So the choice was removing any constraint on the length or adding some truncation in generation_args with truncate_prompt_tokens. I guess not having any constraint by default made more sense and people can define their own backend if they wish to set a different max_model_len and truncate_prompt_tokens?

[ZhlSh17]

Unlike the other two (SearchR1 and R1-Searcher), SearchO1 is model-agnostic so we can't have a one-fits-all max_model_len. Removing it will leave it to the default based on the model's config. Users may manually create a backend to specify their desired max length and/or do truncation.

[cmacdonald]

Thanks both for the justification!