[Generate] Allow custom config values in generate config

[vasqu]

As per title, followup to a series of PRs that address the new config logic

• Generation config boolean defaults #43000
• [Generation] Fix default overwrite for non-None defaults #42958
• 🚨 Generation config defaults are now None #42702

tl;dr: custom values in an uploaded generation config are ignored atm

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Ah, that is what you meant earlier. I assumed that generation_config at this point will already be self.generation_config

Btw, if a few lines above we create not generation_config = GenerationConfig() but generation_config = self.generation_config it will be much easier. I can't think of why it is a bad idea. WDYT?

[vasqu]

Nope, not the case, e.g.

from transformers import AutoModelForCausalLM, AutoTokenizer, GenerationConfig

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B-Instruct",
    device_map="auto",
    torch_dtype="auto",
    attn_implementation="flash_attention_2",
).eval()
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct")
tokenizer.pad_token = tokenizer.eos_token
inputs = tokenizer(
    ["Hello, how are you?", "is this life?"],
    padding=True,
    padding_side="left",
    return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=50, generation_config=GenerationConfig(do_sample=False))
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))

We will have different generation configs so it's not guaranteed.

[zucchini-nlp]

Indeed 🥲
I guess there is no cleaner option then

[zucchini-nlp]

Also, let's allow loading custom fields when running generation_config.from_pretrained. After the recent changes, there is no need for a _from_model_config private attr and we are guaranteed to get only generation attr at init time. So we could just save all kwargs at __init__ as attributes, criiw

[ebezzam]

thanks @vasqu and @zucchini-nlp for iterating! I tried these changes in my VibeVoice branch, and they work 👍

[zucchini-nlp]

Seems like the CI is flaky again 🥲

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43181&sha=cb4bda

[vasqu]

Finally got it