Generate: fix generation with inputs_embeds when input_ids=None for llama and gemma

[njhill]

The changes in #29467 break generation with inputs_embeds when input_ids is None since they expect input_ids to be non-None even for the prefill forward without past_key_values.

@gante

[gante]

Hi @njhill 👋

Can you share an example of failure? We have a test for generation with inputs_embeds (which is passing on e.g. Llama), so our test suite is likely incomplete :)

[njhill]

Thanks @gante, this is specifically when using inputs_embeds and passing input_ids as None. I think both are passed to generate() in the current test. I'll update the PR title to clarify this.

[gante]

@njhill uhmmm the test checks that combination as well 🤔

            # input_ids is not a required input -- if we don't pass it, the newly generated tokens will be the same
            outputs_from_embeds_wo_ids = model.generate(
                inputs_embeds=inputs_embeds, max_new_tokens=20 - inputs_embeds.shape[1]
            )
            self.assertListEqual(
                outputs_from_embeds[:, inputs_embeds.shape[1] :].tolist(),
                outputs_from_embeds_wo_ids.tolist(),
            )

(you can run the test with py.test tests/models/llama/test_modeling_llama.py -k test_generate_from_inputs_embeds_decoder_only)

This means that there is probably something else going on, which could be interesting to pin down :)

[njhill]

@gante ah apologies for not looking at those closely enough, and thank you for the tip of how to run. Let me dig in deeper to see what's going on here.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.