[Bug Report] torch 2.8.0 makes model generation go haywire

[hijohnnylin]

Describe the bug

• model.generate works fine on torch 2.7.1 but in 2.8.0 (on macos at least), it goes haywire

• workaround is to force version <=2.8.0 on macos

• macos 15.6.1

• python 3.11.8

I'll dig into this more later (eg test if it affects linux too) but just flagging this for now so it doesn't get lost.

System Info

• I have checked that there is no similar issue in the repo (required)

[gaabrielfranco]

I had the same problem. I was going crazy trying to find it. Forcing torch back to 2.7.1 also worked for me.

[hijohnnylin]

@gaabrielfranco were you on cuda or mps or cpu?

[gaabrielfranco]

@hijohnnylin I was using mps.

[noahrossi]

I ran into this exact problem as well and it seems to be an upstream issue w/ pytorch: pytorch/issues/161640 . It's fixed in pytorch 2.9.0, and the issue doesn't appear (at least for my reproduction) in 2.7 or 2.9.

[jlarson4]

I added a warning for any users that might attempt to run TransformerLens with Torch 2.8.0 on MPS which will now encourage them to either update to 2.9.0 or rollback their version to 2.7.1, whichever works best for them.