feat/rfc/poc: Agnostic GPU

[kallewoof]

What does this PR do?

This PR adds a tiny "agnostic.gpu" utility that is meant to allow easy replacing of unnecessarily hard-coded vendor-specific code.

The code does not use torch.accelerator as it is still considered experimental, but feedback on that opinion is welcome.

The example provided patches the flash linear attention Fused RMS norm gate patch to not require CUDA. The upstream library claims to be platform agnostic, so in theory, this should work on other platforms as well. This is a POC/RFC only, and I haven't tested whether this works in practice, but that seems to be slightly irrelevant to what this PR is meant to provide (if FusedRMSNormGated is not, in fact, supported on e.g. XPU devices, that is an upstream issue, which does not affect the validity of this patch).

If something that does this is already present and I simply haven't found it, then I'd love to know about it. If there are other reasons that I'm not seeing as for why we need to insist on hard coded cuda checks for a seemingly platform agnostic feature, I'd love to hear that as well.

Code Agent Policy

The code was 90% human-written*. The idea was solidified/sanity-checked through chats with DeepSeek.

(* exception being the "is available" part, which was derived from a snippet DeepSeek threw at me while chatting:

    return (torch.cuda.is_available() or
            (hasattr(torch, 'xpu') and torch.xpu.is_available()) or
            (hasattr(torch.backends, 'mps') and torch.backends.mps.is_available()))

)

• I confirm that this is not a pure code agent PR.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

These seem relevant:

• AMD ROCm: @ivarflakstad
• Intel XPU: @IlyasMoutawwakil
• Ascend NPU: @ivarflakstad

[github-actions]

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=45235&sha=0ff77a

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: olmo_hybrid, qwen3_5, qwen3_5_moe, qwen3_next

[Rocketknight1]

I think we probably just want to remove those device lines entirely, right? Usually the device is set via context or a to() operation

[kallewoof]

I think we probably just want to remove those device lines entirely, right? Usually the device is set via context or a to() operation

That's possible, yeah. There are only 6 occurrences of torch.cuda.current_device() apart from these.

There are, however, 422 torch.cuda. occurrences throughout the code base. It's hard to say what portion of those are CUDA-only, though.

[Rocketknight1]

We can work on the others over time 😅. I'd definitely be curious to know if we could get rid of torch.cuda.current_device() for now, though! Piecemeal engineering, as Popper would say.

[kallewoof]

All right, I'm gonna close this for now, but I may be back with a more suitable example if I get around to finding one. Thanks for the feedback.