[cache] fix HybridCache init when device is passed

[gante]

What does this PR do?

The following example, where we initialize the model with the device argument, results in cuda graph skips. It can be traced to a stray change in #37007, where the device initialization was incorrectly changed. This PR reverts it.

from transformers import pipeline
import torch

pipe = pipeline(
    "image-text-to-text",
    model="google/gemma-3-4b-it",
    device="cuda",
    torch_dtype=torch.bfloat16
)

output = pipe(text="Write a poem on Hugging Face, the company", max_new_tokens=10)
print(output)

[github-actions]

Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. The CI will be paused while the PR is in draft mode. When it is ready for review, please click the Ready for review button (at the bottom of the PR page). This will assign reviewers and trigger CI.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[zucchini-nlp]

Thanks!