Fix xpu to cpu

[jiqing-feng]

This PR fixed moving model from xpu to cpu.
Error can be reproduced by the following codes:

import torch
from transformers import AutoModelForCausalLM
from peft import get_peft_model, LoraConfig

model = AutoModelForCausalLM.from_pretrained("facebook/opt-125m", load_in_8bit=True)
model.cpu()
weights_not_cpu = [name for name, p in model.named_parameters() if p.device != torch.device("cpu")]
lora_config = LoraConfig(use_dora=True)
peft_model = get_peft_model(model, lora_config)
print(peft_model)

Trace log:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, xpu:0 and cpu!

[Titus-von-Koeller]

LGTM, thanks!