fix bug for videomt model device mismatch

[kaixuanliu]

@ydshieh pls help review, thx!

[ydshieh]

since we have this change now, we don't need the

query_tokens.to(frame_hidden_states.device)

below.

It's no big deal, but let's remove it.

[kaixuanliu]

Thx for the advice. Done.

[ydshieh]

@kaixuanliu

Could you share which tests are failing without this PR, as well as the (full) error log you have for them, please 🙏 ? Thanks

[kaixuanliu]

Sure. Before this PR, when I use 4 cards to run test case like:
tests/models/videomt/test_modeling_videomt.py::VideomtForUniversalSegmentationIntegrationTest::test_instance_segmentation_inference , it will fail with error:

for frame_idx in range(num_frames):                                                                                                 frame_hidden_states = hidden_states[:, frame_idx]

            if propagated_query is None:
                query_tokens = self.query.weight[None, :, :].expand(batch_size, -1, -1)
            else:
>               query_tokens = self.query_updater(propagated_query) + self.query.weight[None, :, :].to(
                    frame_hidden_states.device
                )
E               RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:3 and cuda:2!

src/transformers/models/videomt/modeling_videomt.py:1181: RuntimeError

[ydshieh]

Thank you for the provided information. Indeed, this model's integration tests have

model_kwargs = {"device_map": "auto"}

which will lead this kind of issues. So the fix makes a lot of sense (even for user-facing, not just for CI).

[ydshieh]

run-slow: videomt

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/videomt"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	cb3e97d4	workflow commit (merge commit)
PR	0db70bb0	branch commit (from PR)
main	f6ff4ed8	base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

[ydshieh]

run-slow: videomt

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: videomt

[github-actions]

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/videomt"]
quantizations: []

[github-actions]

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	6a456549	workflow commit (merge commit)
PR	64fda37c	branch commit (from PR)
main	a9f5b3a8	base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !