Fix pipeline tests - don't set torch_dtype on non-torch pipelines

[amyeroberts]

What does this PR do?

Fixes errors introduced by the merge of #31342

Example failing run: https://app.circleci.com/pipelines/github/huggingface/transformers/97170/workflows/fd23720b-5884-42d5-8877-e69afcdeda34/jobs/1285703

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

Hi @amyeroberts Thanks for this.

• (more of a nit, see next point too) It makes a lot of sense not default to float32. However, we have many def get_test_pipeline(self, model, tokenizer, processor, torch_dtype="float32"), and I believe it will make even more sense if we change all those default values to None.

• (IMPORTANT) In the tests, we are doing/using

    def get_test_pipeline(self, model, tokenizer, processor, torch_dtype="float32"):
        classifier = ZeroShotClassificationPipeline(
            model=model, tokenizer=tokenizer, candidate_labels=["polics", "health"], torch_dtype=torch_dtype
        )

But if we look the definition of class Pipeline, torch_dtype is never used. It's actually defined a property with

    @property
    def torch_dtype(self) -> Optional["torch.dtype"]:
        """
        Torch dtype of the model (if it's Pytorch model), `None` otherwise.
        """
        return getattr(self.model, "dtype", None)

So we are not actually testing properly with fp32/fp16 as we expect. (I have to verify to make sure). This is important to address, either in this PR or another.

• And the failing is when pipeline(...., torch_dtype) is used. I fixed it in Avoid failure TFBlipModelTest::test_pipeline_image_to_text #31827.

[ydshieh]

yes the changes (themselves) make more sense. But see my overall comments.

[ydshieh]

I confirm the 2nd point in the above comment: we are not testing the torch_dtype properly, by

• running : python3 -m pytest -v tests/models/blip/test_modeling_blip.py::BlipModelTest::test_pipeline_image_to_text_fp16
• breakpoint around: tests/pipelines/test_pipelines_image_to_text.py line 54, after pipe defined.

It shows

(Pdb) pipe.model.dtype
torch.float32

More details

root@21530a8f5c8d:/transformers# python3 -m pytest -v tests/models/blip/test_modeling_blip.py::BlipModelTest::test_pipeline_image_to_text_fp16

============================================================================= test session starts ==============================================================================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase(PosixPath('/transformers/.hypothesis/examples'))
rootdir: /transformers
configfile: pyproject.toml
plugins: typeguard-4.2.1, hydra-core-1.3.2, xdist-3.6.1, rich-0.1.1, dash-2.17.0, timeout-2.3.1, hypothesis-6.102.6
collected 1 item                                                                                                                                                               

tests/models/blip/test_modeling_blip.py::BlipModelTest::test_pipeline_image_to_text_fp16 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /transformers/tests/pipelines/test_pipelines_image_to_text.py(54)get_test_pipeline()
-> Image.open("./tests/fixtures/tests_samples/COCO/000000039769.png"),
(Pdb) pipe
<transformers.pipelines.image_to_text.ImageToTextPipeline object at 0x7f0efc188fa0>
(Pdb) pipe.model.dtype
torch.float32
(Pdb)