Fix missing video inputs for PerceptionLM.

[shuminghu]

Critical: Fixes missing video input for PerceptionLM (accidentally removed in PR)

Minor: Add support for vanilla image that only has C,H,W dims but not tiles dim.
This is non-default image shapes used in PLM but it's useful in demos and low-resoure devices.
e.g., in just added "PLM Simple Fine-tuning Example" under
https://huggingface.co/facebook/Perception-LM-1B#plm-usage

[molbap]

LGTM for the fix, cc @zucchini-nlp who made the initial change!
For the non-standard image inputs, OK but would be better with a test that goes with it

[zucchini-nlp]

Oke, thanks! I think we need to standardize output shapes from the image processor to be consistent though

Maybe we can always return 5D pixels or already flattened 4D pixels? Whichever way looks good, we have models doing both options

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[shuminghu]

@zucchini-nlp The reason shape unification is done in models rather than image_processing is that i noticed in training model sees a different input shape than in eval/inference.

• In training with per_device_train_batch_size=8, when image processor output 4D: processed_images.shape: torch.Size([1, 3, 448, 448]), model input is 5D: pixel_values.shape: torch.Size([8, 1, 3, 448, 448]). , when image processor output is 5D, model input is 6D and will error out.
• In inference, both are 5D, processed_images.shape: torch.Size([1, 37, 3, 448, 448]), pixel_values.shape: torch.Size([1, 37, 3, 448, 448])

[shuminghu]

@zucchini-nlp Let me split the PR and merge the more urgent fix first?

[github-actions]

[For maintainers] Suggested jobs to run (before merge)

run-slow: perception_lm

[zucchini-nlp]

Yeah, lets merge this one first and add it in the next patch release

[shuminghu]

@zucchini-nlp My bad. Just realized this from collate_fn in my training script ( I added one dimension)

    pixel_values = torch.stack(
        [inst["pixel_values"] for inst in instances], dim=0
    )

Let me open another PR for this simple fix for image_preprocessor and update corresponding training script in model card.

The reason shape unification is done in models rather than image_processing is that i noticed in training model sees a different input shape than in eval/inference.

In training with per_device_train_batch_size=8, when image processor output 4D: processed_images.shape: torch.Size([1, 3, 448, 448]), model input is 5D: pixel_values.shape: torch.Size([8, 1, 3, 448, 448]). , when image processor output is 5D, model input is 6D and will error out.
In inference, both are 5D, processed_images.shape: torch.Size([1, 37, 3, 448, 448]), pixel_values.shape: torch.Size([1, 37, 3, 448, 448])