LayoutLMv2Processor: ensure 1-to-1 mapping between images and samples in case of overflowing tokens

[garyhlai]

What does this PR do?

Fixes #13554

Problem re-summarized: when return_offsets_mapping is set to True, LayoutLMv2Processor would break up sequences that are too long into multiple input_ids sequences, causing a mismatch between input_ids (longer in length in the case of overflowing tokens) and images.

This fix would ensure the 1-to-1 mapping between the images and input_ids.

Reproducible Example: (The assertion at the end would fail without the fix, pass with the fix)

import transformers
from PIL import Image
from transformers import LayoutLMv2Processor
from datasets import Features, Sequence, ClassLabel, Value, Array2D, Array3D, load_dataset
import torch

datasets = load_dataset("nielsr/funsd")
labels = datasets['train'].features['ner_tags'].feature.names
id2label = {v: k for v, k in enumerate(labels)}
label2id = {k: v for v, k in enumerate(labels)}

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

def preprocess_data(examples):
  images = [Image.open(path).convert("RGB") for path in examples['image_path']]
  words = examples['words']
  boxes = examples['bboxes']
  word_labels = examples['ner_tags']
  encoded_inputs = processor(images, words, boxes=boxes, word_labels=word_labels,
                             padding="max_length", truncation=True,
                             return_overflowing_tokens=True,
                             stride=50,
                             return_offsets_mapping=True,
                             return_tensors="pt")
  return encoded_inputs

train_data = preprocess_data(datasets["train"])

# this assert would fail without this PR fix. 
assert len(train_data["image"]) == len(train_data["input_ids"])

Required Input from Reviewers

Right now, the LayoutLMv2Processor would return a list for encoded_inputs["image"], regardless of the value of return_tensors. If we want it to return a torch tensor in the case return_tensors=="pt", we have to torch.stack the list (and do similar thing to support "np" and "tf").

Should I implement this in get_overflowing_images? Or should I just leave the return type as list and just print a warning?

Who can review?

@NielsRogge @sgugger @LysandreJik

P.S.

The test_processor_case_1 in test_processor_layoutlmv2.py fails before this PR. I'd be happy to look at it as well but it's unrelated to this PR.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[sgugger]

LGTM but let's wait for @NielsRogge to have a look too!

[NielsRogge]

LGTM, thanks for improving!

[sgugger]

Are you sure you have the exact same version as black as is pinned in our setup? The CI check for style is passing on master, so none of the reformatting unrelated to the changes in your PR is necessary.
The easiest might be to revert your last commit once you have made sure of the version of black, as black doesn't undo the lines it adds.

[sgugger]

Thanks again for your contribution!

[timothyjlaurent]

Thanks for handling this @ghlai9665