Document Question Answering Pipeline fails due to array with an inhomogeneous shape

[tillschander]

System Info

transformers==4.48.1

Who can help?

Maybe @amyeroberts ?

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import pipeline

document_qa = pipeline("document-question-answering", model="impira/layoutlm-document-qa")
word_boxes = [["Britta",[100,157,129,163]],["Beispiel",[134,157,176,165]],["Sanit\u00e4r",[181,157,218,163]],["Herrn",[99,177,143,185]],["Emil",[99,191,133,199]],["Exempel",[138,191,200,201]],["Donaustrasse",[99,205,195,213]],["46",[200,205,216,213]],["11300",[100,219,140,227]],["Berlin",[146,219,190,227]],["-",[223,162,226,162]],["Beispielstrasse",[230,157,309,165]],["12",[316,157,326,163]],["-",[331,162,334,162]],["80888",[339,158,371,163]],["Berlin",[376,157,408,163]],["So",[643,172,659,180]],["erreichen",[664,172,731,180]],["Sie",[737,172,757,180]],["uns",[762,175,786,180]],["Internet",[642,187,694,201]],["www",[737,191,772,196]],["britta-sanitaer.de",[776,188,890,196]],["E-Mail",[643,200,689,214]],["britta.beispiel@",[737,202,842,212]],["gmx.net",[845,203,898,212]],["Telefon",[642,214,693,228]],["030",[738,216,762,224]],["\/",[767,216,771,224]],["999666",[776,216,824,224]],["Fax",[643,230,667,238]],["030",[738,230,762,238]],["\/",[767,230,771,238]],["999777",[776,230,824,238]],["Mobil",[642,242,682,256]],["0179",[738,244,770,252]],["\/",[775,244,779,252]],["999888",[784,244,833,252]],["Steuer-Nr",[643,272,708,280]],["122\/5678\/1234",[738,272,838,280]],["UStD",[643,286,685,294]],["DE12345678",[737,286,827,294]],["Datum",[643,315,688,323]],["30.11.2009",[737,315,811,323]],["Kunde",[643,328,687,336]],["14002",[738,328,778,336]],["Rechnung",[643,342,710,352]],["200910214",[737,342,811,350]],["Angebot:",[99,372,160,382]],["10154",[168,372,207,380]],["vom",[213,375,242,380]],["16.11.2009",[248,372,321,380]],["Objekt:",[100,386,148,396]],["10244",[156,386,195,394]],["Berlin,",[200,386,245,395]],["Charlottenstr.",[251,386,341,394]],["152",[348,386,370,394]],["Sehr",[100,415,130,423]],["geehrter",[135,415,189,425]],["Herr",[194,415,224,423]],["Exempel,",[229,415,293,425]],["nach",[99,442,131,450]],["Ausf\u00fchrung",[136,442,215,452]],["der",[220,442,241,450]],["Arbeiten",[246,442,303,450]],["entsprechend",[309,442,397,453]],["meinem",[401,442,455,450]],["og.",[460,445,479,453]],["Angebot",[485,442,542,452]],["erlaube",[547,442,595,450]],["ich",[601,442,620,450]],["mir",[625,442,647,450]],["wie",[652,442,676,450]],["folgt",[681,442,712,452]],["zu",[717,445,732,450]],["berechnen:",[737,442,808,450]],["Rechnung",[100,470,186,482]],["Nr.",[192,470,219,480]],["200910214",[226,470,316,480]],["Das",[538,473,563,481]],["Rechnungsdatum",[568,473,684,484]],["entspricht",[689,473,754,483]],["dem",[759,473,787,481]],["Leistungsdatum",[793,473,899,484]],["Pos",[99,508,123,516]],["Art-Nr.",[133,508,187,516]],["Bezeichnung",[256,506,347,520]],["Menge",[660,508,708,518]],["Einzelpreis",[724,508,803,518]],["Betrag",[851,508,899,518]],["1",[101,522,105,530]],["Austausch",[257,522,325,530]],["der",[331,522,351,530]],["defekten",[357,522,413,530]],["Zuleitung",[419,522,483,532]],["im",[489,522,505,530]],["2,0",[657,522,677,531]],["Std.",[683,522,708,530]],["30,00",[767,522,804,531]],["60,00",[862,522,898,531]],["WC",[256,536,282,544]],["des",[288,536,309,544]],["Ergeschosses",[314,536,402,546]],["2",[99,564,106,572]],["Materialkosten",[256,564,356,572]],["(Diverses",[362,564,424,574]],["Kleinmaterial)",[430,564,527,574]],["3,0",[658,564,677,573]],["Stk.",[683,564,708,572]],["24,56",[766,563,803,573]],["73,68",[862,564,898,573]],["Zahlbar",[100,606,151,614]],["innerhalb",[157,606,219,614]],["von",[224,609,248,614]],["7",[253,606,260,614]],["Tagen",[266,606,307,616]],["(bis",[312,606,336,616]],["zum",[342,609,370,614]],["07.12.2009)",[375,606,454,616]],["unter",[460,607,494,613]],["Abzug",[499,606,543,617]],["Rechnungsbetrag",[600,604,725,618]],["133,68",[814,606,858,616]],["EUR",[864,606,899,614]],["von",[100,623,124,628]],["3%",[129,620,150,628]],["Skonto",[156,620,203,628]],["(Zahlungsbetrag",[208,620,317,630]],["=",[322,624,330,627]],["129,67",[337,620,380,629]],["EUR).",[385,620,427,630]],["Bis",[433,620,454,628]],["zum",[460,623,487,628]],["14.12.2009",[101,634,174,642]],["ohne",[180,634,212,642]],["Abzug.",[216,634,264,644]],["Umsatzsteuer",[99,672,190,680]],["wird",[195,672,225,680]],["nicht",[230,672,263,680]],["in",[269,672,281,680]],["Rechnung",[285,672,352,682]],["gestellt.",[358,672,409,682]],["Als",[415,672,437,680]],["sogenannter",[443,673,522,682]],["Kleinunternehmer",[527,672,648,680]],["i.",[654,672,661,680]],["S.",[667,672,679,680]],["von",[685,675,708,680]],["$",[715,672,721,682]],["19",[728,672,742,680]],["Abs.",[748,672,777,680]],["1",[784,672,788,680]],["UStG",[795,672,833,680]],["wird",[837,672,867,680]],["auf",[873,672,894,680]],["die",[100,685,119,693]],["Regelbesteuerung",[125,685,244,696]],["verzichtet.",[249,686,318,694]],["Vielen",[100,728,144,736]],["Dank",[149,728,185,736]],["f\u00fcr",[189,728,208,736]],["Ihren",[213,728,247,736]],["Auftrag!",[252,728,308,738]],["Ich",[99,742,120,750]],["bitte",[125,742,154,750]],["um",[159,745,180,750]],["\u00dcberweisung",[185,740,274,752]],["des",[279,742,300,750]],["Rechnungsbetrages",[306,742,435,752]],["innerhalb",[440,742,502,750]],["von",[100,758,124,763]],["14",[135,756,150,764]],["Tagen",[159,756,200,766]],["an",[205,758,219,763]],["die",[225,755,244,763]],["unten",[249,757,285,763]],["genannte",[292,757,351,766]],["Bankverbindung",[356,755,467,766]],["Mit",[99,784,123,792]],["freundlichen",[128,784,212,792]],["Gr\u00fc\u00dfen",[218,784,266,792]],["Britta",[99,826,137,834]],["Beispiel",[142,826,196,836]],["Seite",[840,881,872,889]],["1\/1",[879,881,897,889]],["Zahlungsempf\u00e4nger",[100,900,193,907]],["Bankverbindung",[99,910,177,917]],["aus",[100,922,115,925]],["dem",[119,920,138,925]],["Ausland",[142,920,180,925]],["Gesch\u00e4ftsf\u00fchrung",[100,936,190,943]],["Britta",[302,900,329,905]],["Beispiel",[331,900,368,907]],["Beispielbank,",[302,910,366,917]],["KTO",[369,910,393,915]],["0098765,",[397,910,440,916]],["BLZ",[443,910,465,915]],["88899900",[469,910,515,915]],["BIC",[302,920,321,925]],["asdfasdf,",[325,920,366,926]],["IBAN",[370,920,398,925]],["asdfasdf4848",[402,920,463,925]],["Britta",[302,932,329,946]],["Beispiel",[333,932,368,946]]]
document_qa(question="What is the invoice number?", word_boxes=word_boxes, image=None)

https://colab.research.google.com/drive/1Rk-a68zREdBBuYN8jVMKUcQUG73Me_6Z?usp=sharing

Expected behavior

I would expect that the pipeline works with any list of word boxes. But certain word boxes cause the document question answering pipeline to throw a ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

This error does not always occur. Only certain word box combinations seem to trigger it. For example in the reproduction above I was not able to pinpoint the error to any specific word box.

Also, it seems like this error was introduced with transformers version 4.43 as the example wont throw any errors when using version 4.42. Therefore I suspect that this could be the PR that introduced the bug: #32076

[Rocketknight1]

Hi @tillschander, I believe #32208 and #32076 both attempted to fix this issue, and were reviewed by different core maintainers, so we might have missed the conflict. Let me test reverting #32076 and keeping #32208

[Rocketknight1]

Hi @tillschander, we have a PR open at #35964. Please test it and tell me if it resolves your issue! You can install it with pip install --upgrade git+https://github.com/huggingface/transformers.git@no_p_mask_array

[tillschander]

Wow @Rocketknight1 I did not expect such a quick response! Nice, thank you. Your PR seems to fix the issue. I was not able to reproduce the error with it.

[Rocketknight1]

@tillschander no problem! PR merged, thanks for the bug report.