[Java] Integration test breaks without zeroing out validity vectors

[asfimport]

This is discovered in #1290

I found one the integration test (nested) failed without zeroing out validity vectors before loading the array from json.

I have created three files to reproduce this:
(1) nested.json
(2) nested.good (zeroing out validity vector before reading)
(3) nested.bad (not zeroing out validity vector before reading)

(1) / (2) and (1) / (3) both pass Java integration test, however (1) / (3) fails C++ test - one of the validity vector in (3) doesn't seem to be read correctly.

I am not sure what the issue is because I cannot reproduce an error in Java. I am hoping maybe some one more familiar with C++ could take a look and give some insights what's the wrong with (3).

Reporter: Li Jin / @icexelloss

Original Issue Attachments:

• nested.bad
• nested.good
• nested.json

_{Note: This issue was originally created as ARROW-1779. Please see the migration documentation for further details.}

[asfimport]

Li Jin / @icexelloss:
cc @cpcloud @wesm

This is probably a Java issue but I am kind of stuck figuring out what's wrong because the error happens in C++ integration validation (Java producing, C++ consuming). I have the file to reproduce this

>>> good = pyarrow.RecordBatchFileReader("/Users/ljin/workspace/arrow/nested.good")
^[[A
>>> good_batch = good.get_record_batch(1)
>>> good_batch.column(1)
<pyarrow.lib.StructArray object at 0x10c2a3548>
[
  NA,
  {'f1': None, 'f2': 'BSZRpGI'},
  {'f1': None, 'f2': None},
  {'f1': None, 'f2': None},
  NA,
  NA,
  {'f1': None, 'f2': None},
  {'f1': None, 'f2': None},
  {'f1': 416507125, 'f2': None},
  NA
]

>>> bad = pyarrow.RecordBatchFileReader("/Users/ljin/workspace/arrow/nested.bad")
>>> bad_batch = bad.get_record_batch(1)
>>> bad_batch.column(1)
<pyarrow.lib.StructArray object at 0x10c0c6b88>
[
  {'f1': -1345581951, 'f2': None},
  {'f1': None, 'f2': 'BSZRpGI'},
  {'f1': None, 'f2': None},
  {'f1': None, 'f2': None},
  {'f1': -497925054, 'f2': 'E34Dqdr'},
  {'f1': 94270936, 'f2': '5aksGEG'},
  {'f1': None, 'f2': None},
  {'f1': None, 'f2': None},
  {'f1': 416507125, 'f2': None},
  {'f1': None, 'f2': None}
]

They are supposed to have the same data but the bad one doesn't read validity vector correctly. Can you guys help shed some light?

[asfimport]

Phillip Cloud / @cpcloud:
I'll take a look.

[asfimport]

Li Jin / @icexelloss:
@cpcloud helped me tracked down the issue in nullCount in vector metadata on the Java side. Thanks! I will fix this.

[asfimport]

Li Jin / @icexelloss:
Issue appears to be the same as in:
#1316 (comment)

[asfimport]

Li Jin / @icexelloss:
Resolved in PR #1316