[Rust] [DataFusion] Query returns incorrect row count

[asfimport]

Creating a DataFrame using Pandas (0.25.1 + pyarrow 0.14.1) which includes NaN value as a last entry in the column results in wrong ResultBatch num_rows()

 

Reporter: Adam Lippai / @alippai
Assignee: Andy Grove / @andygrove

PRs and other links:

• GitHub Pull Request #5502
• GitHub Pull Request #5503
• GitHub Pull Request #9

_{Note: This issue was originally created as ARROW-6687. Please see the migration documentation for further details.}

[asfimport]

Adam Lippai / @alippai:
I've managed to reproduce it with simplified content:

df = pd.DataFrame([[np.nan]], columns=['a'])
df.to_parquet('out.parquet')

 

[asfimport]

Andy Grove / @andygrove:
OK, I found the issue, I think.

In datafusion/src/datasource/parquet.rs, change line 247 from:

 for i in 0..def_levels.len() {

to:

for i in 0..levels_read { 

and the test passes

[asfimport]

Andy Grove / @andygrove:
@alippai could you create a PR against parquet-testing to add the NaN file please

[asfimport]

Andy Grove / @andygrove:
Issue resolved by pull request 5503
#5503