[Rust] [BREAKING] Lists should take Field instead of DataType

[asfimport]

There is currently no way of tracking nested field metadata on lists. For example, if a list's children are nullable, there's no way of telling just by looking at the Field.

This causes problems with integration testing, and also affects Parquet roundtrips.

I propose the breaking change of [Large|FixedSize]List taking a Field instead of Box, as this will overcome this issue, and ensure that the Rust implementation passes integration tests.

CC @andygrove @jorgecarleitao @alamb  @jhorstmann ( @carols10cents  as this addresses some of the roundtrip failures).

I'm leaning towards this landing in 3.0.0, as I'd love for us to have completed or made significant traction on the Arrow Parquet writer (and reader), and integration testing, by then.

Reporter: Neville Dipale / @nevi-me
Assignee: Neville Dipale / @nevi-me

PRs and other links:

• GitHub Pull Request #8608

_{Note: This issue was originally created as ARROW-10261. Please see the migration documentation for further details.}

[asfimport]

Jorge Leitão / @jorgecarleitao:
Makes sense to me. :)

[asfimport]

Jörn Horstmann / @jhorstmann:
For my understanding, this is about metadata? So even if there are no null values in one batch or partition you want to mark the elements as potentially nullable?

[asfimport]

Neville Dipale / @nevi-me:
@jhorstmann  nullability should be determined by the overall field for consistency; as you could have 1000 batches of 1000 records, but only have say 5 nulls scattered around.

The main issue is that if I have a non-nullable list, which in turn has a nullable struct with various child fields with differing nullability; I won't know if the struct is nullable, because I lose that information when only taking the field.

Also, in the hypothetical case where the struct has some metadata of its own, it gets lost because we would only keep the DataType, and not other attributes such as dictionary or metadata (HashMap<String, String>).

Interestingly, looking at the CPP implementation, it looks like they still use List, but I can't see how they preserve the extra details that the Rust implementation is failing because of. @pitrou  any ideas?

[asfimport]

Andrew Lamb / @alamb:
[~nevi_me] – this proposal makes sense to me and I think it brings the Rust implementation closer to the C++ implementation, which is a good thing.

From my (cursory) reading of the C+, it appears source link that Lists in C+ use Field rather than DataType do describe each list item's type as well

[asfimport]

Antoine Pitrou / @pitrou:
Indeed, C++ uses List<Field>.

[asfimport]

Neville Dipale / @nevi-me:
Issue resolved by pull request 8608
#8608