feat: support map data type in lance format version 2.2

[xloya]

Close #3620.
Currently, Lance does not support the Map data type. Importing Lance from a data source that supports Map data type requires special handling, which incurs significant processing costs, even though this type is very common in other data sources.

This PR aligns with the Map data type in Arrow, implementing the Map logical data type. In actual encoder, it uses an Offsets Array + List<Struct<key, value>> approach, which is similar to List data type .
And in actual decoder, it will decode the struct array and offset infos to restruct to the Arrow MapArray.

[chatgpt-codex-connector]

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Credits must be used to enable repository wide code reviews.

[xloya]

@westonpace @jackye1995 @Xuanwo @eddyxu PTAL when you have time, thanks!

[codecov]

Codecov Report

❌ Patch coverage is 78.22014% with 93 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance-encoding/src/encoder.rs	55.55%	24 Missing ⚠️
rust/lance-encoding/src/encodings/logical/map.rs	85.03%	18 Missing and 4 partials ⚠️
rust/lance-encoding/src/decoder.rs	67.74%	8 Missing and 2 partials ⚠️
...ust/lance-encoding/src/encodings/logical/struct.rs	64.28%	6 Missing and 4 partials ⚠️
rust/lance-namespace/src/schema.rs	69.69%	9 Missing and 1 partial ⚠️
rust/lance-core/src/datatypes.rs	50.00%	7 Missing ⚠️
rust/lance-core/src/datatypes/schema.rs	22.22%	7 Missing ⚠️
rust/lance-core/src/datatypes/field.rs	97.05%	2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[Xuanwo]

Thank you for working on this!

Lance format 2.1 has been stabilized, so we can't add new data type support in its current form. Otherise older versions of Lance wouldn’t be able to read new Lance files.

We should add this feature in format version 2.2 instead.

[xloya]

Thank you for working on this!

Lance format 2.1 has been stabilized, so we can't add new data type support in its current form. Otherise older versions of Lance wouldn’t be able to read new Lance files.

We should add this feature in format version 2.2 instead.

@Xuanwo Thanks for your review! Currently, most of the logic for adding the Map data type is located in the Field, Encoder, and Decoder modules. These modules generally don't contain format version information. I'm unsure if it's appropriate to limit support for the Map data type to version 2.2+, since version 2.1+ uses the same logic for reading, writing, and encoding/decoding (different from version 2.0 and below). Do you have any suggestions on this?

[Xuanwo]

@Xuanwo Thanks for your review! Currently, most of the logic for adding the Map data type is located in the Field, Encoder, and Decoder modules. These modules generally don't contain format version information. I'm unsure if it's appropriate to limit support for the Map data type to version 2.2+, since version 2.1+ uses the same logic for reading, writing, and encoding/decoding (different from version 2.0 and below). Do you have any suggestions on this?

Take Lance v1.0.0 as an example. If we add the Map type, users might find that data written in Lance v1.0.0 is not readable by Lance v0.39.0, and that's what we want to avoid.

[Xuanwo]

Here some nits about this PR. Maybe @westonpace wanna take an other look.

[xloya]

Here some nits about this PR. Maybe @westonpace wanna take an other look.

@Xuanwo Addressed all the comments, PTAL again when you have time, thanks!

[xloya]

@Xuanwo @westonpace @jackye1995 Gentle pin for this, thanks!

[xloya]

@Xuanwo Address the comments, sorry to bother you again, please take another look when you have time, thanks!

[Xuanwo]

Thank you for working on this! Can you help update our docs as a follow-up?

[xloya]

Thank you for working on this! Can you help update our docs as a follow-up?

Sure, I’ll open a PR to update docs soon.

[westonpace]

This is really good! Sorry I took a while look at it. Thanks for the addition 🚀

[westonpace]

I have one small suggestion I have made here: #5513