feat: add custom_metadata support to RecordBatch with IPC read/write

[rustyconover]

Which issue does this PR close?

• Closes Support per-batch custom_metadata on RecordBatch (IPC Message field) #9444.

What changes are included in this PR?

Add per-batch custom_metadata to RecordBatch, matching the custom_metadata field on the IPC Message flatbuffer envelope. This allows attaching per-batch metadata separate from schema-level metadata, bringing parity with PyArrow's write_batch(custom_metadata=...) API (available since PyArrow v11.0.0).

Changes:

• Add custom_metadata: HashMap<String, String> field to RecordBatch with custom_metadata(), custom_metadata_mut(), with_custom_metadata(), and into_parts_with_custom_metadata() accessors
• IPC writer: serialize custom_metadata to Message flatbuffer
• IPC reader: extract custom_metadata from Message at FileDecoder, StreamReader, and StreamDecoder call sites
• arrow-flight: extract and propagate custom_metadata in flight_data_to_arrow_batch
• arrow-select: preserve custom_metadata through filter_record_batch and take_record_batch
• Metadata preserved through slice(), project(), normalize(), with_schema(), and remove_column()
• PyArrow-generated test data for cross-language interop validation

Are these changes tested?

Yes there are tests in the PR.

Are there any user-facing changes?

There are no breaking changes.

Written with AI assistance; all changes reviewed by the author.

[alamb]

Thanks @rustyconover -- I see the custom_metadata field on the IPC messages and that makes sense to expose somehow

I am less sure about adding a new field to RecordBatch -- mostly as I am not sure about the implications of doing so (though your point that an empty HashMap has no allocations is a good one)

I mostly am thinking about our experience in other libraries trying to handle custom metadata on Fields where many kernels / processing don't preserve the metadata and it has been quite tough

I fear the same thing would happen to this field -- basically that it would not be used by most libraries but they would all pay the size cost on every RecordBatch

[alamb]

I wonder if @tustvold @jhorstmann @viirya or @kylebarron have any thoughts on this matter (adding custom metadata to every RecordBatch)

[rustyconover]

Hi @alamb I've patched arrow-go and arrow-js. And I have others with working patches into arrow-java. So its mostly about connectivity for me.

[rustyconover]

Hi @alamb, thanks for your review. I think the CI passed. Is there more you'd like me to do? Being a new contributor to arrow-rs I'm a bit unsure.

This is going to be the primary user to start: https://github.com/Query-farm/vgi-rpc-rust

[alamb]

Thanks @rustyconover -- there isn't anything I think you need to do

What i think is next needed is some buy in from other maintainers / stakeholders that changing RecordBatch is a reasonable thing to do given the tradeoffs