[Rust] Implement IPC changes to catch up to 1.0.0 format

[asfimport]

There are a number of IPC changes and features which the Rust implementation has fallen behind on. It's effectively using the legacy format that was released in 0.14.x.

Some that I encountered are:

• change padding from 4 bytes to 8 bytes (along with the padding algorithm)

• add an IPC writer option to support the legacy format and updated format

• add error handling for the different metadata versions, we should support v4+ so it's an oversight to not explicitly return errors if unsupported versions are read

  Some of the work already has Jiras open (e.g. body compression), I'll find them and mark them as related to this.

  I'm tight for spare time, but I'll try work on this before the next release (along with the Parquet writer)

Reporter: Neville Dipale / @nevi-me
Assignee: Neville Dipale / @nevi-me

Subtasks:

• [Rust] Update checked-in flatbuffer files
• [Rust] Implement changes to ensure flatbuffer alignment
• [Rust] Allow configuring flight IPC with IpcWriteOptions
• [Rust] Support reading and writing V5 of IPC metadata

Related issues:

• [Rust] Use feature enum (incorporates)
• [Rust] Create implementation of IPC RecordBatch body buffer compression from ARROW-300 (is related to)

PRs and other links:

• GitHub Pull Request #9011
• GitHub Pull Request #9126

_{Note: This issue was originally created as ARROW-9777. Please see the migration documentation for further details.}

[asfimport]

Jorge Leitão / @jorgecarleitao:
[~nevi_me], this is marked as blocking for 3. Are we planning to address it or should we move it to the next release?

[asfimport]

Neville Dipale / @nevi-me:
Hi @jorgecarleitao, that's my intention yes. We've fallen behind a bit, and we might run a rist of hitting a compatibility issue in future, given that more users are starting to use the Rust implementation (specifically IPC).

I've done some work already on the V5 JIRA, reads work well, but some writes still create invalid outputs. If the next release is around Jan, I should be able to get to it.

[asfimport]

Jorge Leitão / @jorgecarleitao:
Ow, that is awesome. I will then assign it to you, then, and mark it as in progress

[asfimport]

Qingyou Meng / @mqy:
@jorgecarleitao [~nevi_me]

Please have a look at PR #9011

[asfimport]

Andy Grove / @andygrove:
We're close to cutting the 3.0.0 release. Is this really a blocker [~nevi_me] or can we release without this?

[asfimport]

Neville Dipale / @nevi-me:
Hi @andygrove, I've opened the PR that completes this, and I've added you as a reviewer.