[C++][FlightRPC] Support non-grpc data planes #30773
Description
The goal is to decouple FlightData payload transmission from gRPC and provide a convenient framework to support other accelerated data planes (ucx, dpdk, xdp, etc.).
As a PoC, I implemented a shared memory data plane for local rpc, the performance is great compared with current best (grpc over unix socket).
Tested throughput and latency on xeon gold 5218, DoGet, batch size 128K.
|streams|grpc over unix socket|shared memory data plane|
|
|-|-|-|-|
|1|3324 MB/s, 35 us|7045 MB/s, 16 us|
|
|2|6289 MB/s, 38 us|13311 MB/s, 17 us|
|
|4|10037 MB/s, 44 us|25012 MB/s, 17 us|
Reporter: Yibo Cai / @cyb70289
Assignee: David Li / @lidavidm
Related issues:
- [C++][FlightRPC] Decouple Arrow Flight RPC from GRPC (is duplicated by)
- [C++][FlightRPC] Refactor auth, middleware into the transport-agnostic layer (is related to)
- [C++][FlightRPC] Refactor remaining methods into transport-agnostic handlers (is related to)
- [C++][FlightRPC] Support alternative transports in the Flight benchmark (is related to)
- [C++][FlightRPC] Implement a UCX transport (is superceded by)
PRs and other links:
Note: This issue was originally created as ARROW-15282. Please see the migration documentation for further details.