[C++][FlightRPC] arrow-flight-benchmark performance degraded significantly (~10%) on the move to gRPC v1.46.3 from v1.35

[eyalbe12]

Describe the enhancement requested

Hi,

Not sure how to categorize this, as this is not really a bug, so I put it under the enhancement category.

When I moved from using ArrowFlight v8 to ArrowFlight v12 I noticed a degradation in performance. In order to verify it, I tried the arrow-flight-benchmark in localhost mode. I ran the following on Ubuntu 20.04 (8 cpu's and 32GB ram):

./release/arrow-flight-perf-server -port 31337 -transport grpc &
./release/arrow-flight-benchmark --server_host=localhost --server_port=31337   --records_per_stream=163333333 --num_streams=32 --num_threads=32 --records_per_batch=16384 --num_perf_runs=5

The V12 benchmark provided speed result which is ~10-15% less compared to V8 speed.

I traced the degradation directly to the following commit:

commit d89c0b4 (HEAD) ARROW-16729: [C++] Bump Abseil/gRPC dependency versions (#13315)

And more specifically, the upgrade of gRPC from v1.35 to v1.46.3 in arrow/cpp/thirdparty/versions.txt .

-ARROW_GRPC_BUILD_VERSION=v1.35.0
-ARROW_GRPC_BUILD_SHA256_CHECKSUM=27dd2fc5c9809ddcde8eb6fa1fa278a3486566dfc28335fca13eb8df8bd3b958
+ARROW_GRPC_BUILD_VERSION=v1.46.3
+ARROW_GRPC_BUILD_SHA256_CHECKSUM=d6cbf22cb5007af71b61c6be316a79397469c58c82a942552a62e708bce60964

Not sure how important this is to you, but I thought you should be aware of this. Do you have any clue why gRPC v1.46.3 causes such speed degradation in ArrowFlight?

Thanks for your hard work.

Component(s)

Benchmarking, C++, FlightRPC

[lidavidm]

Possibly, 1.46 had an unlisted changelog item that removed what had been the default polling engine, epollex, which has been observed to impact other benchmarks. (It's not quite clear when epoll1 became default; it appears to have been 1.45 or 1.46.)

We're not really in control here; pinning to an old gRPC version isn't desirable either.

[kou]

Our customization code https://github.com/apache/arrow/blob/main/cpp/src/arrow/flight/transport/grpc/customize_grpc.h for zero-copy may not be used with gRPC 1.46.

[lidavidm]

Hmm, at least for #35679 I observed that FlightDataDeserialize was still being called. (If it's not called, Flight will break entirely as there isn't really a fallback path.)

[kou]

Then my assumption is wrong. Sorry.

[lidavidm]

At the same time, 10% in the synthetic benchmark may or may not translate to real-world differences?

For two processes on the same machine, my hope is eventually the UCX backend can become production-ready (it's already much faster than gRPC for this case when using shared memory)

[assignUser]

The reason for the degraded performance is due to grpc update and there isn't really anything we can do unless that is fiex up stream (which seems unlikely looking at the PR), so I am going to close the issue. Feel free to reopen if there is anything I have missed.