[Integration] Intermittent (but frequent) flight integration failures with auth:basic_proto

[asfimport]

Link to discussion on list:
https://lists.apache.org/thread.html/r0dcdc2b6334e7f067a828634cf7584406ed859ff4d3fb622fef1bdd7%40%3Cdev.arrow.apache.org%3E

I noticed that the Rust/CPP integration tests are failing seemingly
intermittently on master (and on Rust PRs). The tests pass if they are re-run (enough)

There are several commits that the little red X meaning that CI didn't
pass on master https://github.com/apache/arrow/commits/master

Here are some Some example CI runs that are failing
https://github.com/apache/arrow/runs/1935673508
https://github.com/apache/arrow/runs/1926705212

Here is another example:
https://github.com/apache/arrow/pull/9359/checks?check_run_id=1941967422

Example failure:

==========================================================
Testing file auth:basic_proto
==========================================================
Traceback (most recent call last):
  File "/arrow/dev/archery/archery/integration/util.py", line 139, in run_cmd
    output = subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/opt/conda/envs/arrow/lib/python3.8/subprocess.py", line 411, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/opt/conda/envs/arrow/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/build/cpp/debug/flight-test-integration-client', '-host', 'localhost', '-port=33569', '-scenario', 'auth:basic_proto']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/arrow/dev/archery/archery/integration/runner.py", line 308, in _run_flight_test_case
    consumer.flight_request(port, **client_args)
  File "/arrow/dev/archery/archery/integration/tester_cpp.py", line 116, in flight_request
    run_cmd(cmd)
  File "/arrow/dev/archery/archery/integration/util.py", line 148, in run_cmd
    raise RuntimeError(sio.getvalue())
RuntimeError: Command failed: /build/cpp/debug/flight-test-integration-client -host localhost -port=33569 -scenario auth:basic_proto
With output:
--------------
-- Arrow Fatal Error --
Invalid: Expected UNAUTHENTICATED but got Unavailable

Reporter: Andrew Lamb / @alamb
Assignee: Andrew Lamb / @alamb

Related issues:

• [Rust] Intermittent Flight integration test failures (is related to)

PRs and other links:

• GitHub Pull Request #9583
• GitHub Pull Request #9593

_{Note: This issue was originally created as ARROW-11717. Please see the migration documentation for further details.}

[asfimport]

David Li / @lidavidm:
Looks like the Rust integration runner waits for the Rust server to be ready before allowing the test to continue, so it doesn't seem like it's an issue with the C++ client running before the Rust server runs. And I can't get this to reproduce on a fork, so I'll settle for adding additional debug logging for now.

[asfimport]

David Li / @lidavidm:
Issue resolved by pull request 9593
#9593

[asfimport]

David Li / @lidavidm:
I got another failure here: https://github.com/apache/arrow/pull/9616/checks?check_run_id=2015262892 

Are we sure serve() will ensure the server is ready once it returns? I'm not so familiar with tonic. We might just have to add retries into the C++ client.