[C++][CI] Flight tests fail running on some configurations

[pitrou]

Describe the bug, including details regarding any error messages, version, and platform.

For example on Ubuntu 22.04:
https://github.com/ursacomputing/crossbow/actions/runs/5799341398/job/15719127299#step:6:4848

or Debian 11:
https://github.com/ursacomputing/crossbow/actions/runs/5799341448/job/15719127520

Symptoms are the same:

 70/107 Test  #71: arrow-flight-internals-test ...............***Failed    0.12 sec
Running arrow-flight-internals-test, redirecting output into /build/cpp/build/test-logs/arrow-flight-internals-test.txt (attempt 1/1)
/build/cpp/debug/arrow-flight-internals-test: error while loading shared libraries: libgmockd.so.1.11.0: cannot open shared object file: No such file or directory
/build/cpp/src/arrow/flight

        Start  72: arrow-flight-test
 71/107 Test  #72: arrow-flight-test .........................***Failed    0.12 sec
Running arrow-flight-test, redirecting output into /build/cpp/build/test-logs/arrow-flight-test.txt (attempt 1/1)
/build/cpp/debug/arrow-flight-test: error while loading shared libraries: libgmockd.so.1.11.0: cannot open shared object file: No such file or directory
/build/cpp/src/arrow/flight

Component(s)

C++, Continuous Integration, FlightRPC

[pitrou]

@kou @lidavidm

[pitrou]

This seems to be caused by the LD_LIBRARY_PATH setting in ci/scripts/cpp_test.sh. If I disable LD_LIBRARY_PATH, the tests run fine.

[kou]

It seems that 37cb592 / GH-36956 is related.

[lidavidm]

Hmm, I guess we have to also list GMock here? https://github.com/apache/arrow/blob/main/cpp/src/arrow/flight/CMakeLists.txt#L255

[kou]

It seems that it doesn't solve this problem.

The link command line for arrow-flight-test has googletest_ep-prefix/lib/libgmockd.so but it's not linked to arrow-flight-test directly. (libgtestd.so and libgtest_maind.so are linked directly.)

$ objdump -p build/cpp/debug/arrow-flight-test | grep NEEDED
  NEEDED               libarrow_flight_testing.so.1300
  NEEDED               libarrow_testing.so.1300
  NEEDED               libgtest_maind.so.1.11.0
  NEEDED               libarrow_flight.so.1300
  NEEDED               libgrpc.so.10
  NEEDED               libarrow.so.1300
  NEEDED               libgtestd.so.1.11.0
  NEEDED               libstdc++.so.6
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.6
  NEEDED               ld-linux-x86-64.so.2

arrow-flight-test has rpath to bundled GoogleTest:

$ objdump -p build/cpp/debug/arrow-flight-test | grep RUNPATH
  RUNPATH              /build/cpp/debug:/build/cpp/googletest_ep-prefix/lib

So directed linked libgtestd.so and libgtest_maind.so can be found but libgmockd.so can't:

$ ldd build/cpp/debug/arrow-flight-test | grep libg
	libgtest_maind.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0 (0x00007fc833eb2000)
	libgrpc.so.10 => /lib/x86_64-linux-gnu/libgrpc.so.10 (0x00007fc8337dc000)
	libgtestd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0 (0x00007fc82e1f8000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007fc82dfac000)
	libgmockd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgmockd.so.1.11.0 (0x00007fc82dd06000)
	libgrpc++.so.1 => /lib/x86_64-linux-gnu/libgrpc++.so.1 (0x00007fc82dba7000)
	libgpr.so.10 => /lib/x86_64-linux-gnu/libgpr.so.10 (0x00007fc82d054000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007fc82ceba000)
	libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007fc82caae000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007fc82c99e000)
$ LD_LIBRARY_PATH=${ARROW_HOME}/${CMAKE_INSTALL_LIBDIR:-lib}:${LD_LIBRARY_PATH} ldd build/cpp/debug/arrow-flight-test | grep libg
	libgtest_maind.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0 (0x00007f4fe6589000)
	libgrpc.so.10 => /lib/x86_64-linux-gnu/libgrpc.so.10 (0x00007f4fe5eb3000)
	libgtestd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0 (0x00007f4fe08cf000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f4fe0683000)
	libgmockd.so.1.11.0 => not found
	libgrpc++.so.1 => /lib/x86_64-linux-gnu/libgrpc++.so.1 (0x00007f4fe02da000)
	libgpr.so.10 => /lib/x86_64-linux-gnu/libgpr.so.10 (0x00007f4fdf787000)
	libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f4fdf5ed000)
	libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f4fdf1e1000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f4fdf0d1000)

libgmocked.so is linked implicitly via libarrow_flight_testing.so:

$ lddtree build/cpp/debug/arrow-flight-test
arrow-flight-test => build/cpp/debug/arrow-flight-test (interpreter => /lib64/ld-linux-x86-64.so.2)
    libarrow_flight_testing.so.1300 => /build/cpp/debug/libarrow_flight_testing.so.1300
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_flight_testing.so.1300
        libboost_filesystem.so.1.74.0 => /lib/x86_64-linux-gnu/libboost_filesystem.so.1.74.0
        libgmockd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgmockd.so.1.11.0
    libarrow_testing.so.1300 => /build/cpp/debug/libarrow_testing.so.1300
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_testing.so.1300
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6
    libgtest_maind.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtest_maind.so.1.11.0
    libarrow_flight.so.1300 => /build/cpp/debug/libarrow_flight.so.1300
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_flight.so.1300
        libgrpc++.so.1 => /lib/x86_64-linux-gnu/libgrpc++.so.1
            libabsl_strings.so.20210324 => /lib/x86_64-linux-gnu/libabsl_strings.so.20210324
                libabsl_strings_internal.so.20210324 => /lib/x86_64-linux-gnu/libabsl_strings_internal.so.20210324
                libabsl_int128.so.20210324 => /lib/x86_64-linux-gnu/libabsl_int128.so.20210324
                libabsl_throw_delegate.so.20210324 => /lib/x86_64-linux-gnu/libabsl_throw_delegate.so.20210324
                libabsl_raw_logging_internal.so.20210324 => /lib/x86_64-linux-gnu/libabsl_raw_logging_internal.so.20210324
            libgpr.so.10 => /lib/x86_64-linux-gnu/libgpr.so.10
                libabsl_str_format_internal.so.20210324 => /lib/x86_64-linux-gnu/libabsl_str_format_internal.so.20210324
scanelf: rpath_security_checks(): Security problem NULL DT_RUNPATH in /build/cpp/debug/libarrow_flight.so.1300
        libprotobuf.so.23 => /lib/x86_64-linux-gnu/libprotobuf.so.23
            libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1
    libgrpc.so.10 => /lib/x86_64-linux-gnu/libgrpc.so.10
        libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3
        libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3
        libabsl_time.so.20210324 => /lib/x86_64-linux-gnu/libabsl_time.so.20210324
            libabsl_time_zone.so.20210324 => /lib/x86_64-linux-gnu/libabsl_time_zone.so.20210324
            libabsl_base.so.20210324 => /lib/x86_64-linux-gnu/libabsl_base.so.20210324
                libabsl_spinlock_wait.so.20210324 => /lib/x86_64-linux-gnu/libabsl_spinlock_wait.so.20210324
        libabsl_bad_optional_access.so.20210324 => /lib/x86_64-linux-gnu/libabsl_bad_optional_access.so.20210324
        libcares.so.2 => /lib/x86_64-linux-gnu/libcares.so.2
    libarrow.so.1300 => /build/cpp/debug/libarrow.so.1300
        libbrotlienc.so.1 => /lib/x86_64-linux-gnu/libbrotlienc.so.1
            libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1
        libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1
        libutf8proc.so.2 => /lib/x86_64-linux-gnu/libutf8proc.so.2
        libre2.so.9 => /lib/x86_64-linux-gnu/libre2.so.9
        libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0
        liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1
        libsnappy.so.1 => /lib/x86_64-linux-gnu/libsnappy.so.1
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1
        libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4
            libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14
            libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0
                libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2
            librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1
                libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30
                    libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0
                        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8
                    libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6
                libhogweed.so.6 => /lib/x86_64-linux-gnu/libhogweed.so.6
                libnettle.so.8 => /lib/x86_64-linux-gnu/libnettle.so.8
                libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10
            libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4
            libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5
            libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2
                libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3
                    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
                    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
                libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3
                libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
                libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0
            libldap-2.5.so.0 => /lib/x86_64-linux-gnu/libldap-2.5.so.0
                libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2
            liblber-2.5.so.0 => /lib/x86_64-linux-gnu/liblber-2.5.so.0
    libgtestd.so.1.11.0 => /build/cpp/googletest_ep-prefix/lib/libgtestd.so.1.11.0
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6
    ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2

[kou]

The link command line:

/usr/lib/ccache/c++ \
-Wno-noexcept-type \
-fdiagnostics-color=always \
-Wall \
-Wno-conversion \
-Wno-sign-conversion \
-Wunused-result \
-Wdate-time \
-fno-semantic-interposition \
-msse4.2 \
-g \
-Werror \
-O0 \
-ggdb \
src/arrow/flight/CMakeFiles/arrow-flight-test.dir/flight_test.cc.o \
-o \
debug/arrow-flight-test \
-Wl,-rpath,/build/cpp/debug:/build/cpp/googletest_ep-prefix/lib \
debug/libarrow_flight_testing.so.1300.0.0 \
debug/libarrow_testing.so.1300.0.0 \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_storage.a \
/usr/lib/x86_64-linux-gnu/libcrypto.so \
/usr/lib/x86_64-linux-gnu/libssl.so \
/usr/lib/x86_64-linux-gnu/libbrotlienc.so \
/usr/lib/x86_64-linux-gnu/libbrotlidec.so \
/usr/lib/x86_64-linux-gnu/libbrotlicommon.so \
orc_ep-install/lib/liborc.a \
/usr/lib/x86_64-linux-gnu/libprotobuf.so \
awssdk_ep-install/lib/libaws-cpp-sdk-identity-management.a \
awssdk_ep-install/lib/libaws-cpp-sdk-sts.a \
awssdk_ep-install/lib/libaws-cpp-sdk-cognito-identity.a \
awssdk_ep-install/lib/libaws-cpp-sdk-s3.a \
awssdk_ep-install/lib/libaws-cpp-sdk-core.a \
awssdk_ep-install/lib/libaws-crt-cpp.a \
awssdk_ep-install/lib/libaws-c-s3.a \
awssdk_ep-install/lib/libaws-c-auth.a \
awssdk_ep-install/lib/libaws-c-mqtt.a \
awssdk_ep-install/lib/libaws-c-http.a \
awssdk_ep-install/lib/libaws-c-compression.a \
awssdk_ep-install/lib/libaws-c-sdkutils.a \
awssdk_ep-install/lib/libaws-c-event-stream.a \
awssdk_ep-install/lib/libaws-c-io.a \
awssdk_ep-install/lib/libaws-c-cal.a \
awssdk_ep-install/lib/libaws-checksums.a \
awssdk_ep-install/lib/libaws-c-common.a \
awssdk_ep-install/lib/libs2n.a \
opentelemetry_ep-install/lib/libopentelemetry_trace.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_ostream_span.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_otlp_http.a \
/usr/lib/x86_64-linux-gnu/libutf8proc.so \
/usr/lib/x86_64-linux-gnu/libre2.so \
-ldl \
googletest_ep-prefix/lib/libgtest_maind.so \
googletest_ep-prefix/lib/libgtestd.so \
googletest_ep-prefix/lib/libgmockd.so \
-ldl \
/usr/lib/x86_64-linux-gnu/libgrpc++.so \
debug/libarrow_flight.so.1300.0.0 \
/usr/lib/x86_64-linux-gnu/libgrpc++.so \
/usr/lib/x86_64-linux-gnu/libgrpc.so \
/usr/lib/x86_64-linux-gnu/libgpr.so \
debug/libarrow.so.1300.0.0 \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_storage.a \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_rest_internal.a \
google_cloud_cpp_ep-install/lib/libgoogle_cloud_cpp_common.a \
absl_ep-install/lib/libabsl_cord.a \
absl_ep-install/lib/libabsl_cordz_info.a \
absl_ep-install/lib/libabsl_cord_internal.a \
absl_ep-install/lib/libabsl_cordz_functions.a \
absl_ep-install/lib/libabsl_exponential_biased.a \
absl_ep-install/lib/libabsl_cordz_handle.a \
absl_ep-install/lib/libabsl_synchronization.a \
absl_ep-install/lib/libabsl_stacktrace.a \
absl_ep-install/lib/libabsl_graphcycles_internal.a \
absl_ep-install/lib/libabsl_symbolize.a \
absl_ep-install/lib/libabsl_debugging_internal.a \
absl_ep-install/lib/libabsl_malloc_internal.a \
absl_ep-install/lib/libabsl_demangle_internal.a \
absl_ep-install/lib/libabsl_bad_optional_access.a \
absl_ep-install/lib/libabsl_str_format_internal.a \
absl_ep-install/lib/libabsl_time.a \
absl_ep-install/lib/libabsl_strings.a \
absl_ep-install/lib/libabsl_strings_internal.a \
absl_ep-install/lib/libabsl_throw_delegate.a \
absl_ep-install/lib/libabsl_base.a \
absl_ep-install/lib/libabsl_spinlock_wait.a \
absl_ep-install/lib/libabsl_int128.a \
absl_ep-install/lib/libabsl_civil_time.a \
absl_ep-install/lib/libabsl_time_zone.a \
absl_ep-install/lib/libabsl_bad_variant_access.a \
absl_ep-install/lib/libabsl_raw_logging_internal.a \
absl_ep-install/lib/libabsl_log_severity.a \
crc32c_ep-install/lib/libcrc32c.a \
/usr/lib/x86_64-linux-gnu/libbrotlienc.so \
/usr/lib/x86_64-linux-gnu/libbrotlidec.so \
/usr/lib/x86_64-linux-gnu/libbrotlicommon.so \
orc_ep-install/lib/liborc.a \
/usr/lib/x86_64-linux-gnu/libz.so \
/usr/lib/x86_64-linux-gnu/liblz4.so \
awssdk_ep-install/lib/libaws-cpp-sdk-identity-management.a \
awssdk_ep-install/lib/libaws-cpp-sdk-sts.a \
awssdk_ep-install/lib/libaws-cpp-sdk-cognito-identity.a \
awssdk_ep-install/lib/libaws-cpp-sdk-s3.a \
awssdk_ep-install/lib/libaws-cpp-sdk-core.a \
awssdk_ep-install/lib/libaws-crt-cpp.a \
awssdk_ep-install/lib/libaws-c-s3.a \
awssdk_ep-install/lib/libaws-c-auth.a \
awssdk_ep-install/lib/libaws-c-mqtt.a \
awssdk_ep-install/lib/libaws-c-http.a \
awssdk_ep-install/lib/libaws-c-compression.a \
awssdk_ep-install/lib/libaws-c-sdkutils.a \
awssdk_ep-install/lib/libaws-c-event-stream.a \
awssdk_ep-install/lib/libaws-c-io.a \
awssdk_ep-install/lib/libaws-c-cal.a \
/usr/lib/x86_64-linux-gnu/libssl.so \
/usr/lib/x86_64-linux-gnu/libcrypto.so \
awssdk_ep-install/lib/libaws-checksums.a \
awssdk_ep-install/lib/libaws-c-common.a \
awssdk_ep-install/lib/libs2n.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_ostream_span.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_otlp_http.a \
opentelemetry_ep-install/lib/libopentelemetry_otlp_recordable.a \
opentelemetry_ep-install/lib/libopentelemetry_trace.a \
opentelemetry_ep-install/lib/libopentelemetry_resources.a \
opentelemetry_ep-install/lib/libopentelemetry_common.a \
opentelemetry_ep-install/lib/libopentelemetry_exporter_otlp_http_client.a \
opentelemetry_ep-install/lib/libopentelemetry_proto.a \
/usr/lib/x86_64-linux-gnu/libprotobuf.so \
opentelemetry_ep-install/lib/libopentelemetry_http_client_curl.a \
/usr/lib/x86_64-linux-gnu/libcurl.so \
/usr/lib/x86_64-linux-gnu/libutf8proc.so \
/usr/lib/x86_64-linux-gnu/libre2.so \
-ldl \
jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc_pic.a \
mimalloc_ep/src/mimalloc_ep/lib/mimalloc-2.0/libmimalloc-debug.a \
-lrt \
/usr/lib/x86_64-linux-gnu/libzstd.so \
/usr/lib/x86_64-linux-gnu/libsnappy.so.1.1.8 \
/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.74.0 \
/usr/lib/x86_64-linux-gnu/libboost_system.so.1.74.0