[C++] Static build with AWS SDK

[asfimport]

I can't find my way around the build system when using the S3 client.

It seems that only shared target is allowed when the S3 feature is ON. In the thirdparty toolchain, when printing:

FATAL_ERROR "FIXME: Building AWS C++ SDK from source will link with wrong libcrypto"

What is actually meant is that static build will not work, correct ? If it is the case, should libarrow.a be generated at all when S3 feature is on ?

What can be done to fix this ? What does it mean that the SDK links to the wrong libcrypto ? Is it fixable ? Or is their a way to have the static build but maintain a dynamic link to a shared version of the SDK ?

 

 

Reporter: Rémi Dettai / @rdettai

Related issues:

• [C++] Create and install static library containing all dependencies built by Arrow (is part of)

_{Note: This issue was originally created as ARROW-8565. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
As far as I remember, this means that the AWS SDK build procedure will by default compile its own version of OpenSSL.
I would say it's probably fixable, but you will have to find out how and submit a PR for it :-)

[asfimport]

Rémi Dettai / @rdettai:
Thanks ! With my mediocrity in understanding cmake, I'm definitively not the man for the job... ;) As a workaround, is it possible to build statically from arrow and only maintain a shared dependency on the SDK ?

[asfimport]

Antoine Pitrou / @pitrou:
The solution is to build and install the AWS SDK separately, or to use pre-built binaries. Hopefully CMake will be able to pick them up, if they are installed in the right place.

[asfimport]

Rémi Dettai / @rdettai:
That's what I am trying to do. I have a static build of the sdk that is correctly picked up (${AWSSDK_LINK_LIBRARIES} seems defined) but when i try to link to it in an example with

target_link_libraries(example PRIVATE parquet_static ${AWSSDK_LINK_LIBRARIES})

I keep getting awefull undefined reference to Aws::XXX errors :(

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Have you tried with a shared build of aws sdk?

[asfimport]

Rémi Dettai / @rdettai:
Yes, but I could not manage to make it work with the static build of arrow and I don't want to use the shared lib of arrow as it generates a set of binaries that is too large for my usecase (embeding into aws lambda with optimized coldstart)

[asfimport]

Rémi Dettai / @rdettai:
I finally managed to make a static build but it requires changing the root CMakeLists to append AWSSDK_LINK_LIBRARIES to ARROW_STATIC_LINK_LIBS. I'm not sure this is safe to do because I really don't understand how the Arrow and AWS SDK dependencies are competing.

In the end the static build is barely more compact than the shared one, because the C++ AWS SDK build is a whole adventure. I'm continuing my investigation to see if I can come up with something nicer, but I really lack some cmake expertise :)

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
I'm not sure if you're aware, but the AWS SDK supports selectively building components in the bundled library. The following should make a smaller build that supports S3.

cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DBUILD_ONLY="s3;core;config;transfer"

[asfimport]

Rémi Dettai / @rdettai:
Yes I know, BUILD_ONLY the SDK would take forever to build !

Now I understand the build system a bit better, I believe this issue is a subset of ARROW-7605 that aims at fixing static build in general. I'm closing this issue in favor of the more generic one !