[C++] Rework CMake third-party logic

[asfimport]

Instead of the current approach we are taking with the *_HOME variables, we should use more CMake features and also give users of Arrow a more high-level control. This is going to be a rather lengthy issue with a lot of subtasks.

• Let the user decide on the top-level how dependencies should be handled. At the moment I can think of the following modes:
  • AUTO: Guess the packaging system we're running in, use this where possible, otherwise build the dependencies through the ExternalProject logic.
  • BUNDLED: Don't use any dependencies, build them all through ExternalProject
  • SYSTEM: Use CMake's find_package and find_library without any custom paths. If packages are on non-default locations, let the user indicate it from the outside using the *_ROOT variables.
  • CONDA: Same as SYSTEM but set all *_ROOT variables to ENV\{CONDA_PREFIX\}.
  • BREW: This uses SYSTEM but asks brew for some dependencies for their installation prefix.
• prefer dynamic linkage where possible
• Use pkg-config and *Targets.cmake files in projects that publish these
• Ensure that the necessary integration tests are in place (Fedora, Debian, Ubuntu, Alpine)
• Integration tests that Arrow's *Targets.cmake and arrow.pc work.

Reporter: Uwe Korn / @xhochy
Assignee: Uwe Korn / @xhochy

Related issues:

• [C++] cmake script assumes that double-conversion installs static libs (is a parent of)
• [C++] Use the CMake's standard find features (is a parent of)
• [C++/CI] Activate flight build in ci/docker_build_cpp.sh (is a parent of)
• [CI] Add gandiva to the docker-compose setup (contains)

PRs and other links:

• GitHub Pull Request #3688

_{Note: This issue was originally created as ARROW-4611. Please see the migration documentation for further details.}

[asfimport]

Kouhei Sutou / @kou:
Is ARROW-4383 related?
I'm implementing the AUTO mode for GFlags in ARROW-4599.

[asfimport]

Uwe Korn / @xhochy:
@kou  Yes, both are related. If there are no objections to this approach. I would like to go over all this and make a general cleanup. Currently we're just doing small cleanups but we're still facing a lot of issues with packagers so I'm happy to spent a bit more time to get the things cleaned up in general. I'll probably start first with some integration tests (read dockerfiles) so that I can ensure that I don't break existing installations.

 

[asfimport]

Kouhei Sutou / @kou:
OK.
Should I wait for your work? Or should I work with you?
I think that I should wait for your work. The general cleanup by multiple developers will cause conflicts.

[asfimport]

Uwe Korn / @xhochy:
Wait a week for me. When I'm then not done, I have to reevaluate how much time I can allocate.

[asfimport]

Kouhei Sutou / @kou:
OK!

[asfimport]

Uwe Korn / @xhochy:
@kou @wesm Please take a look at a1cd7f8 This is roughly what I image with the change done just for double-conversion.

[asfimport]

Kouhei Sutou / @kou:
I've commented to it.
I like this approach.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3688
#3688