[C++/Python] Split C++ and Python unit test Travis CI jobs, run all C++ tests (including Gandiva) together

[asfimport]

Our main C++/Python job is bumping up against the 50 minute limit lately, so it is time to do a little bit of reorganization

I suggest that we do this:

• Build and test all C++ code including Gandiva in a single job on Linux and macOS

• Run Python unit tests (but not the C++ tests) on Linux and macOS in a separate job

  Code coverage will need to get uploaded in the Linux jobs for both of these, so a little bit of surgery is required

Reporter: Wes McKinney / @wesm
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #3208

_{Note: This issue was originally created as ARROW-3803. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Note that C++ code coverage needs to be enabled when running the Python tests, too, as that covers some parts of the C++ source tree.

[asfimport]

Wes McKinney / @wesm:
Makes sense

[asfimport]

Antoine Pitrou / @pitrou:
I've started looking into this. One inefficiency is that we build C++ unit tests in the Python build, just so we can run arrow-python-test.

[asfimport]

Wes McKinney / @wesm:
Indeed, I opened https://issues.apache.org/jira/browse/ARROW-4010 about improving the scripts to support modular build targets. I think we need to be able to write ninja python and only build the libraries and arrow-python-test

[asfimport]

Wes McKinney / @wesm:
See also https://issues.apache.org/jira/browse/ARROW-3184, a pre-requisite

[asfimport]

Antoine Pitrou / @pitrou:
I'm not sure it's a good idea to complicate the CMake scripts. I wonder if arrow-python-test is really useful, in any case.

[asfimport]

Wes McKinney / @wesm:
@pitrou unless you are already far into this, since I've been working a lot on the build scripts and CMake stuff this week, I can go ahead and take this

[asfimport]

Antoine Pitrou / @pitrou:
I'm not very far indeed, so you may take this if you want.

[asfimport]

Antoine Pitrou / @pitrou:
By the way, Gandiva needs to be Valgrind-sanitized. I can take this part up.

[asfimport]

Francois Saint-Jacques / @fsaintjacques:
Can't we just add an extra label to python unit tests and target them accordingly with ctest?

[asfimport]

Wes McKinney / @wesm:
@fsaintjacques I've already addressed that particular issue in the other patches this week. The builds are faster now so this has grown somewhat less urgent since things aren't timing out as much now

Now the main issue is that we have the C++ unit tests split between two builds – we may have to wait until the valgrind failures in Gandiva are fixed cc @pravindra

[asfimport]

Pindikura Ravindra / @pravindra:
@shyambits2004 is fixing the valgrind errors - ARROW 3979. We'll try to get it done early next week.

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 3208
#3208