ARROW 4649: [C++][CI][R] Add job that builds brew install apache-arrow --HEAD

[nealrichardson]

Note that this fails: see https://travis-ci.org/nealrichardson/arrow/jobs/545841211 for output.

I'm guessing this needs to pass in order for the upcoming release to be successful. cc @kou

[wesm]

-1 -- this isn't a nightly build, this is in Travis CI. Should we set this up in Crossbow / Buildbot instead?

[nealrichardson]

It's definitely not nightly. But it's done.

IMO our efforts are best spent right now in fixing the homebrew failure that this uncovers, which AFAICT should be release blocking. I will gladly work to migrate this off Travis after the release, but given what I know of how to extend buildbot/crossbow, adding a new job that does macOS and R to one of them is going to be some significant effort, and I'd rather spend that time elsewhere before 0.14 goes out.

[nealrichardson]

I'm also working on https://issues.apache.org/jira/browse/ARROW-5590 now so the net number of Travis jobs will not increase, in case that helps.

[kou]

I think that we should cover this by GLib test on macOS:

diff --git a/.travis.yml b/.travis.yml
index 36a2dccf..02aaa2dc 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -281,9 +281,9 @@ matrix:
     script:
     - if [ $ARROW_CI_C_GLIB_AFFECTED = "1" ]; then $TRAVIS_BUILD_DIR/ci/travis_script_c_glib.sh; fi
     - $TRAVIS_BUILD_DIR/ci/travis_script_ruby.sh
-  - name: "[OS X] C++ & GLib & Ruby w/ XCode 8.3 & homebrew"
+  - name: "[OS X] C++ & GLib & Ruby w/ XCode 10.2 & homebrew"
     compiler: clang
-    osx_image: xcode8.3
+    osx_image: xcode10.2
     os: osx
     env:
     - ARROW_TRAVIS_ORC=1

For GLib, we don't much care about old macOS and Xcode.

[nealrichardson]

Unless I'm missing something, I don't think the change you suggest is enough. The Travis job I added installs the C++ Arrow library from the head of master using brew install apache-arrow --HEAD. The job you proposed editing installs its dependencies with Homebrew but it doesn't build Arrow via Homebrew.

[nealrichardson]

Also, regardless of where this is tested, IMO the bigger issue is that brew install apache-arrow --HEAD fails: https://travis-ci.org/nealrichardson/arrow/jobs/545841211#L382-L396. This leads me to suspect that our 0.14 Homebrew release will also fail unless we fix this.

[kou]

We should not use Homebrew to test master on macOS. Because the apache-arrow formula is for the released Apache Arrow.
For example, it uses -DXXX_HOME https://github.com/Homebrew/homebrew-core/blob/master/Formula/apache-arrow.rb#L37-L43 but they aren't needed in master.
For example, we can't test new feature such as Flight with the apache-arrow Formula.

GLib is expected to be installed by Homebrew on macOS. So GLib CI on macOS should use the (almost) same configuration as the apache-arrow formula for the next release.

==> cmake ../cpp -DCMAKE_C_FLAGS_RELEASE=-DNDEBUG -DCMAKE_CXX_FLAGS_RELEASE=-DNDEBUG -DCMAKE_INSTALL_PREFIX=/usr/local/Cellar/apache-arrow/HEAD-02fd62d_1 -DCMAKE_BUILD_TYPE=Release -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_VERBOSE_MAKEFILE=ON -Wno-dev -DARROW_ORC=ON -DARROW_PARQUET=ON -DARROW_PLASMA=ON -DARROW_PROTOBUF_USE_SHARED=ON -DARROW_PYTHON=ON -DARROW_INSTALL_NAME_RPATH=OFF -DFLATBUFFERS_HOME=/usr/local/opt/flatbuffers -DLZ4_HOME=/usr/local/opt/lz4 -DPROTOBUF_HOME=/usr/local/opt/protobuf -DPYTHON_EXECUTABLE=/usr/local/opt/python/bin/python3 -DSNAPPY_HOME=/usr/local/opt/snappy -DTHRIFT_HOME=/usr/local/opt/thrift -DZSTD_HOME=/usr/local/opt/zstd
Last 15 lines from /Users/travis/Library/Logs/Homebrew/apache-arrow/01.cmake:
  2): Symbol not found: ___addtf3
    Referenced from: /usr/local/opt/gcc/lib/gcc/9/libquadmath.0.dylib
    Expected in: /usr/lib/libSystem.B.dylib
   in /usr/local/opt/gcc/lib/gcc/9/libquadmath.0.dylib
  
Call Stack (most recent call first):
  src/arrow/python/CMakeLists.txt:23 (find_package)
-- Configuring incomplete, errors occurred!

I think that this is a build environment problem.
I couldn't reproduce this on my local macOS and GLib CI: https://travis-ci.org/kou/arrow/jobs/547937677
Can you reproduce on your local macOS?

___addtf3 is included in the followings:

% for x in /usr/local/opt/gcc/lib/gcc/9/*.dylib; do nm $x | grep -q "T ___addtf3" && echo $x; done
/usr/local/opt/gcc/lib/gcc/9/libgcc_ext.10.4.dylib
/usr/local/opt/gcc/lib/gcc/9/libgcc_ext.10.5.dylib
/usr/local/opt/gcc/lib/gcc/9/libgcc_s.1.dylib
/usr/local/opt/gcc/lib/gcc/9/libgcc_s_ppc64.1.dylib
/usr/local/opt/gcc/lib/gcc/9/libgcc_s_x86_64.1.dylib

Normally, a library should use one of them instead of /usr/lib/libSystem.B.dylib to find ___addtf3. I think that language: r Travis CI environment changes system library search path.

[nealrichardson]

I guess I don't understand the point of having brew install apache-arrow --HEAD at all if we don't expect it to work. And I'm not following the logic for how we should have confidence that our Homebrew formula still works and what that means for our readiness for 0.14 release.

Locally, I didn't get that failure, but I got a different one. Wouldn't be surprised at all if language: r altered the search path, but it does support homebrew and you can use brew to install dependencies. Maybe the search path could be tweaked back to work (assuming it's supposed to work).

[kou]

I think that using the latest macOS and all libraries of Homebrew is enough for Homebrew support of the next release.
If we want to work more robustly, we should create a tap https://docs.brew.sh/Taps for Apache Arrow master that uses CMake options for master (0.14.0) not the released version (0.13.0). For example, the tap formula will use -DARROW_FLIGHT=ON but the formula in homebrew/homebrew-core doesn't use it.
Then we can send a pull request to homebrew/homebrew-core when we release the next release (0.14.0) as soon as possible.

Locally, I didn't get that failure, but I got a different one.

I commented a solution to the issue.

[wesm]

Since we make changes to the build system periodically, it can be hard to keep the formula working in between releases, so as long as we are testing master against Homebrew dependencies then we can fix up the formula at release time

[wesm]

Is there a way we can maintain and test the Homebrew formula as part of this repository? That might be one way to make things more civilized

[nealrichardson]

@wesm IIUC we can set up the homebrew tap and use that in our CI, but the tap formula itself would have to live in a separate repository.

I'm in favor of doing that or similar, so that the release process is essentially copying the formula from our tap repo into a PR to homebrew-core. Should we close this issue/PR and make a new one that is appropriately scoped?

[kou]

I've created #4641 that tests master against Homebrew dependencies.

[nealrichardson]

Closing this: looks like we'll cover Homebrew formula testing by other means.