[R] Improve configure script for contributor experience

[paleolimbot]

Describe the enhancement requested

There are a few things that I've noticed over the last few months of working with/installing Arrow and the R package on various platforms. I think some of these may have come up in previous tickets and as I go through actually trying some of these I will attempt to locate them.

The main idea is that when a contributor does a git clone of Arrow and opens up the package in RStudio or VScode, the static nightly builds of Arrow built for the R package should be sufficient for a user to run devtools:test(). When I was debugging the "files are kept open too long on Windows" problem I used that strategy to avoid setting up a Windows Arrow build and it worked great! (Maybe this is even documented already...if it is, moving that documentation to a more prominent place may be helpful).

Another common thing I've run up against is that the configure script enables pkg-config by default. The build of Arrow resolved by pkg-config is almost never the right one...it's usually too old or has the wrong features enabled. At the very least we need to check the version of whatever gets resolved by pkg-config and maybe we should consider disabling it by default. This is a particular issue with homebrew.

Component(s)

R

[nealrichardson]

I think the homebrew issue is separate from pkg-config. The brew check is after the pkg-config path: https://github.com/apache/arrow/blob/main/r/configure#L128 and unlike the pkg-config block above it, there is no check/warning about version mismatch. We should probably only use the homebrew library if there is a version match. Probably also ok if we wanted to make homebrew apache-arrow opt-in via environment variable.

Possibly unrelated and just FYI, we do use pkg-config elsewhere (later) in configure to read the .pc files to know which other -l lib flags need to be added. This is used even in the bundled static build.

[paleolimbot]

When I'm going through this for real I will double check, but on my machine I have pkg-config on PATH, which a little too happily resolves to the homebrew install of Arrow. This was the issue Lionel had when attempting to do the dplyr PR.

There is an open ticket somewhere about using "arrow" vs "arrow-dataset" as the pkg-config name, too (arrow-dataset is more likely to resolve to the feature set we need).

[nealrichardson]

Sure, I believe you (and I'm not going to brew install apache-arrow and mess my env up to find out 😹). I don't think anything has ever been removed from the configure script, only added, so there's lots of if-elses in there and it's possible some aren't being reached often or ever. It's ripe for a (careful) refactor, for sure.

[nealrichardson]

Related: #35140

[nealrichardson]

I've dug into the configure script a bit in #35147 and have a few thoughts:

• We should remove the special brew case. Seems it would only be used if you have apache-arrow installed via brew but don't have pkg-config. If you want to use the brew libraries, either install pkg-config or specify ARROW_HOME=$(brew --prefix apache-arrow).
• If we use pkg-config to find libraries, we should do a strict version check like we do in other build scripts (allowing x.y.z.1 to match x.y.z but disallowing x.y.z.9000) and reject using the found system libraries if they don't match, rather than just warn. Set ARROW_USE_PKG_CONFIG=true to use them anyway.

These don't solve the developer experience from a checkout entirely, but they eliminate some guns pointed at our feet. nixlibs.R will download a nightly binary if you're on .9000, but that's just for linux, and even then there are issues: depending on how you install, it may download every time; if it doesn't download every time, we'll need to provide instruction on how to keep it up to date if you contribute again in the future.

On mac we don't host nightly libarrow binaries (bottles), so we can't just use them. The out of the box developer experience should just work, with no more brew conflicts, but it would be slow.

[paleolimbot]

All good points! For Mac I think it's OK that we don't have a binary as long as it doesn't try to build libarrow again every single time (and as long as the build "just works", which it should since we test it on CI all the time).

[nealrichardson]

All good points! For Mac I think it's OK that we don't have a binary as long as it doesn't try to build libarrow again every single time (and as long as the build "just works", which it should since we test it on CI all the time).

The autobrew script is very conservative/paranoid about ensuring success on CRAN machines, so it installs everything into a tempdir and even pulls a copy brew every time. But if you strip out that, I think it boils down to:

brew tap autobrew/cran
brew install --HEAD apache-arrow-static

which should almost always work, and if you point it at a copy of the formula from the checkout, should be guaranteed to work (because that's what we test nightly). For that matter though, you could just use the regular homebrew formula, without the autobrew tap, as long as we can guarantee that we don't have version mismatch. So either way, doing that version check more strictly is the first step.

[nealrichardson]

Another thought I had while 🚶 🐕 : we could detect that we're in a git checkout, and if so, instead of doing autobrew, do the bundled build. And when the bundled build happens in a git checkout, we could make it build into a (gitignored) directory inside the project. Then we would add that location to our places to look for arrow already on the system, which we'd use, assuming it passes the version check. So the first time, it would build, and the next time you installed the R package, it would pick up the C++ build and not have to redo it.

The downside to going farther down this direction is that we will need better tooling or at least instructions for what to do when this build gets out of sync, but even if the instruction is rm -rf, it would solve the problem of iterating on a single branch at one time. For anyone who is a repeat contributor who would run into the sync issues, we should probably be nudging them towards maintaining a proper C++ build anyway, so this is for the one-time fix use case.

[nealrichardson]

Well I've been thoroughly nerd-sniped 🤣 I'm hacking away at configure in #35147 now, I think I will stop short of the changes above about the casual developer experience, but it should hopefully be easier to add and reason about after I'm done rewriting.

[nealrichardson]

I left a TODO for this in #35147. It would be interesting to test out what the new/casual contributor experience is now with those changes--it's possible that the situation is not so bad anymore--though I don't personally have time to push farther on it right now.