[R] package development builds broken for Windows and MacOS

[jeroen]

Describe the bug, including details regarding any error messages, version, and platform.

If we try to build the R package on Windows or MacOS from a checkout, it fails at a point where it is supposed to download a suitable nightly build of libarrow:

remotes::install_github('apache/arrow/r')
# -- R CMD build -------------------------------------------
# -  preparing 'arrow': (392ms)
# -  checking DESCRIPTION meta-information
# -  cleaning src
# -  checking for LF line-endings in source and make files and shell scripts
# -  checking for empty or unneeded directories
# -  building 'arrow_12.0.0.9000.tar.gz'
# Installing package into ‘C:/Users/Jeroen/AppData/Local/R/win-library/4.3’
# (as ‘lib’ is unspecified)
# * installing *source* package 'arrow' ...
# ** using staged installation
# Warning message:
# In unzip("lib.zip", exdir = "windows") :
#   error 1 in extracting from zip file
...
# using C++17
# g++  -std=gnu++17 -I"C:/PROGRA~1/R/R-43~1.0/include" -DNDEBUG -I../windows//include -DARROW_STATIC -DPARQUET_STATIC -DARROW_DS_STATIC               -DARROW_ACERO_STATIC -DARROW_R_WITH_PARQUET -DARROW_R_WITH_ACERO               -DARROW_R_WITH_DATASET -DARROW_R_WITH_JSON -DARROW_R_WITH_S3 -DARROW_R_WITH_GCS -I'C:/Users/Jeroen/AppData/Local/R/win-library/4.3/cpp11/include'   -I"C:/rtools43/x86_64-w64-mingw32.static.posix/include"     -O2 -Wall  -mfpmath=sse -msse2 -mstackrealign  -c RTasks.cpp -o RTasks.o
# In file included from RTasks.cpp:18:
# ./r_task_group.h:20:10: fatal error: arrow/util/parallel.h: No such file or directory
#    20 | #include <arrow/util/parallel.h>
#       |          ^~~~~~~~~~~~~~~~~~~~~~~
# compilation terminated.

Same on MacOS.

I'm not sure if it is related, but the builds on https://nightlies.apache.org/arrow/r/libarrow/bin/ have not been updated for 6 weeks. But even before that it was not working, I think the R package script does not not find the correct URLs.

Component(s)

R

[paleolimbot]

I'm not sure that remotes::install_github() has ever correctly downloaded nightlies...it's on my TODO list to make this work on Windows (#35049), though, since it's incredibly time-consuming to test development versions on Windows without it. Neal just did #35147 which should fix remotes::install_github('apache/arrow/r', build = FALSE).

Either way we'll need the hosted nightly builds! 😬

[nealrichardson]

(Setting aside Windows), R CMD build && R CMD INSTALL won't work without some additional build prep to put a copy the C++ source inside the R package directory. R CMD INSTALL (aka remotes::install_github('apache/arrow/r', build = FALSE)) does work out of the box because the build script can look for the C++ source in the monorepo parent directory at ../cpp/src. This is all using our bundled build script, not autobrew on mac.

Our nightly packaging job has this pre-build step to copy in the C++ source, along with putting a copy of the homebrew/autobrew formula inside the package and pinning the C++ library version in the formula to the same git SHA as the R package, so that they are in sync: https://github.com/ursacomputing/crossbow/blob/nightly-packaging-2023-05-08-0-github-r-binary-packages/.github/workflows/crossbow.yml#L47-L86 After doing this, R CMD build && R CMD INSTALL will work.

Windows is different and we don't currently have a path that will call the mingw makepkg command to build the C++ libraries at R package installation time. Presumably it could be done, but I don't think there's much interest in it.

As @paleolimbot says, we could (just) download the latest nightly binary packages instead of building from source, and that will almost always work. That's probably the way to go on Windows; on mac/linux, there's probably an env var we'll want to expose to control that behavior--perhaps it's the fallback you get instead of *** Proceeding without libarrow (no local source), and you can opt-in to it even if you have the source available?

As for our nightly binary packaging workflow, I'm not sure what's going on. The job that builds them continues to report success (latest: https://github.com/ursacomputing/crossbow/actions/runs/4906238724/jobs/8760994170). I don't see where they are uploaded to the nightlies repository though, and I have a faint recollection that that's a separate process. @assignUser do you know what's up, and why we aren't seeing that the packages aren't being successfully uploaded?

[assignUser]

tools/winlibs.R tries to download a libarrow zip matching the version (12.0.0.900 here) but the nightlies are numbered consecutively (which is a recent'ish change from being date based) so we would need to beef up the script in that regard.

I will have a look at the upload @nealrichardson

[assignUser]

#35478 fixes the nightly issue, not sure why I failed to update that workflow when we updated the rest after the switch...

[paleolimbot]

The version of pkgbuild that includes r-lib/pkgbuild#157 isn't released yet, but we can migrate the c++ vendoring + pruning to bootstrap.R + add a line to DESCRIPTION, after which we don't need build = FALSE (when the required version of pkgbuild is installed).

[nealrichardson]

@paleolimbot very cool. That should help with the remotes::install_github() path. I think we still have a gap if someone is doing the R CMD stuff at the CLI (which we do in our CI scripts), but still that will allow us to consolidate that bootstrapping logic in one place, which is an improvement for sure.

[jeroen]

@paleolimbot while on-the-fly vendoring can be useful for some, it deviates the CI even further from the build process used for (CRAN) releases, which relies on 3rd parties (me) providing libarrow builds for win/mac that the R package can download.

I think the priority should be to make the arrow CI self sustaining and focus on getting such libarrow nightlies and releases built and published for the platforms that you want to support.

[nealrichardson]

@paleolimbot while on-the-fly vendoring can be useful for some, it deviates the CI even further from the build process used for (CRAN) releases, which relies on 3rd parties (me) providing libarrow builds for win/mac that the R package can download.

We vendor the C++ source before submitting to CRAN, so the .tar.gz source package one installs there has it included. What @paleolimbot was mentioning would make that pre-build step we do become a part of the remotes/devtools workflow, so it would be more like the CRAN scenario.

I think the priority should be to make the arrow CI self sustaining and focus on getting such libarrow nightlies and releases built and published for the platforms that you want to support.

For windows, we already build and host the static libraries for releases (e.g. https://apache.jfrog.io/ui/native/arrow/r/12.0.0/libarrow/bin/windows/), so we may be able to just drop the rwinlib path if that's helpful. Mac is more involved with the autobrew/bottle setup, but I'm sure we could find ways to make that more self-sustaining.

[jeroen]

We vendor the C++ source before submitting to CRAN, so the .tar.gz source package one installs there has it included. What @paleolimbot was mentioning would make that pre-build step we do become a part of the remotes/devtools workflow, so it would be more like the CRAN scenario.

OK that makes sense. How is the vendored libarrow code currently copied into the CRAN source package? Is this scripted somewhere I can reproduce it?

[nealrichardson]

It's currently make sync-cpp, though putting that in bootstrap.R like Dewey proposed would help so that you don't have to maintain ad-hoc if arrow logic.

For the nightly/on-demand autobrew build, we don't use that vendored source (though presumably there would be a way?). Instead we use a copy of the formula(e) where we pin to a git SHA, so that when we brew install --HEAD it builds the right C++ code. That looks like https://github.com/ursacomputing/crossbow/blob/nightly-packaging-2023-05-08-0-github-r-binary-packages/.github/workflows/crossbow.yml#L47-L86 (the workflow is templated).

[jeroen]

It's currently make sync-cpp, though putting that in bootstrap.R like Dewey proposed would help so that you don't have to maintain ad-hoc if arrow logic. For the nightly/on-demand autobrew build, we don't use that vendored source (though presumably there would be a way?).

OK thanks, this fixes the Linux build. If I try to build a copy of arrow with vendored source on Windows, it fails with:

   ===== ./arrow.Rcheck/00install.out =====
  * installing *source* package 'arrow' ...
  ** using staged installation
  trying URL 'https://nightlies.apache.org/arrow/r/libarrow/bin/windows/arrow-12.0.0.9000.zip'
  Error in download.file(sprintf(template, version), "lib.zip", quiet = quietly) : 
    cannot open URL 'https://nightlies.apache.org/arrow/r/libarrow/bin/windows/arrow-12.0.0.9000.zip'

I see the nightlies on https://nightlies.apache.org/arrow/r/libarrow/bin/windows/ have a different version scheme for example todays build is arrow-12.0.0.100000043.zip.

[nealrichardson]

Right, we don't do a bundled build on windows, the makepkg-mingw happens out of band. Pulling a nightly binary (doing the appropriate version matching of .9000 to latest) on Windows was what @paleolimbot was discussing doing in #35049.