[R] CRAN packaging checklist for version 17.0.0

[jonkeane]

Describe the enhancement requested

Packaging checklist for CRAN release

For a high-level overview of the release process see the
Apache Arrow Release Management Guide.

cc @jonkeane @amoeba @nealrichardson @assignUser @paleolimbot for coordination on the various tasks to release

We will want to cherry pick:

• GH-43194: [R] R_existsVarInFrame isn't available earlier than R 4.2 #43243
• MINOR: [R] Add news bullet for mutate() enhancement in 17.0 #43189
• GH-43044: [R] So-called non-API entry points #43173
• GH-43153: [R] pull on a grouped query returns the wrong column #43172
• GH-43163: [R] Fix bindings in Math group generics #43162
• MINOR: [R] add back dplyr:: to left_join calls #43348
• GH-43349: [R] Fix altrep string columns from readr #43351
• GH-43467: [C++] Add support for the official LZ4 CMake package #43468
• GH-43228: [C++] Fix Abseil compile error on GCC 13 #43157
• MINOR: [R] Work around test failure in tidyquery revdep #43498
• MINOR: [R] switch on R_VERSION #43504
• GH-43627: [R] Fix summarize() performance regression (pushdown) #43649
• GH-43735: [R] AWS SDK fails to build on one of CRAN's M1 builders #43736

Before the release candidate is cut

• Create a GitHub issue entitled [R] CRAN packaging checklist for version X.X.X and copy this checklist to the issue.
• Review deprecated functions to advance their deprecation status, including removing preprocessor directives that no longer apply (search for ARROW_VERSION_MAJOR in r/src).
• Evaluate the status of any failing nightly tests and nightly packaging builds. These checks replicate most of the checks that CRAN runs, so we need them all to be passing or to understand that the failures may (though won't necessarily) result in a rejection from CRAN.
• Check current CRAN check results
• Ensure the contents of the README are accurate and up to date.
• Run urlchecker::url_check() on the R directory at the release candidate.
  commit. Ignore any errors with badges as they will be removed in the CRAN release branch.
• Polish NEWS but do not update version numbers (this is done automatically later). You can find commits by, for example, git log --oneline aa057d0..HEAD | grep "\[R\]"
• Run preliminary reverse dependency checks using archery docker run r-revdepcheck.
• For major releases, prepare tweet thread highlighting new features.

Wait for the release candidate to be cut:

After release candidate has been cut

• Create a CRAN-release branch from the release candidate commit

Prepare and check the .tar.gz that will be released to CRAN.

• git fetch upstream && git checkout release-X.X.X-rcXX && git clean -f -d
• Run make build. This copies Arrow C++ into tools/cpp, prunes some
  unnecessary components, and runs R CMD build to generate the source tarball.
  Because this will install the package, you will need to ensure that the version
  of Arrow C++ available to the configure script is the same as the version
  that is vendored into the R package (e.g., you may need to unset ARROW_HOME).
• devtools::check_built("arrow_X.X.X.tar.gz") locally
• Run reverse dependency checks using archery docker run r-revdepcheck.

Release vote

• Release vote passed!

Generate R package to submit to CRAN

• If the release candidate commit updated, rebase the CRAN release branch
  on that commit.
• Pick any commits that were made to main since the release commit that
  were needed to fix CRAN-related submission issues identified in the above
  steps.
• Remove badges from README.md
• Run urlchecker::url_check() on the R directory
• Create a PR entitled WIP: [R] Verify CRAN release-10.0.1-rc0. Add
  a comment @github-actions crossbow submit --group r to run all R crossbow
  jobs against the CRAN-specific release branch.
• Run Rscript tools/update-checksums.R <libarrow version> to download the checksums for the pre-compiled binaries from the ASF artifactory into the tools directory.
• Regenerate arrow_X.X.X.tar.gz (i.e., make build)

Ensure linux binary packages are available:

• Ensure linux binaries are available in the artifactory:
  https://apache.jfrog.io/ui/repos/tree/General/arrow/r

Check binary Arrow C++ distributions specific to the R package

• Upload the .tar.gz to win-builder (r-devel only)
  and confirm (with Nic, who will automatically receive an email about the results) that the check is clean.
  This step cannot be completed before Jeroen has put the binaries in the MinGW repository, i.e. here, here, and here.
• Upload the .tar.gz to MacBuilder
  and confirm that the check is clean
• Check install.packages("arrow_X.X.X.tar.gz") on Ubuntu and ensure that the
  hosted binaries are used
• devtools::check_built("arrow_X.X.X.tar.gz") locally one more time (for luck)

CRAN submission

• Upload arrow_X.X.X.tar.gz to the
  CRAN submit page
• Confirm the submission email

Wait for CRAN...

• Accepted!
• Tag the tip of the CRAN-specific release branch
• Add a new line to the matrix in the backwards compatability job
• (patch releases only) Update the package version in ci/scripts/PKGBUILD, dev/tasks/homebrew-formulae/autobrew/apache-arrow.rb, r/DESCRIPTION, and r/NEWS.md
• (CRAN-only releases) Rebuild news page with pkgdown::build_news() and submit a PR to the asf-site branch of the docs site with the contents of arrow/r/docs/news/index.html replacing the current contents of arrow-site/docs/r/news/index.html
• (CRAN-only releases) Bump the version number in r/pkgdown/assets/versions.json, and update this on the the asf-site branch of the docs site too.
• Update the packaging checklist template to reflect any new realities of the
  packaging process.
• Wait for CRAN-hosted binaries on the
  CRAN package page to reflect the
  new version
• Tweet!

Component(s)

R

Component(s)

R

[jonkeane]

@assignUser Did you use Rscript tools/update-checksums.R <libarrow version> in the past to generate the contents of 12ec842 ?

[assignUser]

Yes :)
Let me know if I can help with anything!

[amoeba]

I added the NEWS.md entry from #43189 to the release blog post PR.

[assignUser]

@jonkeane mac builder: https://mac.r-project.org/macbuilder/results/1721447405-da61deb72185bfbc/
win builder should reach you by mail, if that's clean I think only a revdep check is missing. @paleolimbot didn't you have a super quick script that doesn't take a million hours to run?

[jonkeane]

Thanks for sending those, Jacob.

Here's the winbuilder output: https://win-builder.r-project.org/SrOKYg8yT1lm/

https://win-builder.r-project.org/SrOKYg8yT1lm/00check.log

The only thing in there that looks worrying is

* checking R code for possible problems ... [33s] NOTE
mutate.ArrowTabular: no visible global function definition for
  'left_join'
mutate.Dataset: no visible global function definition for 'left_join'
mutate.RecordBatchReader: no visible global function definition for
  'left_join'
mutate.arrow_dplyr_query: no visible global function definition for
  'left_join'
Undefined global functions or variables:
  left_join
* checking Rd files ... OK

Which I suspect is from https://github.com/apache/arrow/pull/41576/files and can be dealt with by adding dplyr:: onto those calls again

[amoeba]

I brought in the changes from maint-17.0.0-r (see my main branch) and kicked off two recheck jobs (1, 2).

Edit: I need to tweak recheck a bit to run so the linked jobs failed right away. I'll update them once recheck is running.
Edit 2: Updated URLs.

[nealrichardson]

Which I suspect is from https://github.com/apache/arrow/pull/41576/files and can be dealt with by adding dplyr:: onto those calls again

Sorry! I thought I fixed those but I guess I added more. LMK if you need me to make a PR.

[jonkeane]

No problem, I've got one at #43348

I'll merge that if CI shows it gets rid of the note (and there are no other bad consequences) then cherrypick that back to the maintenance branch.

This isn't a here-and-now problem, but boy do I wish we could say "these notes we don't care about but all the others we do" for things like this 🤦

[jonkeane]

@amoeba am I reading these logs from recheck correctly that with new arrow parquetize is failing in this example:

> ### Name: csv_to_parquet
> ### Title: Convert a csv or a txt file to parquet format
> ### Aliases: csv_to_parquet
> 
> ### ** Examples
> 
> 
> # Conversion from a local csv file to a single parquet file :
> 
> csv_to_parquet(
+   path_to_file = parquetize_example("region_2022.csv"),
+   path_to_parquet = tempfile(fileext=".parquet")
+ )
Reading data...
Writing data...
Error in Table__from_dots(dots, schema, option_use_threads()) : 
  No Set_elt found for ALTSTRING class [class: vroom_chr, pkg: vroom]
Calls: csv_to_parquet ... as_arrow_table.data.frame -> <Anonymous> -> Table__from_dots
Reading data...
Execution halted

[amoeba]

Hey @jonkeane, I think that's the right interpretation.

[jonkeane]

Ah, that is a regression on us: #43349 I need to step away for a bit, in case someone wants to investigate what's up there. I don't think much has changed with our altrep code, but this could be from our non-api avoidance changes (or possibly the change we made to avoid executing functions while reading data).

[jonkeane]

Ok, I've merged the fix for the altrep thing that was hitting parquetize. @amoeba would you mind triggering the revdep checks again?

[jonkeane]

I just got a note about errors from lz4 on CRAN's devel debian runners.

With current Debian testing, installation now fails, see the install log. Apparently this is from yesterday's system upgrade of

liblz4-dev:amd64 (1.9.4-2, 1.9.4-3), liblz4-1:amd64 (1.9.4-2, 1.9.4-3)

I can confirm that ARROW_WITH_LZ4=OFF ./configure works where ./configure now fails.

We will need to fix this as will in our 17 submission (and get it in within two weeks too).

We are linking against the shared, system lz4 which is fine in this case because CRAN does not distribute these binaries. But one possibility is to force non-shared lz4 and it'll grab source + build.

cc @assignUser @nealrichardson