[R] Add support for more compression codecs in Windows build

[asfimport]

When I attempt to write a parquet file using lz4, zstd, or brotli compression using R arrow 0.15.0, I am unable to do so due to the codec support not being built (example below).

 

> arrow::write_parquet(payout_strategy, sink = "records_test_lz4.parquet",compression = "lz4")
Error in parquet___arrow___FileWriter__WriteTable(self, table, chunk_size) : 
 Arrow error: IOError: Arrow error: NotImplemented: LZ4 codec support not built

 

I believe that the error is generated through https://github.com/apache/arrow/blob/master/cpp/src/arrow/util/compression.cc#L124-L145, but I am not sure how to call 

install.packages("arrow")

in R to enable the ARROW_WITH_ZSTD/LZ4/BROTLI flags, or whether I should be doing installing zstd separately from arrow and then doing something pre- or post-install to link zstd with arrow. From #1209, it appears that zstd support has been added to arrow and parquet in general, and the R package readme (https://github.com/apache/arrow/tree/master/r) notes "On macOS and Windows, installing a binary package from CRAN will handle Arrow's C++ dependencies for you", but I get the sense that does not apply to zstd.

 

Is there guidance as to how to enable zstd and other compression codecs prior to or after downloading the R arrow package? Could this be added to the R documentation somewhere for future reference?

Environment: Windows 10
Reporter: Grant Nguyen / @gnguy
Assignee: Grant Nguyen / @gnguy

PRs and other links:

• GitHub Pull Request #5814

_{Note: This issue was originally created as ARROW-6960. Please see the migration documentation for further details.}

[asfimport]

Neal Richardson / @nealrichardson:
It is not possible to add them after the fact. You need the Arrow C++ library built with those codecs, and then you'd need to install the R package from source, linking to that library. On other platforms, I could point you at a few options for doing that, but not Windows.

On Windows, we could try to support more codecs in the official build. If you wanted to submit a PR to do that, you'd want to modify https://github.com/apache/arrow/blob/master/ci/PKGBUILD (a copy of https://github.com/r-windows/rtools-packages/blob/master/mingw-w64-arrow/PKGBUILD, which feeds into what CRAN uses to build its binaries) to include additional dependencies and turn the right cmake flags on, which you identified. https://github.com/r-windows/rtools-packages shows what libraries should be available already--I see lz4 but not zstd or brotli. So lz4 might be trivial to add. For the others, you'd need to first add it to https://github.com/r-windows/rtools-packages/; its README has a brief note at the bottom for how to approach it.

[asfimport]

Grant Nguyen / @gnguy:
Thanks @nealrichardson  for the very detailed explanation, that helps a lot! I will look into this in the next few days – the lz4 addition to PKGBUILD seems like a good starting point – not sure that I have quite the level of expertise to add zstd and brotli to rtools but will investigate further.

[asfimport]

Neal Richardson / @nealrichardson:
Sounds good. After you work out the lz4, if you wanted to move on to zstd, you could start by copying https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-zstd/PKGBUILD to rtools-packages (fork it and make a PR adding it). Appveyor will test it for you, and Jeroen can help you with the details. https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-brotli/PKGBUILD exists too but looks a little more involved because you'd probably want to prune the python-specific build targets.

[asfimport]

Grant Nguyen / @gnguy:
Thanks for the guidance with this, I have made the updates to PKGBUILD and made a PR: #5814. 

[asfimport]

Neal Richardson / @nealrichardson:
Issue resolved by pull request 5814
#5814