ARROW-11735: [R] Allow Parquet and Arrow Dataset to be optional components #9610

ianmcook · 2021-03-01T23:20:10Z

Not implemented for Windows yet (that's ARROW-11884)

github-actions · 2021-03-01T23:20:36Z

https://issues.apache.org/jira/browse/ARROW-11735

nealrichardson

In general this looks like the right direction. I'm not certain that the configure script changes are exactly what we want but I'll need to think some more about it.

r/R/dataset-partition.R

r/data-raw/codegen.R

r/tests/testthat/latin1.R

r/tests/testthat/test-dataset.R

r/tests/testthat/test-python-flight.R

r/configure

nealrichardson · 2021-03-02T17:50:53Z

Oh, one other thing: in inst/build_arrow_static/sh, you'll want to make ARROW_PARQUET and ARROW_DATASET configurable by env var but default ON, just like ARROW_JEMALLOC. That way, we can make a CI job that sets them to OFF in the env so we can test this.

r/configure

nealrichardson · 2021-03-03T21:51:23Z

r/tools/autobrew

@@ -48,7 +48,8 @@ fi
 # Hardcode this for my custom autobrew build
 rm -f $BREWDIR/lib/*.dylib
 AWS_LIBS="-laws-cpp-sdk-config -laws-cpp-sdk-transfer -laws-cpp-sdk-identity-management -laws-cpp-sdk-cognito-identity -laws-cpp-sdk-sts -laws-cpp-sdk-s3 -laws-cpp-sdk-core -laws-c-event-stream -laws-checksums -laws-c-common -lpthread -lcurl"
-PKG_LIBS="-L$BREWDIR/lib -lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lthrift -llz4 -lsnappy -lzstd $AWS_LIBS"
+PKG_LIBS="-lparquet -larrow_dataset -larrow -larrow_bundled_dependencies -lthrift -llz4 -lsnappy -lzstd $AWS_LIBS"
+PKG_DIRS="-L$BREWDIR/lib"


We will need to remember to upstream this change when we release to CRAN

What is the relationship of this script (r/tools/autobrew) to https://github.com/autobrew/scripts/blob/master/apache-arrow?

It's based on an earlier version of Jeroen's autobrew, and it's closest now to how he actually builds the C++ dependencies. It looks like the new autobrew script is based on some artifacts that are bundled up in a separate process (see https://github.com/autobrew/bundler/blob/master/apache-arrow.sh).

I created ARROW-11861 to remind us of this.

Turns out we also needed to add -DARROW_R_WITH_PARQUET and -DARROW_R_WITH_DATASET to the cflags in https://github.com/autobrew/scripts/blob/master/apache-arrow (ARROW-12604)

ianmcook · 2021-03-04T15:17:54Z

Oh, one other thing: in inst/build_arrow_static/sh, you'll want to make ARROW_PARQUET and ARROW_DATASET configurable by env var but default ON, just like ARROW_JEMALLOC. That way, we can make a CI job that sets them to OFF in the env so we can test this.

Done in 8c5b235dddac47833b229323b4a216049eae5fea

ianmcook · 2021-03-04T20:38:38Z

@github-actions crossbow submit test-r-minimal-build

ianmcook · 2021-03-04T20:45:19Z

@nealrichardson do you want LIBARROW_MINIMAL to toggle Dataset and Parquet? I assume no.

dev/tasks/tasks.yml

nealrichardson · 2021-03-04T21:25:22Z

@nealrichardson do you want LIBARROW_MINIMAL to toggle Dataset and Parquet? I assume no.

Correct, we don't want to change that meaning (yet).

github-actions · 2021-03-04T21:42:52Z

Revision: 6f70c9911262d5d956c7f4ded49647269af848d9

Submitted crossbow builds: ursacomputing/crossbow @ actions-175

Task	Status
test-r-minimal-build

nealrichardson · 2021-03-04T23:07:32Z

dev/tasks/r/azure.linux.yml

@@ -47,6 +47,9 @@ jobs:
        export R_ORG={{ r_org }}
        export R_IMAGE={{ r_image }}
        export R_TAG={{ r_tag }}
+        export ARROW_DATASET={{ arrow_dataset|default("ON") }}
+        export ARROW_PARQUET={{ arrow_parquet|default("ON") }}


Unfortunately this didn't pass through all the way: https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=1721&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=6c939d89-0d1a-51f2-8b30-091a7a82e98c&l=240

Setting them here affects the environment in which docker-compose is called, but it doesn't forward into the docker environment.

I believe what you instead want is to pass them into docker-compose below, something like docker-compose -e ARROW_DATASET={{ arrow_dataset|default("ON") }} -e ARROW_PARQUET={{ arrow_parquet|default("ON") }} -e ARROW_S3={{ arrow_s3|default("ON") }} r

Got it, thanks. Let's see if it works to add them to .env (066ce065124cf7dcaa08225cae83177594da3965) and if not then I'll do it with -e.

Oops, need to append not overwrite. Fixed in 318371fbaa07f1b6fcdf71f58d7d5460a931da13

I think the only danger to appending to the .env file is that if you ran this locally, wouldn't it modify the file locally?

Ok, if folks might be running these commands locally then I can see how that would cause damage.
Looks like it didn't work anyhow. I will investigate.

Ah yeah, I think it's the same problem--note (for comparison) that R_ORG et al. are in the .env file, and they're environment variables for docker-compose, not passed into the docker container.

The fact that ARROW_R_DEV is in the env file, and then passed through in docker-compose.yml to the build environments, may just be due to my ignorance of all of this at the time I added it.

Ok, thanks. I tried a few different things just for kicks, then used -e per your suggestion in 9479c6ce079c7230b0e290dcb5658436d1d62fde

The build looks good after 9479c6c:
https://dev.azure.com/ursacomputing/crossbow/_build/latest?definitionId=1&branchName=actions-184-azure-test-r-minimal-build

ianmcook · 2021-03-04T23:50:08Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2021-03-04T23:51:09Z

Revision: 318371fbaa07f1b6fcdf71f58d7d5460a931da13

Submitted crossbow builds: ursacomputing/crossbow @ actions-177

Task	Status
test-r-minimal-build

ianmcook · 2021-03-05T02:16:38Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2021-03-05T02:17:35Z

Revision: 27c514cb1b14e7125e8ba50714c6480c93dc231a

Submitted crossbow builds: ursacomputing/crossbow @ actions-181

Task	Status
test-r-minimal-build

ianmcook · 2021-03-05T02:38:55Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2021-03-05T02:39:48Z

Revision: a810b3fce373c7f5090e367e5a6afc70db60414a

Submitted crossbow builds: ursacomputing/crossbow @ actions-182

Task	Status
test-r-minimal-build

ianmcook · 2021-03-05T03:06:30Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2021-03-05T03:07:52Z

Revision: 9479c6ce079c7230b0e290dcb5658436d1d62fde

Submitted crossbow builds: ursacomputing/crossbow @ actions-184

Task	Status
test-r-minimal-build

ianmcook · 2021-03-05T05:00:10Z

@github-actions crossbow submit -g r

ianmcook · 2021-03-05T05:02:36Z

@github-actions crossbow submit test-r-without-dataset-parquet-s3

github-actions · 2021-03-05T05:09:02Z

Revision: 6eda5896c638ca1c7af577c02ec56bd802a2fe42

Submitted crossbow builds: ursacomputing/crossbow @ actions-185

Task	Status
conda-linux-gcc-py36-cpu-r36
conda-linux-gcc-py37-cpu-r40
conda-osx-clang-py36-r36
conda-osx-clang-py37-r40
conda-win-vs2017-py36-r36
conda-win-vs2017-py37-r40
homebrew-r-autobrew
test-r-linux-as-cran
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos7-devtoolset-8
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-r-version-compatibility
test-r-versions
test-r-without-dataset-parquet-s3
test-ubuntu-18.04-r-sanitizer

github-actions · 2021-03-05T05:09:30Z

Revision: 6eda5896c638ca1c7af577c02ec56bd802a2fe42

Submitted crossbow builds: ursacomputing/crossbow @ actions-186

Task	Status
test-r-without-dataset-parquet-s3

ianmcook · 2021-03-05T14:04:38Z

The environment variables are being set in the container now, and Arrow is building with Dataset and Parquet off, but S3 is remaining on despite the environment variable.

…cipe

ianmcook · 2021-03-06T02:07:56Z

Sorry, I just merged the dataset mutate() PR which added a cpp function. Can you please rebase and update the annotation on that function? Otherwise this is looking good, will merge.

Done

ianmcook · 2021-03-06T02:12:34Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2021-03-06T02:13:58Z

Revision: b5ed1c0

Submitted crossbow builds: ursacomputing/crossbow @ actions-197

Task	Status
test-r-minimal-build

nealrichardson · 2021-03-06T03:14:03Z

@github-actions crossbow submit test-r-minimal-build

github-actions · 2021-03-06T03:14:45Z

Revision: 96e9104

Submitted crossbow builds: ursacomputing/crossbow @ actions-198

Task	Status
test-r-minimal-build

dev/tasks/r/azure.linux.yml

nealrichardson

This is great, thanks for doing this.

I'm going to try one more thing with the crossbow jobs, will revert if it's no good, and then merge.

nealrichardson · 2021-03-06T03:38:08Z

@github-actions crossbow submit test-r-rstudio* test-r-minimal-build

github-actions · 2021-03-06T03:38:55Z

Revision: 1b1680f

Submitted crossbow builds: ursacomputing/crossbow @ actions-199

Task	Status
test-r-minimal-build
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos7-devtoolset-8
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42

github-actions bot added Component: R Component: Parquet labels Mar 1, 2021

nealrichardson reviewed Mar 1, 2021

View reviewed changes

nealrichardson reviewed Mar 2, 2021

View reviewed changes

nealrichardson reviewed Mar 3, 2021

View reviewed changes

r/configure Outdated Show resolved Hide resolved

nealrichardson reviewed Mar 3, 2021

View reviewed changes

ianmcook marked this pull request as ready for review March 4, 2021 15:21

ianmcook commented Mar 4, 2021

View reviewed changes

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

nealrichardson reviewed Mar 4, 2021

View reviewed changes

ianmcook force-pushed the ARROW-11735 branch from e34597b to 6eda589 Compare March 5, 2021 04:56

ianmcook added 12 commits March 5, 2021 21:07

Respect ARROW_DATASET and ARROW_PARQUET env vars

21ce0a5

Remove arrow_with_parquet() condition around examples and use \dontrun

0d92497

Add nightly test of minimal build

1de7958

Add ARROW_ env vars to .env

411cb70

Append don't overwrite .env

1de8e1e

Fix pipes and use a temporary .env file

726295b

Look ma no pipes

6262778

Use -e to pass vars into container

2bb70e8

Name the job something without "minimal"

815d45a

Rename test to "minimal" and set LIBARROW_MINIMAL=true

f81acbe

Use ARROW_R_WITH_PARQUET and ARROW_R_WITH_DATASET on Windows

cbc7377

Use ARROW_R_WITH_PARQUET and ARROW_R_WITH_DATASET in Windows conda re…

b5ed1c0

…cipe

ianmcook force-pushed the ARROW-11735 branch from e46ce97 to b5ed1c0 Compare March 6, 2021 02:07

Regenerate arrowExports.cpp

96e9104

nealrichardson reviewed Mar 6, 2021

View reviewed changes

dev/tasks/r/azure.linux.yml Outdated Show resolved Hide resolved

nealrichardson approved these changes Mar 6, 2021

View reviewed changes

Update dev/tasks/r/azure.linux.yml

1b1680f

nealrichardson closed this in b07027e Mar 6, 2021

ianmcook mentioned this pull request Apr 29, 2021

Add missing cflags for apache-arrow 4.0.0 autobrew/scripts#10

Merged

This was referenced Apr 29, 2021

[R] Allow Parquet and Arrow Dataset to be optional components #27592

Closed

[R][Packaging] Apply changes in r/tools/autobrew upstream #18545

Closed

ARROW-11735: [R] Allow Parquet and Arrow Dataset to be optional components #9610

ARROW-11735: [R] Allow Parquet and Arrow Dataset to be optional components #9610

Uh oh!

Conversation

ianmcook commented Mar 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 1, 2021

Uh oh!

nealrichardson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nealrichardson commented Mar 2, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianmcook Apr 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianmcook commented Mar 4, 2021

Uh oh!

ianmcook commented Mar 4, 2021

Uh oh!

ianmcook commented Mar 4, 2021

Uh oh!

Uh oh!

nealrichardson commented Mar 4, 2021

Uh oh!

github-actions bot commented Mar 4, 2021

Uh oh!

nealrichardson Mar 4, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianmcook commented Mar 4, 2021

Uh oh!

github-actions bot commented Mar 4, 2021

Uh oh!

ianmcook commented Mar 5, 2021

Uh oh!

github-actions bot commented Mar 5, 2021

Uh oh!

ianmcook commented Mar 5, 2021

Uh oh!

github-actions bot commented Mar 5, 2021

Uh oh!

ianmcook commented Mar 1, 2021 •

edited

Loading

ianmcook Apr 29, 2021 •

edited

Loading

nealrichardson Mar 4, 2021 •

edited

Loading