ARROW-12094: [C++][R] Fix re2 building on clang/libc++ #9811

nealrichardson · 2021-03-26T18:00:03Z

Also includes a followup to ARROW-11736 to add the utf8proc/re2 features to arrow_info() and use that in skip_if_not_available().

github-actions · 2021-03-26T18:00:22Z

https://issues.apache.org/jira/browse/ARROW-12094

nealrichardson · 2021-03-26T18:02:10Z

@github-actions crossbow submit test-r-linux-as-cran

github-actions · 2021-03-26T18:03:25Z

Revision: e85b6b00caeb7b3e8ea33f923e7529fb4a13e2f8

Submitted crossbow builds: ursacomputing/crossbow @ actions-236

Task	Status
test-r-linux-as-cran

nealrichardson · 2021-03-26T18:25:30Z

@github-actions crossbow submit test-r-linux-as-cran

xhochy · 2021-03-26T18:27:18Z

@nealrichardson Do you have the error message / build log available?

nealrichardson · 2021-03-26T18:30:55Z

https://github.com/ursacomputing/crossbow/runs/2197449739?check_suite_focus=true#step:7:210 is the failure

xhochy · 2021-03-26T18:39:06Z

The R arrow package is built using std=gnu11, if you can try and compile it with gnu17, chances are high that you get rid of the error. Where is the C++ part of Arrow built? I cannot find that in the logs.

nealrichardson · 2021-03-26T18:44:29Z

Unfortunately because this is the as-cran test job, the C++ build output is suppressed (otherwise CRAN checks fail spuriously). It could be reproduced locally with docker-compose, if it were worthwhile. But we don't require C++ 17 for Arrow, only C++ 11, and we have to use the same compiler settings that R uses in order to avoid compiler mismatch: https://github.com/apache/arrow/blob/master/r/tools/nixlibs.R#L318. So I don't understand how we could compile re2 with gnu17.

github-actions · 2021-03-26T18:45:16Z

Revision: a347ac948a1871bab0b78d22602b2475cd1b10a0

Submitted crossbow builds: ursacomputing/crossbow @ actions-237

Task	Status
test-r-linux-as-cran

kou · 2021-03-28T06:00:12Z

It's caused by ignoring -stdlib=c++ specified by CXX environment variable that is provided by R CMD config CXX.
We use CMAKE_CXX_COMPILER to tell CXX environment variable information to bundled libraries. But it only has compiler path such as /usr/bin/clang++ in CXX=/usr/bin/clang++ -std=gnu++14 -stdlib=libc++. It doesn't have other options such as -std=gnu++14 -stdlib=libc++. CMAKE_CXX_COMPILER_ARG1 has them.

How about passing CMAKE_CXX_COMPILER_ARG1 information to bundled libraries?

diff --git a/cpp/cmake_modules/ThirdpartyToolchain.cmake b/cpp/cmake_modules/ThirdpartyToolchain.cmake
index e19544182..ef2ccfd0a 100644
--- a/cpp/cmake_modules/ThirdpartyToolchain.cmake
+++ b/cpp/cmake_modules/ThirdpartyToolchain.cmake
@@ -620,8 +620,8 @@ endif()
 # ----------------------------------------------------------------------
 # ExternalProject options
 
-set(EP_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${UPPERCASE_BUILD_TYPE}}")
-set(EP_C_FLAGS "${CMAKE_C_FLAGS} ${CMAKE_C_FLAGS_${UPPERCASE_BUILD_TYPE}}")
+set(EP_CXX_FLAGS "${CMAKE_CXX_COMPILER_ARG1} ${CMAKE_CXX_FLAGS} ${CMAKE_CXX_FLAGS_${UPPERCASE_BUILD_TYPE}}")
+set(EP_C_FLAGS "${CMAKE_C_COMPILER_ARG1} ${CMAKE_C_FLAGS} ${CMAKE_C_FLAGS_${UPPERCASE_BUILD_TYPE}}")
 
 if(NOT MSVC_TOOLCHAIN)
   # Set -fPIC on all external projects

…info() and register string libs there

nealrichardson · 2021-03-28T15:04:18Z

@github-actions crossbow submit test-r-linux-as-cran

github-actions · 2021-03-28T15:05:03Z

Revision: f402130

Submitted crossbow builds: ursacomputing/crossbow @ actions-242

Task	Status
test-r-linux-as-cran

nealrichardson · 2021-03-28T16:41:36Z

@github-actions autotune

nealrichardson · 2021-03-28T16:49:07Z

Thanks @kou! That seems to work.

kou · 2021-03-28T21:49:07Z

@github-actions crossbow submit -g nightly

github-actions · 2021-03-28T21:49:52Z

Revision: f52c105

Submitted crossbow builds: ursacomputing/crossbow @ actions-243

Task	Status
centos-7-amd64
centos-8-aarch64
centos-8-amd64
conda-clean
conda-linux-gcc-py36-aarch64
conda-linux-gcc-py36-cpu-r36
conda-linux-gcc-py36-cuda
conda-linux-gcc-py37-aarch64
conda-linux-gcc-py37-cpu-r40
conda-linux-gcc-py37-cuda
conda-linux-gcc-py38-aarch64
conda-linux-gcc-py38-cpu
conda-linux-gcc-py38-cuda
conda-linux-gcc-py39-aarch64
conda-linux-gcc-py39-cpu
conda-linux-gcc-py39-cuda
conda-osx-clang-py36-r36
conda-osx-clang-py37-r40
conda-osx-clang-py38
conda-osx-clang-py39
conda-win-vs2017-py36-r36
conda-win-vs2017-py37-r40
conda-win-vs2017-py38
debian-buster-amd64
debian-buster-arm64
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
gandiva-jar-osx
gandiva-jar-ubuntu
homebrew-cpp
homebrew-r-autobrew
nuget
python-sdist
test-build-vcpkg-win
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.6
test-conda-python-3.6-pandas-0.23
test-conda-python-3.7
test-conda-python-3.7-dask-latest
test-conda-python-3.7-hdfs-3.2
test-conda-python-3.7-kartothek-latest
test-conda-python-3.7-kartothek-master
test-conda-python-3.7-pandas-latest
test-conda-python-3.7-pandas-master
test-conda-python-3.7-spark-branch-3.0
test-conda-python-3.7-turbodbc-latest
test-conda-python-3.7-turbodbc-master
test-conda-python-3.8
test-conda-python-3.8-dask-master
test-conda-python-3.8-hypothesis
test-conda-python-3.8-jpype
test-conda-python-3.8-pandas-latest
test-conda-python-3.8-pandas-nightly
test-conda-python-3.8-spark-master
test-debian-10-cpp
test-debian-10-go-1.15
test-debian-10-python-3
test-debian-c-glib
test-debian-ruby
test-fedora-33-cpp
test-fedora-33-python-3
test-r-install-local
test-r-linux-as-cran
test-r-minimal-build
test-r-rhub-ubuntu-gcc-release
test-r-rocker-r-base-latest
test-r-rstudio-r-base-3.6-bionic
test-r-rstudio-r-base-3.6-centos7-devtoolset-8
test-r-rstudio-r-base-3.6-centos8
test-r-rstudio-r-base-3.6-opensuse15
test-r-rstudio-r-base-3.6-opensuse42
test-r-version-compatibility
test-r-versions
test-ubuntu-16.04-cpp
test-ubuntu-18.04-cpp
test-ubuntu-18.04-cpp-release
test-ubuntu-18.04-cpp-static
test-ubuntu-18.04-docs
test-ubuntu-18.04-python-3
test-ubuntu-18.04-r-sanitizer
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-14
test-ubuntu-20.04-cpp-17
test-ubuntu-20.04-cpp-thread-sanitizer
test-ubuntu-c-glib
test-ubuntu-ruby
ubuntu-bionic-amd64
ubuntu-bionic-arm64
ubuntu-focal-amd64
ubuntu-focal-arm64
ubuntu-groovy-amd64
ubuntu-groovy-arm64
wheel-manylinux2010-cp36m-amd64
wheel-manylinux2010-cp37m-amd64
wheel-manylinux2010-cp38-amd64
wheel-manylinux2010-cp39-amd64
wheel-manylinux2014-cp36m-amd64
wheel-manylinux2014-cp36m-arm64
wheel-manylinux2014-cp37m-amd64
wheel-manylinux2014-cp37m-arm64
wheel-manylinux2014-cp38-amd64
wheel-manylinux2014-cp38-arm64
wheel-manylinux2014-cp39-amd64
wheel-manylinux2014-cp39-arm64
wheel-osx-high-sierra-cp36m
wheel-osx-high-sierra-cp37m
wheel-osx-high-sierra-cp38
wheel-osx-high-sierra-cp39
wheel-osx-mavericks-cp36m
wheel-osx-mavericks-cp37m
wheel-osx-mavericks-cp38
wheel-osx-mavericks-cp39
wheel-windows-cp36m
wheel-windows-cp37m
wheel-windows-cp38
wheel-windows-cp39

nealrichardson · 2021-03-29T00:21:28Z

@github-actions crossbow submit test-r-versions

github-actions · 2021-03-29T00:22:21Z

Revision: 342ea03

Submitted crossbow builds: ursacomputing/crossbow @ actions-244

Task	Status
test-r-versions

nealrichardson · 2021-03-29T00:30:43Z

@kou other than the test-r-versions nightly (for which I've pushed a fix), everything else looks no worse than master. I'll merge once CI confirms the fix. Thanks again for your help! ❤️

kou

+1

Could you update the pull request description before we merge this?

github-actions bot added the Component: R label Mar 26, 2021

nealrichardson added 6 commits March 28, 2021 08:01

Followup to ARROW-11736: move all skip_if_not_available to use arrow_…

13cf74e

…info() and register string libs there

Handle one more skip case

b5c4140

Try to work around re2 failure on fedora-clang-devel

afbccda

More 🤦

5c5aace

Revert R build script special case

fc3cfbd

ARG1 cmake solution from @kou

f402130

nealrichardson force-pushed the re2-cran branch from a347ac9 to f402130 Compare March 28, 2021 15:03

github-actions bot added the Component: C++ label Mar 28, 2021

Autoformat/render all the things [automated commit]

f52c105

kou changed the title ~~ARROW-12094: [C++][R] Fix/workaround re2 building on clang/libc++~~ ARROW-12094: [C++][R] Fix re2 building on clang/libc++ Mar 28, 2021

Allow new skip_if_not_available to work on R < 3.5

342ea03

kou approved these changes Mar 29, 2021

View reviewed changes

nealrichardson closed this in 22e452d Mar 29, 2021

asfimport mentioned this pull request Mar 29, 2021

[C++][R] Fix/workaround re2 building on clang/libc++ #27918

Closed

ARROW-12094: [C++][R] Fix re2 building on clang/libc++ #9811

ARROW-12094: [C++][R] Fix re2 building on clang/libc++ #9811

Uh oh!

Conversation

nealrichardson commented Mar 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 26, 2021

Uh oh!

nealrichardson commented Mar 26, 2021

Uh oh!

github-actions bot commented Mar 26, 2021

Uh oh!

nealrichardson commented Mar 26, 2021

Uh oh!

xhochy commented Mar 26, 2021

Uh oh!

nealrichardson commented Mar 26, 2021

Uh oh!

xhochy commented Mar 26, 2021

Uh oh!

nealrichardson commented Mar 26, 2021

Uh oh!

github-actions bot commented Mar 26, 2021

Uh oh!

kou commented Mar 28, 2021

Uh oh!

nealrichardson commented Mar 28, 2021

Uh oh!

github-actions bot commented Mar 28, 2021

Uh oh!

nealrichardson commented Mar 28, 2021

Uh oh!

nealrichardson commented Mar 28, 2021

Uh oh!

kou commented Mar 28, 2021

Uh oh!

github-actions bot commented Mar 28, 2021

Uh oh!

nealrichardson commented Mar 29, 2021

Uh oh!

github-actions bot commented Mar 29, 2021

Uh oh!

nealrichardson commented Mar 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nealrichardson commented Mar 26, 2021 •

edited

Loading

nealrichardson commented Mar 29, 2021 •

edited

Loading