Skip to content

ARROW-13507: [R] LTO job on CRAN fails #10849

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jonkeane wants to merge 5 commits into from
Closed

ARROW-13507: [R] LTO job on CRAN fails #10849

jonkeane wants to merge 5 commits into from

Conversation

@jonkeane
Copy link
Member

@jonkeane jonkeane commented Aug 2, 2021

A test to see if we can (for now) build r-debug before using it

@github-actions
Copy link

github-actions bot commented Aug 2, 2021

https://issues.apache.org/jira/browse/ARROW-13507

nealrichardson
nealrichardson reviewed Aug 2, 2021
View reviewed changes
dev/tasks/docker-tests/azure.linux.yml Outdated
git clone https://github.com/nealrichardson/r-debug.git
git checkout lto
cd r-debug
./buildall.sh
Copy link
Member

@nealrichardson nealrichardson Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A possibly faster way to do this is to add a new job like for the r-sanitizer, and add dockerfile like https://github.com/nealrichardson/r-debug/blob/lto/r-debug-5/Dockerfile:

FROM wch1/r-debug

# RDlto: R built with --enable-lto
RUN wget https://raw.githubusercontent.com/nealrichardson/r-debug/lto/r-devel/buildR.sh
RUN buildR.sh lto
RUN RDlto -q -e 'install.packages("pak", repos = "https://r-lib.github.io/p/pak/dev")'
RUN RDlto -q -e 'pak::pkg_install(c("devtools", "Rcpp", "roxygen2", "testthat", "memoise", "rmarkdown"))'

Copy link
Member

@nealrichardson nealrichardson Aug 2, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, turns out R-hub has a docker image already: https://github.com/r-hub/rhub-linux-builders/blob/master/debian-gcc-devel-lto/Dockerfile

@jonkeane
Copy link
Member Author

jonkeane commented Aug 2, 2021

@github-actions crossbow submit test-r-r_hub-debian-gcc-devel-lto-latest

@jonkeane
Copy link
Member Author

jonkeane commented Aug 2, 2021

@github-actions crossbow submit test-r-rhub-debian-gcc-devel-lto-latest

This should fail with the CRAN error at https://www.stats.ox.ac.uk/pub/bdr/LTO/arrow.out

@github-actions
Copy link

github-actions bot commented Aug 2, 2021

Revision: e278e8a

Submitted crossbow builds: ursacomputing/crossbow @ actions-693

Task Status
test-r-rhub-debian-gcc-devel-lto-latest Azure

@jonkeane
Copy link
Member Author

jonkeane commented Aug 3, 2021

@github-actions crossbow submit test-r-rhub-debian-gcc-devel-lto-latest

@github-actions
Copy link

github-actions bot commented Aug 3, 2021

Revision: d27fd38

Submitted crossbow builds: ursacomputing/crossbow @ actions-703

Task Status
test-r-rhub-debian-gcc-devel-lto-latest Azure

@jonkeane
Copy link
Member Author

jonkeane commented Aug 3, 2021

I also ran this locally removing all of the -fno-lto flags that were added in ARROW-13199, and I get a segfault there as well (this is at the same point as the crossbow build above which tries to remove that flag only when we know we're on LTO):

16: tools:::.test_load_package("arrow", "/arrow/r/check/arrow.Rcheck/00LOCK-arrow/00new")
An irrecoverable exception occurred. R is aborting now ...
Segmentation fault
ERROR: loading failed

So it looks like we need to do something more to be able to build/link on the CRAN LTO check.

cpcloud
cpcloud reviewed Aug 3, 2021
View reviewed changes
r/inst/build_arrow_static.sh Outdated
-DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON \
-DCMAKE_FIND_PACKAGE_NO_PACKAGE_REGISTRY=ON \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-ON} \
-DCMAKE_INTERPROCEDURAL_OPTIMIZATION=${ARROW_LTO:-ON} \
Copy link
Contributor

@cpcloud cpcloud Aug 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be ${ARROW_LTO:-$ARROW_DEFAULT_PARAM}? As it currently is, it looks like LTO will always be turned on.

Copy link
Member Author

@jonkeane jonkeane Aug 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed this to -DCMAKE_INTERPROCEDURAL_OPTIMIZATION=${ARROW_LTO:-OFF} Confusingly, $ARROW_DEFAULT_PARAM here is about adding extra features like compression libraries (based on $LIBARROW_MINIMAL above).

cpcloud
cpcloud reviewed Aug 3, 2021
View reviewed changes
r/inst/build_arrow_static.sh Outdated
fi

if echo "$ARROW_R_CXXFLAGS" | grep -q "flto"; then
ARROW_LTO="ON"
Copy link
Contributor

@cpcloud cpcloud Aug 3, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this have an else clause setting ARROW_LTO to "OFF"?

@jonkeane
Copy link
Member Author

jonkeane commented Aug 3, 2021

@github-actions crossbow submit test-r-rhub-debian-gcc-devel-lto-latest

@github-actions
Copy link

github-actions bot commented Aug 3, 2021

Revision: 02bd56a82ff42565d9db8a788452b4d6eb4a1264

Submitted crossbow builds: ursacomputing/crossbow @ actions-707

Task Status
test-r-rhub-debian-gcc-devel-lto-latest Azure

@jonkeane
Copy link
Member Author

jonkeane commented Aug 3, 2021

@github-actions crossbow submit test-r-rhub-debian-gcc-devel-lto-latest

@github-actions
Copy link

github-actions bot commented Aug 3, 2021

Revision: 6f2f0f77d4ce62a966b537dd85d6e32e28d1648f

Submitted crossbow builds: ursacomputing/crossbow @ actions-709

Task Status
test-r-rhub-debian-gcc-devel-lto-latest Azure

@jonkeane
Copy link
Member Author

jonkeane commented Aug 3, 2021

@github-actions crossbow submit test-r-rhub-debian-gcc-devel-lto-latest

@github-actions
Copy link

github-actions bot commented Aug 3, 2021

Revision: 6f2f0f77d4ce62a966b537dd85d6e32e28d1648f

Submitted crossbow builds: ursacomputing/crossbow @ actions-711

Task Status
test-r-rhub-debian-gcc-devel-lto-latest Azure

@jonkeane jonkeane force-pushed the ARROW-13507-r-lto branch from 6f2f0f7 to 2a8fc19 Compare August 4, 2021 14:07
@jonkeane
Copy link
Member Author

jonkeane commented Aug 4, 2021

@github-actions crossbow submit test-r-rhub-debian-gcc-devel-lto-latest

@github-actions
Copy link

github-actions bot commented Aug 4, 2021

Revision: 2a8fc19

Submitted crossbow builds: ursacomputing/crossbow @ actions-713

Task Status
test-r-rhub-debian-gcc-devel-lto-latest Azure

nealrichardson pushed a commit to nealrichardson/arrow that referenced this pull request Aug 5, 2021
@jonkeane @nealrichardson
ARROW-13507: [R] LTO job on CRAN fails
a09dd18 
A test to see if we can (for now) build r-debug before using it

Closes apache#10849 from jonkeane/ARROW-13507-r-lto

Authored-by: Jonathan Keane <jkeane@gmail.com>
Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>
@asfimport asfimport mentioned this pull request Aug 5, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@cpcloud cpcloud cpcloud left review comments

@nealrichardson nealrichardson nealrichardson approved these changes

Assignees

No one assigned

Labels

Component: C++ Component: R

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@jonkeane @cpcloud @nealrichardson