Skip to content

GH-36456: [CI][R] Unlink system OpenSSL to avoid mixing OpenSSL versions #36522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kou wants to merge 8 commits into from
Closed

GH-36456: [CI][R] Unlink system OpenSSL to avoid mixing OpenSSL versions #36522

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
f28684d
GH-36456: [R] Remove system OpenSSL to avoid mixing OpenSSL versions
kou Jul 7, 2023
20b1ba7
Use unlink
kou Jul 7, 2023
bf9bf44
find
kou Jul 7, 2023
67f7d7d
Uninstall conda's OpenSSL
kou Jul 9, 2023
a1e0265
Find
kou Jul 9, 2023
9ea5823
Find
kou Jul 10, 2023
42dd623
grep
kou Jul 10, 2023
e79c31e
bash
kou Jul 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 17 additions & 9 deletions dev/tasks/r/github.macos.autobrew.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,14 +35,20 @@ jobs:
- "{{ macros.r_oldrel.ver }}"
steps:
{{ macros.github_checkout_arrow()|indent }}
- name: Setup hosted
if: matrix.platform != 'macos-10.13'
run: |
# minio and sccache are pre-installed on the self-hosted 10.13 runner.
# Install minio for tests.
brew install minio
brew install sccache

# Ensure removing OpenSSL from the default paths to avoid
# mixing OpenSSL in the default paths and OpenSSL installed
# by autobrew.
brew unlink openssl || :
Copy link
Member

@pitrou pitrou Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm... can we instead make sure our build chain selects OpenSSL consistently? Users will otherwise probably encounter a similar problem if they try compiling Arrow themselves.

Copy link
Member

@assignUser assignUser Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That would also be my preference though it seems to be tricky and I haven't had the time to investigate.

Copy link
Member Author

@kou kou Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that it's difficult.

For example, please consider the following case:

  • OpenSSL 1.1 is installed in /usr/local
  • OpenSSL 3 is install in /opt/openssl
  • c-ares is installed in /usr/local
  • Build bundled gRPC with c-ares and OpenSSL 3
  • gRPC uses -I/usr/local/include from c-ares and -I/opt/openssl/include from OpenSSL 3
  • -I/opt/openssl/include -I/usr/local/include will work but -I/usr/local/include -I/opt/openssl/include will not work because OpenSSl 1.1 headers in /usr/local/include are used
  • I don't think that we can always use include flags from OpenSSL 3 as the first include flag because there are more version mixable libraries such as Abseil

- name: Configure autobrew script
run: |
# minio and sccache are pre-installed on the self-hosted 10.13 runner
if [ {{ '${{ matrix.platform }}' }} != macos-10.13 ]; then
# install minio for tests
brew install minio
brew install sccache
fi
cd arrow/r
{{ macros.pin_brew_formulae(is_fork)|indent }}
- uses: r-lib/actions/setup-r@v2
Expand All @@ -57,7 +63,7 @@ jobs:
# between pre-installed R version on the self-hosted runners
rig default {{ '${{ matrix.r-version }}' }}
rig system setup-user-lib
rig system add-pak
rig system add-pak
- uses: r-lib/actions/setup-r-dependencies@v2
with:
working-directory: 'arrow/r'
Expand All @@ -72,8 +78,10 @@ jobs:
ARROW_USE_PKG_CONFIG: false
ARROW_R_DEV: true
FORCE_AUTOBREW: true
{{ macros.github_set_sccache_envvars()|indent(8)}}
run: arrow/ci/scripts/r_test.sh arrow
{{ macros.github_set_sccache_envvars()|indent(8)}}
run: |
find / 2>&1 | grep include/openssl || :
arrow/ci/scripts/r_test.sh arrow
Comment on lines +83 to +84
Copy link
Member

@pitrou pitrou Jul 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit embarassed, but I don't understand what these two lines do...

Copy link
Member

@assignUser assignUser Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the first line is just for debug purposes printing all find hits of include/openssl or nothing. The script setup an env for r testing and then test/checks the r package.

Copy link
Member

@assignUser assignUser Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/ursacomputing/crossbow/actions/runs/5504413947/jobs/10030597584#step:8:19

Copy link
Member Author

@kou kou Jul 12, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. It's just a debug print.
I want to find OpenSSl header files in the system but I couldn't find them...

- name: Dump install logs
run: cat arrow/r/check/arrow.Rcheck/00install.out
if: always()
Expand Down
8 changes: 8 additions & 0 deletions dev/tasks/r/github.packages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,11 @@ jobs:

rig system setup-user-lib
rig system add-pak

# ensure removing OpenSSL from the default paths to avoid
# mixing OpenSSL in the default paths and OpenSSL installed
# by autobrew
brew unlink openssl || :
{{ macros.github_setup_local_r_repo(false, true)|indent }}
- name: Prepare Dependency Installation

Expand All @@ -181,6 +186,9 @@ jobs:
- name: Install sccache
if: startsWith(matrix.platform, 'macos')
run: brew install sccache
- name: Find
run: |
find / 2>&1 | grep include/openssl || :
- name: Build Binary
id: build
shell: Rscript {0}
Expand Down