Skip to content

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure #11583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pitrou wants to merge 3 commits into from
Closed

ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure #11583

pitrou wants to merge 3 commits into from

Conversation

@pitrou
Copy link
Member

@pitrou pitrou commented Nov 1, 2021 •
edited
Loading

ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit -g cpp

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW

Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename pull request title in the following format?

ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

@pitrou pitrou changed the title [CI] [WIP] Always compile GTest from source [CI] Always compile GTest from source Nov 1, 2021
@pitrou pitrou marked this pull request as draft November 1, 2021 15:39
@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 209b1e6382c4fb4bc81e4f6a3e725fe2d1faddb9

Submitted crossbow builds: ursacomputing/crossbow @ actions-1063

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-debian-10-cpp Github Actions
test-debian-11-cpp Github Actions
test-fedora-33-cpp Github Actions
test-ubuntu-18.04-cpp Github Actions
test-ubuntu-18.04-cpp-release Github Actions
test-ubuntu-18.04-cpp-static Github Actions
test-ubuntu-20.04-cpp Github Actions
test-ubuntu-20.04-cpp-14 Github Actions
test-ubuntu-20.04-cpp-17 Github Actions
test-ubuntu-20.04-cpp-bundled Github Actions
test-ubuntu-20.04-cpp-thread-sanitizer Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 4da0482af9451acbbd5c7efef6d83040e5d48d97

Submitted crossbow builds: ursacomputing/crossbow @ actions-1064

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit test-conda-*

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit -g verify-rc-source

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1065

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

@github-actions crossbow submit vcpkg

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1066

Task Status
verify-rc-source-cpp-linux-amd64 Github Actions
verify-rc-source-cpp-macos-amd64 Github Actions
verify-rc-source-cpp-macos-arm64 Github Actions
verify-rc-source-csharp-linux-amd64 Github Actions
verify-rc-source-csharp-macos-amd64 Github Actions
verify-rc-source-csharp-macos-arm64 Github Actions
verify-rc-source-go-linux-amd64 Github Actions
verify-rc-source-go-macos-amd64 Github Actions
verify-rc-source-go-macos-arm64 Github Actions
verify-rc-source-integration-linux-amd64 Github Actions
verify-rc-source-integration-macos-amd64 Github Actions
verify-rc-source-integration-macos-arm64 Github Actions
verify-rc-source-java-linux-amd64 Github Actions
verify-rc-source-java-macos-amd64 Github Actions
verify-rc-source-js-linux-amd64 Github Actions
verify-rc-source-js-macos-amd64 Github Actions
verify-rc-source-js-macos-arm64 Github Actions
verify-rc-source-python-linux-amd64 Github Actions
verify-rc-source-python-macos-amd64 Github Actions
verify-rc-source-python-macos-arm64 Github Actions
verify-rc-source-ruby-linux-amd64 Github Actions
verify-rc-source-ruby-macos-amd64 Github Actions
verify-rc-source-ruby-macos-arm64 Github Actions
verify-rc-source-windows Github Actions

@github-actions
Copy link

github-actions bot commented Nov 1, 2021

Revision: 70a5ce2ae1beeac15b2745eaa4a701d4b8b2cb54

Submitted crossbow builds: ursacomputing/crossbow @ actions-1067

Task Status
test-build-vcpkg-win Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 1, 2021

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: a9cd09f34907e8183ebcdbca0f44d3145e17b11c

Submitted crossbow builds: ursacomputing/crossbow @ actions-1068

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@pitrou pitrou changed the title [CI] Always compile GTest from source ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure Nov 2, 2021
@github-actions
Copy link

github-actions bot commented Nov 2, 2021

https://issues.apache.org/jira/browse/ARROW-14554

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 7cf7cd67bf5f9748a95787eb0b824bae6d787036

Submitted crossbow builds: ursacomputing/crossbow @ actions-1069

Task Status
test-build-cpp-fuzz Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 860317fead19201a884c7b2029a4ba5d9b8eb62f

Submitted crossbow builds: ursacomputing/crossbow @ actions-1072

Task Status
test-build-cpp-fuzz Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit -g fuzz

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 90971770ecc6e54fef48de05e50aa9e8f44c4713

Submitted crossbow builds: ursacomputing/crossbow @ actions-1073

Task Status
test-build-cpp-fuzz Github Actions

@pitrou
ARROW-14554: [C++][CI] Fix OSS-Fuzz build failure
f356fd1 
ARROW-14393 downgraded the bundled GTest version to a version that doesn't build on the OSS-Fuzz setup
(and there doesn't seem to be an obvious fix or workaround around those build errors).

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Also, add a Crossbow task for the OSS-Fuzz build check, since the Github Actions cron job failure went unnoticed.
@pitrou pitrou marked this pull request as ready for review November 2, 2021 15:52
@pitrou pitrou requested a review from kszucs November 2, 2021 15:52
@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-*

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: 0bc448e

Submitted crossbow builds: ursacomputing/crossbow @ actions-1074

Task Status
test-conda-cpp Github Actions
test-conda-cpp-valgrind Azure
test-conda-python-3.10 Github Actions
test-conda-python-3.6 Github Actions
test-conda-python-3.6-pandas-0.23 Github Actions
test-conda-python-3.7 Github Actions
test-conda-python-3.7-hdfs-2.9.2 Github Actions
test-conda-python-3.7-hdfs-3.2.1 Github Actions
test-conda-python-3.7-kartothek-latest Github Actions
test-conda-python-3.7-kartothek-master Github Actions
test-conda-python-3.7-pandas-0.24 Github Actions
test-conda-python-3.7-pandas-latest Github Actions
test-conda-python-3.7-spark-v3.1.2 Github Actions
test-conda-python-3.8 Github Actions
test-conda-python-3.8-hypothesis Github Actions
test-conda-python-3.8-pandas-latest Github Actions
test-conda-python-3.8-pandas-nightly Github Actions
test-conda-python-3.8-spark-v3.2.0 Github Actions
test-conda-python-3.9 Github Actions
test-conda-python-3.9-dask-latest Github Actions
test-conda-python-3.9-dask-master Github Actions
test-conda-python-3.9-pandas-master Github Actions
test-conda-python-3.9-spark-master Github Actions

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-*

@pitrou
Copy link
Member Author

pitrou commented Nov 2, 2021

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions
Copy link

github-actions bot commented Nov 2, 2021

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1075

Task Status
test-conda-cpp-valgrind Azure

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

@github-actions crossbow submit test-conda-cpp-valgrind

@github-actions
Copy link

github-actions bot commented Nov 3, 2021

Revision: e035ed5

Submitted crossbow builds: ursacomputing/crossbow @ actions-1077

Task Status
test-conda-cpp-valgrind Azure

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

@kszucs Can we merge this? I'd like to unblock the OSS-Fuzz builds.

@kszucs
Copy link
Member

kszucs commented Nov 3, 2021

Hmm, it appears we cannot exercise the release verification script using crossbow (it tries to fetch Arrow 1.0.0)? @kszucs

We can only build releases with those tasks by providing additional parameters: #11511 (comment)

@kszucs
Copy link
Member

kszucs commented Nov 3, 2021

Re-bump the bundled GTest version, and try to fix the issue in the Windows verification script differently.

Do we have a JIRA for that?

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

I don't understand the question. What do you want a JIRA for?

@kszucs kszucs closed this in 64b80dd Nov 3, 2021
@pitrou pitrou deleted the oss-fuzz-gtest branch November 3, 2021 15:48
@ursabot
Copy link

ursabot commented Nov 3, 2021 •
edited
Loading

Benchmark runs are scheduled for baseline = 09b79a1 and contender = 64b80dd. 64b80dd is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️1.03% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.36% ⬆️0.04%] ursa-thinkcentre-m75q
Supported benchmarks:
ursa-i9-9960x: langs = Python, R, JavaScript
ursa-thinkcentre-m75q: langs = C++, Java
ec2-t3-xlarge-us-east-2: cloud = True

@kszucs
Copy link
Member

kszucs commented Nov 3, 2021

I don't understand the question. What do you want a JIRA for?

Does this fix the windows verification issue as well?

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021 •
edited
Loading

Does this fix the windows verification issue as well?

Hopefully, but I have no way to verify it (as explained above :-)).

@pitrou
Copy link
Member Author

pitrou commented Nov 3, 2021

(see diff in https://github.com/apache/arrow/pull/11583/files#diff-6c1e203cf55c63a1f177b4b0c97dc1aa62ea5eed5bd8e017e70703ddf119e343L90-L92)

@asfimport asfimport mentioned this pull request Nov 4, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kszucs kszucs kszucs approved these changes

Assignees

No one assigned

Labels

Component: C++

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@pitrou @kszucs @ursabot