GH-34615: [CI][C++] Add CI job for basic format support without ARROW_COMPUTE #34617

benibus · 2023-03-17T20:00:49Z

This adds a crossbow job for ARROW_IPC, ARROW_PARQUET, and ARROW_CSV - based on a minimal Ubuntu image.

The job primarily aims to test the core Arrow library + basic format support without the full kernel registry provided by ARROW_COMPUTE. Note that ARROW_JSON is implicitly enabled as well, since it's a dependency of ARROW_TESTING.

Closes: [CI][C++] Add CI job for basic format support without ARROW_COMPUTE #34615
Closes: [CI][C++] arrow-compute-internals-test fails with No function registered with name: equal on test-cuda-cpp #34655

github-actions · 2023-03-17T20:01:14Z

Closes: [CI][C++] Add CI job for basic format support without ARROW_COMPUTE #34615

github-actions · 2023-03-17T20:01:17Z

⚠️ GitHub issue #34615 has been automatically assigned in GitHub to PR creator.

kou · 2023-03-19T21:20:39Z

There are still some enabled components:

https://github.com/apache/arrow/actions/runs/4462272725/jobs/7836694435?pr=34617#step:6:1866

--   ARROW_BUILD_UTILITIES=ON [default=OFF]
--       Build Arrow commandline utilities
--   ARROW_CSV=ON [default=OFF]
--       Build the Arrow CSV Parser Module
--   ARROW_FILESYSTEM=ON [default=OFF]
--       Build the Arrow Filesystem Layer
--   ARROW_IPC=ON [default=ON]
--       Build the Arrow IPC extensions
--   ARROW_JSON=ON [default=OFF]
--       Build Arrow with JSON support (requires RapidJSON)
--   ARROW_PARQUET=ON [default=OFF]
--       Build the Parquet libraries

Are they intentional?

benibus · 2023-03-19T23:21:59Z

Are they intentional?

ARROW_BUILD_UTILITIES and ARROW_FILESYSTEM are unintentional (I think the latter is being enabled by PARQUET_REQUIRE_ENCRYPTION=ON). Regarding the others, see my edit to the body message (I attempted to clarify my intent as well).

benibus · 2023-03-19T23:22:36Z

Also, fair warning: I've never actually messed with CI system before so there may be a better way to do this. I just went with one of the preexisting Ubuntu images since it seemed like a reasonable base (to me, at least).

kou · 2023-03-20T00:36:25Z

OK. Then "minimal" may be misleading. For example, we use "minimal" as "all components are disabled" in https://github.com/apache/arrow/blob/main/cpp/CMakePresets.json#L49 . Can we use other label for this job?

If we enable Parquet, CI time isn't short. How about running this as a nightly CI job instead like test-*-cpp-* in https://github.com/apache/arrow/blob/main/dev/tasks/tasks.yml#L1171-L1256 ?

benibus · 2023-03-20T15:18:44Z

I agree that it's misleading.

We could also handle both cases pretty easily, I think. The "minimal" image I have here could disable everything as it implies. Then, the nightly task would use that image but enable CSV/Parquet.

benibus · 2023-03-20T17:53:44Z

@github-actions crossbow submit test-ubuntu-20.04-cpp-minimal-with-formats

github-actions · 2023-03-20T22:24:08Z

Revision: aa7cb5b4fae7bd3505c8f6dbc976ae5131028bfd

Submitted crossbow builds: ursacomputing/crossbow @ actions-7a8059874a

Task	Status
test-ubuntu-20.04-cpp-minimal-with-formats

github-actions · 2023-03-20T22:25:09Z

Revision: aa7cb5b4fae7bd3505c8f6dbc976ae5131028bfd

Submitted crossbow builds: ursacomputing/crossbow @ actions-89628f20c6

Task	Status
test-ubuntu-20.04-cpp-minimal-with-formats

.github/workflows/cpp.yml

dev/tasks/tasks.yml

kou · 2023-03-21T13:39:59Z

Could you also update "minimal" in the title and the description of this PR and GH-34615?

kou · 2023-03-21T21:42:09Z

@github-actions crossbow submit test-ubuntu-20.04-cpp-minimal-with-formats

github-actions · 2023-03-21T21:44:38Z

Revision: 9147c15ce4120b9b6fc0d9b233d50d7bc1d1d2a7

Submitted crossbow builds: ursacomputing/crossbow @ actions-b9824dee1c

Task	Status
test-ubuntu-20.04-cpp-minimal-with-formats

raulcd · 2023-03-22T11:26:20Z

I've updated the body to include the closes #34655 as suggested.

raulcd · 2023-03-22T11:26:51Z

@github-actions crossbow submit test-cuda-cpp

github-actions · 2023-03-22T11:29:18Z

Revision: 9147c15ce4120b9b6fc0d9b233d50d7bc1d1d2a7

Submitted crossbow builds: ursacomputing/crossbow @ actions-2e55c6dbfe

Task	Status
test-cuda-cpp

raulcd · 2023-03-22T11:49:05Z

The test-cuda-cpp seems to fail with a similar error to the one reported on #34655 :

 '_error_or_value25.status()' failed with Key error: No function registered with name: sort_indices
/arrow/cpp/src/arrow/compute/exec/order_by_node_test.cc:61: Failure
Failed

@benibus let me know if I should open a different issue for that or should be fixed here. Thanks!

benibus · 2023-03-22T13:19:53Z

@raulcd Yeah, I saw that last night. Probably best to handle it here as well.

I'll do a light CMake refactor to hopefully prevent more of these issues since it's too easy to make mistakes ATM.

Some temporal casts will fail without timezone info

compute-expression-test was recently merged into compute-internals-test as part of a larger refactor. However, the expression tests depend on additional kernels (i.e. equal) so I re-separated them.

benibus · 2023-03-22T15:04:34Z

@kou @raulcd test-ubuntu-20.04-cpp-minimal-with-formats and test-cuda-cpp should pass on this branch now.

benibus · 2023-03-22T15:31:37Z

@github-actions crossbow submit test-cuda-cpp

github-actions · 2023-03-22T15:34:14Z

Revision: 0a79eb3

Submitted crossbow builds: ursacomputing/crossbow @ actions-53125ee05c

Task	Status
test-cuda-cpp

benibus · 2023-03-22T15:36:20Z

@github-actions crossbow submit test-ubuntu-20.04-cpp-minimal-with-formats

github-actions · 2023-03-22T15:38:50Z

Revision: 0a79eb3

Submitted crossbow builds: ursacomputing/crossbow @ actions-a36f0bd26a

Task	Status
test-ubuntu-20.04-cpp-minimal-with-formats

kou

+1

ursabot · 2023-03-23T10:50:49Z

Benchmark runs are scheduled for baseline = 3600bd8 and contender = 4487be0. 4487be0 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished ⬇️0.0% ⬆️0.0%] ec2-t3-xlarge-us-east-2
[Finished ⬇️0.18% ⬆️0.0%] test-mac-arm
[Finished ⬇️0.0% ⬆️0.0%] ursa-i9-9960x
[Finished ⬇️0.09% ⬆️0.03%] ursa-thinkcentre-m75q
Buildkite builds:
[Finished] 4487be02 ec2-t3-xlarge-us-east-2
[Finished] 4487be02 test-mac-arm
[Finished] 4487be02 ursa-i9-9960x
[Finished] 4487be02 ursa-thinkcentre-m75q
[Finished] 3600bd80 ec2-t3-xlarge-us-east-2
[Finished] 3600bd80 test-mac-arm
[Finished] 3600bd80 ursa-i9-9960x
[Finished] 3600bd80 ursa-thinkcentre-m75q
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java

… ARROW_COMPUTE (apache#34617) This adds a crossbow job for `ARROW_IPC`, `ARROW_PARQUET`, and `ARROW_CSV` - based on a minimal Ubuntu image. The job primarily aims to test the core Arrow library + basic format support without the full kernel registry provided by `ARROW_COMPUTE`. Note that `ARROW_JSON` is implicitly enabled as well, since it's a dependency of `ARROW_TESTING`. * Closes: apache#34615 * Closes: apache#34655 Authored-by: benibus <bpharks@gmx.com> Signed-off-by: Sutou Kouhei <kou@clear-code.com>

github-actions bot added the awaiting review Awaiting review label Mar 17, 2023

benibus force-pushed the GH-34615-minimal-cpp-ci-job branch 2 times, most recently from 5510338 to eee9253 Compare March 19, 2023 00:25

github-actions bot added the Component: C++ label Mar 19, 2023

benibus force-pushed the GH-34615-minimal-cpp-ci-job branch from eee9253 to e3d60b9 Compare March 19, 2023 19:13

benibus marked this pull request as ready for review March 19, 2023 21:06

benibus requested review from assignUser, kou, raulcd and westonpace as code owners March 19, 2023 21:06

benibus force-pushed the GH-34615-minimal-cpp-ci-job branch from e3d60b9 to aa7cb5b Compare March 20, 2023 16:58

kou reviewed Mar 21, 2023

View reviewed changes

.github/workflows/cpp.yml Outdated Show resolved Hide resolved

dev/tasks/tasks.yml Outdated Show resolved Hide resolved

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Mar 21, 2023

benibus mentioned this pull request Mar 21, 2023

[CI][C++] arrow-compute-internals-test fails with No function registered with name: equal on test-cuda-cpp #34655

Closed

benibus force-pushed the GH-34615-minimal-cpp-ci-job branch from aa7cb5b to 9147c15 Compare March 21, 2023 16:08

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Mar 21, 2023

benibus changed the title ~~GH-34615: [CI][C++] Add CI job for minimal C++ build~~ GH-34615: [CI][C++] Add CI job for basic format support without ARROW_COMPUTE Mar 21, 2023

benibus requested a review from kou March 21, 2023 16:44

benibus added 5 commits March 22, 2023 10:11

Add CI image ubuntu-cpp-minimal

3f7f99a

Install tzdata in ubuntu docker images

9a0ad0b

Some temporal casts will fail without timezone info

Separate compute-expression-test

a1bea2e

compute-expression-test was recently merged into compute-internals-test as part of a larger refactor. However, the expression tests depend on additional kernels (i.e. equal) so I re-separated them.

Add Crossbow job with additional formats

b59c35a

Use params.flags to enable components

67df515

benibus force-pushed the GH-34615-minimal-cpp-ci-job branch from 9147c15 to 42e52be Compare March 22, 2023 14:16

Refactor compute cmake files

0a79eb3

benibus force-pushed the GH-34615-minimal-cpp-ci-job branch from 42e52be to 0a79eb3 Compare March 22, 2023 14:18

kou approved these changes Mar 23, 2023

View reviewed changes

kou merged commit 4487be0 into apache:main Mar 23, 2023

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Mar 23, 2023

kou mentioned this pull request Sep 18, 2024

GH-40570: [CI] Default environment to Ubuntu 22.04 instead of 20.04 #44151

Merged

GH-34615: [CI][C++] Add CI job for basic format support without ARROW_COMPUTE #34617

GH-34615: [CI][C++] Add CI job for basic format support without ARROW_COMPUTE #34617

Uh oh!

Conversation

benibus commented Mar 17, 2023 • edited by raulcd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 17, 2023

Uh oh!

github-actions bot commented Mar 17, 2023

Uh oh!

kou commented Mar 19, 2023

Uh oh!

benibus commented Mar 19, 2023

Uh oh!

benibus commented Mar 19, 2023

Uh oh!

kou commented Mar 20, 2023

Uh oh!

benibus commented Mar 20, 2023

Uh oh!

benibus commented Mar 20, 2023 • edited by kou Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 20, 2023

Uh oh!

github-actions bot commented Mar 20, 2023

Uh oh!

Uh oh!

Uh oh!

kou commented Mar 21, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Mar 21, 2023

Uh oh!

github-actions bot commented Mar 21, 2023

Uh oh!

raulcd commented Mar 22, 2023

Uh oh!

raulcd commented Mar 22, 2023

Uh oh!

github-actions bot commented Mar 22, 2023

Uh oh!

raulcd commented Mar 22, 2023

Uh oh!

benibus commented Mar 22, 2023

Uh oh!

benibus commented Mar 22, 2023

Uh oh!

benibus commented Mar 22, 2023

Uh oh!

github-actions bot commented Mar 22, 2023

Uh oh!

benibus commented Mar 22, 2023

Uh oh!

github-actions bot commented Mar 22, 2023

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

ursabot commented Mar 23, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

benibus commented Mar 17, 2023 •

edited by raulcd

Loading

benibus commented Mar 20, 2023 •

edited by kou

Loading

kou commented Mar 21, 2023 •

edited

Loading