Skip to content

chore: Run Comet tests for more Spark versions #1582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kazuyukitanimura merged 11 commits into from
Apr 1, 2025
Merged

chore: Run Comet tests for more Spark versions #1582

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f62f8dd
Run Comet tests for more Spark versions
andygrove Mar 31, 2025
6a91eec
update
andygrove Mar 31, 2025
3efeca9
update
andygrove Mar 31, 2025
a7e6820
update docs
andygrove Mar 31, 2025
423a345
address feedback
andygrove Mar 31, 2025
5fdbf7f
address feedback
andygrove Mar 31, 2025
8fa43ce
update
andygrove Mar 31, 2025
f2ed228
update
andygrove Mar 31, 2025
87c608f
update
andygrove Mar 31, 2025
ffebd9d
update
andygrove Mar 31, 2025
bc9170a
update
andygrove Mar 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 37 additions & 0 deletions .github/workflows/pr_build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,8 @@ env:
RUST_VERSION: stable

jobs:

# run Comet test with full matrix of Java and Scala versions for the latest supported 3.5.x release
linux-test:
strategy:
matrix:
Expand Down Expand Up @@ -76,6 +78,41 @@ jobs:
# upload test reports only for java 17
upload-test-reports: ${{ matrix.java_version == '17' }}

# run Comet tests for older 3.5.x releases with one java and scala versions
linux-test-35-minor-versions:
strategy:
matrix:
os: [ubuntu-latest]
java_version: [17]
test-target: [java]
spark-version: ['3.5']
spark-patch-version: ['3.5.2', '3.5.3', '3.5.4']
scala-version: ['2.12']
is_push_event:
- ${{ github.event_name == 'push' }}
fail-fast: false
name: ${{ matrix.os }}/java ${{ matrix.java_version }}-spark-${{matrix.spark-patch-version}}-scala-${{matrix.scala-version}}/${{ matrix.test-target }}
runs-on: ${{ matrix.os }}
container:
image: amd64/rust
steps:
- uses: actions/checkout@v4
- name: Setup Rust & Java toolchain
uses: ./.github/actions/setup-builder
with:
rust-version: ${{env.RUST_VERSION}}
jdk-version: ${{ matrix.java_version }}
- if: matrix.test-target == 'rust'
name: Rust test steps
uses: ./.github/actions/rust-test
Comment on lines +105 to +107
Copy link
Copy Markdown
Contributor

@kazuyukitanimura kazuyukitanimura Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the rust test seems to be redundant, but ok

Copy link
Copy Markdown
Member Author

@andygrove andygrove Apr 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree. We also run the rust tests for multiple spark versions which is also redundant. I will create a follow-on PR to improve.

- if: matrix.test-target == 'java'
name: Java test steps
uses: ./.github/actions/java-test
with:
maven_opts: "-Pspark-${{ matrix.spark-version }},scala-${{ matrix.scala-version }} -Dspark.version=${{ matrix.spark-patch-version }}"
# upload test reports only for java 17
upload-test-reports: ${{ matrix.java_version == '17' }}

linux-test-native-datafusion-scan:
strategy:
matrix:
Expand Down
8 changes: 6 additions & 2 deletions docs/source/user-guide/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,15 @@ We recommend only using Comet with Spark versions where we currently have both C
Other versions may work well enough for development and evaluation purposes.

| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI |
| ------------- | ------------ | ------------- | ----------------- | ----------------- |
| ------------- | ------------ | ------------- |-------------------| ----------------- |
| 3.4.3 | 8/11/17 | 2.12/2.13 | Yes | Yes |
| 3.5.4 | 8/11/17 | 2.12/2.13 | No | Yes |
| 3.5.2 | 8/11/17 | 2.12/2.13 | Partial\* | No |
| 3.5.3 | 8/11/17 | 2.12/2.13 | Partial\* | No |
| 3.5.4 | 8/11/17 | 2.12/2.13 | Partial\* | Yes |
| 3.5.5 | 8/11/17 | 2.12/2.13 | Yes | Yes |

\* For older Spark 3.5.x releases, we do not test the full matrix of supported Java and Scala versions in CI.

Experimental support is provided for the following versions of Apache Spark and is intended for development/testing
use only and should not be used in production yet.

Expand Down
Loading