Skip to content

docs: Update supported Spark versions #1580

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kazuyukitanimura merged 7 commits into from
Mar 31, 2025
Merged

docs: Update supported Spark versions #1580

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c6c1f62
update supported Spark versions
andygrove Mar 31, 2025
593684e
update
andygrove Mar 31, 2025
7c50c4b
update
andygrove Mar 31, 2025
ca62492
update
andygrove Mar 31, 2025
4f64e72
update
andygrove Mar 31, 2025
f02a2a2
update
andygrove Mar 31, 2025
ddaa601
address feedback
andygrove Mar 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 15 additions & 5 deletions docs/source/user-guide/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,22 +30,32 @@ Make sure the following requirements are met and software installed on your mach

### Supported Spark Versions

Comet currently supports the following versions of Apache Spark:
Comet currently supports the following versions of Apache Spark. Note that this documentation is for the main
branch. Please refer to the version of this documentation in the appropriate GitHub release branch for released
Comet versions.

- 3.4.x (Java 8/11/17, Scala 2.12/2.13)
- 3.5.x (Java 8/11/17, Scala 2.12/2.13)
We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI.
Other versions may work well enough for development and evaluation purposes.

| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI |
| ------------- | ------------ | ------------- | ----------------- | ----------------- |
| 3.4.3 | 8/11/17 | 2.12/2.13 | Yes | Yes |
| 3.5.4 | 8/11/17 | 2.12/2.13 | No | Yes |
| 3.5.5 | 8/11/17 | 2.12/2.13 | Yes | Yes |

Experimental support is provided for the following versions of Apache Spark and is intended for development/testing
use only and should not be used in production yet.

- 4.0.0-preview1 (Java 17/21, Scala 2.13)
| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI |
| -------------- | ------------ | ------------- | ----------------- | ----------------- |
| 4.0.0-preview1 | 17 | 2.13 | Yes | Yes |

Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by
Cloud Service Providers.

## Using a Published JAR File

Comet jar files are available in [Maven Central](https://central.sonatype.com/namespace/org.apache.datafusion) for amd64 and arm64 architectures for Linux. For Apple OSX, it
Comet jar files are available in [Maven Central](https://central.sonatype.com/namespace/org.apache.datafusion) for amd64 and arm64 architectures for Linux. For Apple OSX, it
is currently necessary to build from source.

Here are the direct links for downloading the Comet 0.7.0 jar file.
Expand Down