Skip to content

chore: Fix jdk documentation for spark #979

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adi-kmt wants to merge 5 commits into from
Closed

chore: Fix jdk documentation for spark #979

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
4cc4492
Fix jdk documentation for spark
Sep 28, 2024
0506224
Fixing appache spark md link
adi-kmt Sep 29, 2024
fb23d21
Adding spark documentation link to readme
Sep 29, 2024
e2cd615
Fixing java version based on the spark preview documentation
adi-kmt Sep 29, 2024
46408e0
Merge branch 'main' into documentation_jdk_version_fix
adi-kmt Oct 18, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions docs/source/user-guide/installation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,22 @@ Make sure the following requirements are met and software installed on your mach

### Supported Spark Versions
Copy link
Copy Markdown
Contributor

@comphead comphead Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
### Supported Spark Versions


Comet currently supports the following versions of Apache Spark:
### Supported Apache Spark Versions
For more details, refer to the documentation: [Apache Spark supported by Comet](overview.md#supported-apache-spark-versions)

- **Apache Spark 3.3, 3.4, or 3.5**
Copy link
Copy Markdown
Contributor

@comphead comphead Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would like to remove duplications in doc for Supported Apache Versions @adi-kmt Is there any specific reason of having this duplication?

Copy link
Copy Markdown
Author

@adi-kmt adi-kmt Oct 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Main motive was to indicate the different jdk versions wrt to the apache spark versions. Do you think it could be written in a better way?

- **Supported Java Versions**: JDK 8 and above
- **GLIBC**: 2.17 (CentOS 7) and above
Copy link
Copy Markdown
Contributor

@parthchandra parthchandra Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our latest release is built on Ubuntu 20.04 which has glibc 2.31. Could we update that here?

Copy link
Copy Markdown
Member

@viirya viirya Sep 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For pre-built binary release, we can mention what glibc we use. Although I think it doesn't mean older glibc is not supported, users still can build by themselves and use older glibc.

- **Scala**: 2.12/2.13
- **Apache Spark 4.0** *(Experimental Support - is intended for development/testing use only and should not be used in production yet.)*
- **Supported Java Versions**: JDK 17/21
- **GLIBC**: 2.17 (CentOS 7) and above
Copy link
Copy Markdown
Contributor

@comphead comphead Oct 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for GLIBC Im thinking if its good to specify a Linux distribution... I would leave just a version and put a command to check GLIBC like ldd --version

- **Scala**: 2.13

- 3.3.x (Java 8/11/17, Scala 2.12/2.13)
- 3.4.x (Java 8/11/17, Scala 2.12/2.13)
- 3.5.x (Java 8/11/17, Scala 2.12/2.13)

Experimental support is provided for the following versions of Apache Spark and is intended for development/testing
use only and should not be used in production yet.

- 4.0.0-preview1 (Java 17/21, Scala 2.13)

Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by
Cloud Service Providers.

Expand Down