From c6c1f628726f347647c50a3ca02e74ee561c8856 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 09:35:13 -0600 Subject: [PATCH 1/7] update supported Spark versions --- docs/source/user-guide/installation.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index 88b6511316..d64c21cddd 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -30,22 +30,31 @@ Make sure the following requirements are met and software installed on your mach ### Supported Spark Versions -Comet currently supports the following versions of Apache Spark: +Comet currently supports the following versions of Apache Spark. Note that this documentation is for the main +branch. Please refer to GitHub history for released Comet versions. -- 3.4.x (Java 8/11/17, Scala 2.12/2.13) -- 3.5.x (Java 8/11/17, Scala 2.12/2.13) +We recommend only using Comet with Spark versions where we currently have tests enabled in CI. Other versions may work +well enough for development and evaluation purposes. + +| Spark Version | Java Version | Scala Version | Tested in CI | +| ------------- | ------------ | --------------- | ------------ | +| 3.4.3 | 8/11/17 | Scala 2.12/2.13 | Yes | +| 3.5.4 | 8/11/17 | Scala 2.12/2.13 | Partially | +| 3.5.5 | 8/11/17 | Scala 2.12/2.13 | Yes | Experimental support is provided for the following versions of Apache Spark and is intended for development/testing use only and should not be used in production yet. -- 4.0.0-preview1 (Java 17/21, Scala 2.13) +| Spark Version | Java Version | Scala Version | Tested in CI | +| -------------- | ------------ | ------------- | ------------ | +| 4.0.0-preview1 | 11/17 | Scala 2.13 | Yes | Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers. ## Using a Published JAR File -Comet jar files are available in [Maven Central](https://central.sonatype.com/namespace/org.apache.datafusion) for amd64 and arm64 architectures for Linux. For Apple OSX, it +Comet jar files are available in [Maven Central](https://central.sonatype.com/namespace/org.apache.datafusion) for amd64 and arm64 architectures for Linux. For Apple OSX, it is currently necessary to build from source. Here are the direct links for downloading the Comet 0.7.0 jar file. From 593684ee4d7abae4ce2c2847326782f0100a820a Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 09:38:13 -0600 Subject: [PATCH 2/7] update --- docs/source/user-guide/installation.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index d64c21cddd..f662af739d 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -36,18 +36,18 @@ branch. Please refer to GitHub history for released Comet versions. We recommend only using Comet with Spark versions where we currently have tests enabled in CI. Other versions may work well enough for development and evaluation purposes. -| Spark Version | Java Version | Scala Version | Tested in CI | -| ------------- | ------------ | --------------- | ------------ | -| 3.4.3 | 8/11/17 | Scala 2.12/2.13 | Yes | -| 3.5.4 | 8/11/17 | Scala 2.12/2.13 | Partially | -| 3.5.5 | 8/11/17 | Scala 2.12/2.13 | Yes | +| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | +| ------------- | ------------ | --------------- | ----------------- | ----------------- | +| 3.4.3 | 8/11/17 | Scala 2.12/2.13 | Yes | Yes | +| 3.5.4 | 8/11/17 | Scala 2.12/2.13 | No | Yes | +| 3.5.5 | 8/11/17 | Scala 2.12/2.13 | Yes | Yes | Experimental support is provided for the following versions of Apache Spark and is intended for development/testing use only and should not be used in production yet. -| Spark Version | Java Version | Scala Version | Tested in CI | -| -------------- | ------------ | ------------- | ------------ | -| 4.0.0-preview1 | 11/17 | Scala 2.13 | Yes | +| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | +| -------------- | ------------ | ------------- | ----------------- | ----------------- | +| 4.0.0-preview1 | 11/17 | Scala 2.13 | Yes | Yes | Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers. From 7c50c4bf27d4be843ef79acecd3c22ef18f1c756 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 09:38:29 -0600 Subject: [PATCH 3/7] update --- docs/source/user-guide/installation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index f662af739d..4e214dcc28 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -33,8 +33,8 @@ Make sure the following requirements are met and software installed on your mach Comet currently supports the following versions of Apache Spark. Note that this documentation is for the main branch. Please refer to GitHub history for released Comet versions. -We recommend only using Comet with Spark versions where we currently have tests enabled in CI. Other versions may work -well enough for development and evaluation purposes. +We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI. +Other versions may work well enough for development and evaluation purposes. | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | | ------------- | ------------ | --------------- | ----------------- | ----------------- | From ca624929671bf80ee8d945198f1f46cade5f7d29 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 09:40:45 -0600 Subject: [PATCH 4/7] update --- docs/source/user-guide/installation.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index 4e214dcc28..d76eabae5e 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -36,18 +36,18 @@ branch. Please refer to GitHub history for released Comet versions. We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI. Other versions may work well enough for development and evaluation purposes. -| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | -| ------------- | ------------ | --------------- | ----------------- | ----------------- | -| 3.4.3 | 8/11/17 | Scala 2.12/2.13 | Yes | Yes | -| 3.5.4 | 8/11/17 | Scala 2.12/2.13 | No | Yes | -| 3.5.5 | 8/11/17 | Scala 2.12/2.13 | Yes | Yes | +| Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | +| ------------- | ------------ |---------------| ----------------- | ----------------- | +| 3.4.3 | 8/11/17 | 2.12/2.13 | Yes | Yes | +| 3.5.4 | 8/11/17 | 2.12/2.13 | No | Yes | +| 3.5.5 | 8/11/17 | 2.12/2.13 | Yes | Yes | Experimental support is provided for the following versions of Apache Spark and is intended for development/testing use only and should not be used in production yet. | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | | -------------- | ------------ | ------------- | ----------------- | ----------------- | -| 4.0.0-preview1 | 11/17 | Scala 2.13 | Yes | Yes | +| 4.0.0-preview1 | 11/17 | 2.13 | Yes | Yes | Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers. From 4f64e725007e08e76ff2000662ed7b1c25cf02b3 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 09:41:05 -0600 Subject: [PATCH 5/7] update --- docs/source/user-guide/installation.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index d76eabae5e..884a3bdf11 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -33,11 +33,11 @@ Make sure the following requirements are met and software installed on your mach Comet currently supports the following versions of Apache Spark. Note that this documentation is for the main branch. Please refer to GitHub history for released Comet versions. -We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI. +We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI. Other versions may work well enough for development and evaluation purposes. | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | -| ------------- | ------------ |---------------| ----------------- | ----------------- | +| ------------- | ------------ | ------------- | ----------------- | ----------------- | | 3.4.3 | 8/11/17 | 2.12/2.13 | Yes | Yes | | 3.5.4 | 8/11/17 | 2.12/2.13 | No | Yes | | 3.5.5 | 8/11/17 | 2.12/2.13 | Yes | Yes | @@ -47,7 +47,7 @@ use only and should not be used in production yet. | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | | -------------- | ------------ | ------------- | ----------------- | ----------------- | -| 4.0.0-preview1 | 11/17 | 2.13 | Yes | Yes | +| 4.0.0-preview1 | 11/17 | 2.13 | Yes | Yes | Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers. From f02a2a24e1a98b1f75862a0e3f17d4706599d4f4 Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 09:54:28 -0600 Subject: [PATCH 6/7] update --- docs/source/user-guide/installation.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index 884a3bdf11..77645a1c51 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -31,7 +31,8 @@ Make sure the following requirements are met and software installed on your mach ### Supported Spark Versions Comet currently supports the following versions of Apache Spark. Note that this documentation is for the main -branch. Please refer to GitHub history for released Comet versions. +branch. Please refer to the version of this documentation in the appropriate GitHub release branch for released +Comet versions. We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI. Other versions may work well enough for development and evaluation purposes. From ddaa60138dda99ba99b9e044bffe6c18ffdc61ff Mon Sep 17 00:00:00 2001 From: Andy Grove Date: Mon, 31 Mar 2025 15:44:53 -0600 Subject: [PATCH 7/7] address feedback --- docs/source/user-guide/installation.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/source/user-guide/installation.md b/docs/source/user-guide/installation.md index 77645a1c51..6e57407d9c 100644 --- a/docs/source/user-guide/installation.md +++ b/docs/source/user-guide/installation.md @@ -31,7 +31,7 @@ Make sure the following requirements are met and software installed on your mach ### Supported Spark Versions Comet currently supports the following versions of Apache Spark. Note that this documentation is for the main -branch. Please refer to the version of this documentation in the appropriate GitHub release branch for released +branch. Please refer to the version of this documentation in the appropriate GitHub release branch for released Comet versions. We recommend only using Comet with Spark versions where we currently have both Comet and Spark tests enabled in CI. @@ -48,7 +48,7 @@ use only and should not be used in production yet. | Spark Version | Java Version | Scala Version | Comet Tests in CI | Spark Tests in CI | | -------------- | ------------ | ------------- | ----------------- | ----------------- | -| 4.0.0-preview1 | 11/17 | 2.13 | Yes | Yes | +| 4.0.0-preview1 | 17 | 2.13 | Yes | Yes | Note that Comet may not fully work with proprietary forks of Apache Spark such as the Spark versions offered by Cloud Service Providers.