From d68c702f7ed66732937c53f65170632176ebaf6a Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Fri, 8 Aug 2025 12:37:16 -0700 Subject: [PATCH] docs: add deprecation for hadoop and java11 (#18286) Co-authored-by: Victoria Lim Co-authored-by: Lucas Capistrant --- docs/ingestion/data-formats.md | 10 ++++++++-- docs/ingestion/faq.md | 4 ---- docs/ingestion/hadoop.md | 9 +++++++++ docs/ingestion/index.md | 5 ++--- docs/operations/java.md | 8 ++------ docs/operations/other-hadoop.md | 8 ++++++++ docs/tutorials/cluster.md | 4 ++-- docs/tutorials/index.md | 4 ++-- docs/tutorials/tutorial-batch-hadoop.md | 7 +++++++ docs/tutorials/tutorial-kerberos-hadoop.md | 8 ++++++++ docs/tutorials/tutorial-query.md | 1 - 11 files changed, 48 insertions(+), 20 deletions(-) diff --git a/docs/ingestion/data-formats.md b/docs/ingestion/data-formats.md index 1bb2de9918ff..a595b1c6ded7 100644 --- a/docs/ingestion/data-formats.md +++ b/docs/ingestion/data-formats.md @@ -962,11 +962,17 @@ Each line can be further parsed using [`parseSpec`](#parsespec). ### Avro Hadoop Parser -:::info - You need to include the [`druid-avro-extensions`](../development/extensions-core/avro.md) as an extension to use the Avro Hadoop Parser. +:::caution[Deprecated] + +Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) + +You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239) + ::: :::info +You need to include [`druid-avro-extensions`](../development/extensions-core/avro.md) as an extension to use the Avro Hadoop Parser. + See the [Avro Types](../development/extensions-core/avro.md#avro-types) section for how Avro types are handled in Druid ::: diff --git a/docs/ingestion/faq.md b/docs/ingestion/faq.md index 24e119585ab6..3fab83f0ea99 100644 --- a/docs/ingestion/faq.md +++ b/docs/ingestion/faq.md @@ -49,10 +49,6 @@ Other common reasons that hand-off fails are as follows: 4) Deep storage is improperly configured. Make sure that your segment actually exists in deep storage and that the Coordinator logs have no errors. -## How do I get HDFS to work? - -Make sure to include the `druid-hdfs-storage` and all the hadoop configuration, dependencies (that can be obtained by running command `hadoop classpath` on a machine where hadoop has been setup) in the classpath. And, provide necessary HDFS settings as described in [deep storage](../design/deep-storage.md) . - ## How do I know when I can make query to Druid after submitting batch ingestion task? You can verify if segments created by a recent ingestion task are loaded onto historicals and available for querying using the following workflow. diff --git a/docs/ingestion/hadoop.md b/docs/ingestion/hadoop.md index db665f9769db..3dd738f78910 100644 --- a/docs/ingestion/hadoop.md +++ b/docs/ingestion/hadoop.md @@ -23,6 +23,15 @@ sidebar_label: "Hadoop-based" ~ under the License. --> +:::caution[Deprecated] + +Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) + +You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239) + +::: + + Apache Hadoop-based batch ingestion in Apache Druid is supported via a Hadoop-ingestion task. These tasks can be posted to a running instance of a Druid [Overlord](../design/overlord.md). Please refer to our [Hadoop-based vs. native batch comparison table](index.md#batch) for comparisons between Hadoop-based, native batch (simple), and native batch (parallel) ingestion. diff --git a/docs/ingestion/index.md b/docs/ingestion/index.md index b2c9002df927..de90051fca09 100644 --- a/docs/ingestion/index.md +++ b/docs/ingestion/index.md @@ -28,8 +28,7 @@ your source system and stores it in data files called [_segments_](../design/seg In general, segment files contain a few million rows each. For most ingestion methods, the Druid [Middle Manager](../design/middlemanager.md) processes or the -[Indexer](../design/indexer.md) processes load your source data. The sole exception is Hadoop-based ingestion, which -uses a Hadoop MapReduce job on YARN. +[Indexer](../design/indexer.md) processes load your source data. During ingestion, Druid creates segments and stores them in [deep storage](../design/deep-storage.md). Historical nodes load the segments into memory to respond to queries. For streaming ingestion, the Middle Managers and indexers can respond to queries in real-time with arriving data. For more information, see [Storage overview](../design/storage.md). @@ -66,7 +65,7 @@ supervisor. There are three available options for batch ingestion. Batch ingestion jobs are associated with a controller task that runs for the duration of the job. -| **Method** | [Native batch](./native-batch.md) | [SQL](../multi-stage-query/index.md) | [Hadoop-based](hadoop.md) | +| **Method** | [Native batch](./native-batch.md) | [SQL](../multi-stage-query/index.md) | [Hadoop-based (deprecated)](hadoop.md) | |---|-----|--------------|------------| | **Controller task type** | `index_parallel` | `query_controller` | `index_hadoop` | | **How you submit it** | Send an `index_parallel` spec to the [Tasks API](../api-reference/tasks-api.md). | Send an [INSERT](../multi-stage-query/concepts.md#load-data-with-insert) or [REPLACE](../multi-stage-query/concepts.md#overwrite-data-with-replace) statement to the [SQL task API](../api-reference/sql-ingestion-api.md#submit-a-query). | Send an `index_hadoop` spec to the [Tasks API](../api-reference/tasks-api.md). | diff --git a/docs/operations/java.md b/docs/operations/java.md index d16f78c8abe1..0b8a474ac950 100644 --- a/docs/operations/java.md +++ b/docs/operations/java.md @@ -27,11 +27,7 @@ a Java runtime for Druid. ## Selecting a Java runtime -Druid fully supports Java 11 and Java 17. The project team recommends Java 17. - -:::info -Note: Starting with Apache Druid 32.0.0, support for Java 8 has been removed. -::: + The project team recommends Java 17. Although you can use Java 11, support for it is deprecated. The project team recommends using an OpenJDK-based Java distribution. There are many free and actively-supported distributions available, including @@ -74,7 +70,7 @@ Exception in thread "main" java.lang.ExceptionInInitializerError ``` Druid's out-of-box configuration adds these parameters transparently when you use the bundled `bin/start-druid` or -similar commands. In this case, there is nothing special you need to do to run successfully on Java 11 or 17. However, +similar commands. In this case, there is nothing special you need to do to run successfully. However, if you have customized your Druid service launching system, you will need to ensure the required Java parameters are added. There are many ways of doing this. Choose the one that works best for you. diff --git a/docs/operations/other-hadoop.md b/docs/operations/other-hadoop.md index ba19a8326435..a82b331de4bb 100644 --- a/docs/operations/other-hadoop.md +++ b/docs/operations/other-hadoop.md @@ -23,6 +23,14 @@ title: "Working with different versions of Apache Hadoop" --> +:::caution[Deprecated] + +Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) + +You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239) + +::: + Apache Druid can interact with Hadoop in two ways: 1. [Use HDFS for deep storage](../development/extensions-core/hdfs.md) using the druid-hdfs-storage extension. diff --git a/docs/tutorials/cluster.md b/docs/tutorials/cluster.md index f2128489216c..cd435c5e1ceb 100644 --- a/docs/tutorials/cluster.md +++ b/docs/tutorials/cluster.md @@ -133,8 +133,8 @@ The [basic cluster tuning guide](../operations/basic-cluster-tuning.md) has info We recommend running your favorite Linux distribution. You will also need -* [Java 11 or 17](../operations/java.md) -* Python 2 or Python 3 +* [Java 17](../operations/java.md) +* Python 3 :::info If needed, you can specify where to find Java using the environment variables diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index 187f3cb952b0..f55b26e60f6c 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -40,8 +40,8 @@ You can follow these steps on a relatively modest machine, such as a workstation The software requirements for the installation machine are: * Linux, Mac OS X, or other Unix-like OS. (Windows is not supported) -* [Java 11 or 17](../operations/java.md) -* Python 3 (preferred) or Python 2 +* [Java 17](../operations/java.md) +* Python 3 * Perl 5 Java must be available. Either it is on your path, or set one of the `JAVA_HOME` or `DRUID_JAVA_HOME` environment variables. diff --git a/docs/tutorials/tutorial-batch-hadoop.md b/docs/tutorials/tutorial-batch-hadoop.md index a71823544af5..c75fc7d35e89 100644 --- a/docs/tutorials/tutorial-batch-hadoop.md +++ b/docs/tutorials/tutorial-batch-hadoop.md @@ -23,6 +23,13 @@ sidebar_label: Load from Apache Hadoop ~ under the License. --> +:::caution[Deprecated] + +Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) + +You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239) + +::: This tutorial shows you how to load data files into Apache Druid using a remote Hadoop cluster. diff --git a/docs/tutorials/tutorial-kerberos-hadoop.md b/docs/tutorials/tutorial-kerberos-hadoop.md index 24fc290b6a6d..cace9b8794f4 100644 --- a/docs/tutorials/tutorial-kerberos-hadoop.md +++ b/docs/tutorials/tutorial-kerberos-hadoop.md @@ -23,6 +23,14 @@ sidebar_label: Kerberized HDFS deep storage ~ under the License. --> +:::caution[Deprecated] + +Hadoop-based ingestion is deprecated. We recommend one of Druid's other supported ingestion methods, such as [SQL-based ingestion](../multi-stage-query/index.md) or [MiddleManager-less ingestion using Kubernetes](../development/extensions-core/k8s-jobs.md) + +You must now explicitly opt-in to using the deprecated `index_hadoop` task type. To opt-in, set `druid.indexer.task.allowHadoopTaskExecution` to `true` in your `common.runtime.properties` file. For more information, see [#18239](https://github.com/apache/druid/pull/18239) + +::: + ## Hadoop Setup diff --git a/docs/tutorials/tutorial-query.md b/docs/tutorials/tutorial-query.md index 54563513a483..66b38974db2a 100644 --- a/docs/tutorials/tutorial-query.md +++ b/docs/tutorials/tutorial-query.md @@ -32,7 +32,6 @@ by following one of them: * [Load a file](../tutorials/tutorial-batch.md) * [Load stream data from Kafka](../tutorials/tutorial-kafka.md) -* [Load a file using Hadoop](../tutorials/tutorial-batch-hadoop.md) There are various ways to run Druid SQL queries: from the web console, using a command line utility and by posting the query by HTTP. We'll look at each of these.