By default dont allow index_hadoop tasks to run on a cluster, forcing operators to acknowledge that they are using a deprecated feature#18239
Merged
cryptoe merged 6 commits intoapache:masterfrom Jul 18, 2025
Conversation
… operators to acknolwedge that they are using a deprecated feature
capistrant
commented
Jul 11, 2025
| } | ||
| log.warn("Running deprecated index_hadoop task [%s]. " | ||
| + "Hadoop indexing framework is deprecated and will be removed in a future release. " | ||
| + "Please migrate to the new indexing framework.", |
Contributor
Author
There was a problem hiding this comment.
trusted autocomplete from copilot for this msg 😆 should have read it closer. "the new indexing framework" is not useful. Will update
gianm
approved these changes
Jul 11, 2025
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+50
to
+59
| final HadoopIndexTask task = new HadoopIndexTask( | ||
| null, | ||
| new HadoopIngestionSpec( | ||
| DataSchema.builder() | ||
| .withDataSource("foo") | ||
| .withGranularity( | ||
| new UniformGranularitySpec( | ||
| Granularities.DAY, | ||
| null, | ||
| ImmutableList.of(Intervals.of("2010-01-01/P1D")) |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation Note test
capistrant
added a commit
to capistrant/incubator-druid
that referenced
this pull request
Jul 18, 2025
… operators to acknowledge that they are using a deprecated feature (apache#18239) * By default dont allow index_hadoop tasks to run on a cluster, forcing operators to acknolwedge that they are using a deprecated feature * update unclear recommendation from log * Fixup codeql warning * fix UT
ashibhardwaj
pushed a commit
to ashibhardwaj/druid
that referenced
this pull request
Jul 23, 2025
… operators to acknowledge that they are using a deprecated feature (apache#18239) * By default dont allow index_hadoop tasks to run on a cluster, forcing operators to acknolwedge that they are using a deprecated feature * update unclear recommendation from log * Fixup codeql warning * fix UT
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Druid 34 publicly deprecating
index_hadooptype tasks. The eventual removal ofindex_hadoopis a big change and will force any users of the task type to make a large change to their cluster operations. This PR aims to ensure no operator is caught off guard by the deprecation and planned removal. It will failindex_hadooptasks with an error stating the why, unless the operator updates their runtime configs to allowindex_hadoopAlternate approach
I'm sure we could shift the failure left and fail before the overlord even tries to submit the task.
Release note
Druid cluster operators must opt-in to using the now deprecated
index_hadooptask type in their Druid clusters. If you wish to be able to continue submittingindex_hadooptyped tasks, please set the following runtime property totrue:druid.indexer.task.allowHadoopTaskExecutionNote that this property needs to be set in the local context of your running ingest task. The easiest way to achieve this is to set it in
common.runtime.properties.Key changed/added classes in this PR
TaskConfigIndexHadoopTaskThis PR has: