Make intermediate store for shuffle tasks an extension point

docs/configuration/index.md

@@ -1298,6 +1298,7 @@ Processing properties set on the Middlemanager will be passed through to Peons.
 |`druid.processing.columnCache.sizeBytes`|Maximum size in bytes for the dimension value lookup cache. Any value greater than `0` enables the cache. It is currently disabled by default. Enabling the lookup cache can significantly improve the performance of aggregators operating on dimension values, such as the JavaScript aggregator, or cardinality aggregator, but can slow things down if the cache hit rate is low (i.e. dimensions with few repeating values). Enabling it may also require additional garbage collection tuning to avoid long GC pauses.|`0` (disabled)|
 |`druid.processing.fifo`|If the processing queue should treat tasks of equal priority in a FIFO manner|`false`|
 |`druid.processing.tmpDir`|Path where temporary files created while processing a query should be stored. If specified, this configuration takes priority over the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`|
+|`druid.processing.intermediaryData.storage.type`|Storage type for storing intermediary segments of data shuffle between native parallel index tasks. Current choice are only "local" which stores segment files in local storage of Middle Managers (or Indexer).|local|
 
 The amount of direct memory needed by Druid is at least
 `druid.processing.buffer.sizeBytes * (druid.processing.numMergeBuffers + druid.processing.numThreads + 1)`. You can

indexing-service/src/main/java/org/apache/druid/indexing/common/task/batch/parallel/ShuffleClient.java

@@ -19,19 +19,21 @@
 
 package org.apache.druid.indexing.common.task.batch.parallel;
 
+import org.apache.druid.guice.annotations.ExtensionPoint;
 import org.apache.druid.indexing.worker.shuffle.IntermediaryDataManager;
 
 import java.io.File;
 import java.io.IOException;
 
 /**
  * An interface for intermediate data shuffle during the parallel indexing.
- * The only available implementation for production code is {@link HttpShuffleClient} and
- * this interface is more for easier testing.
+ *
+ * Extension can implement this interface to fetch intermediary data at custom location such as various cloud storages.
  *
  * @see IntermediaryDataManager
  * @see PartialSegmentMergeTask
  */
+@ExtensionPoint
 public interface ShuffleClient
 {
   /**