Flaky IT: perfect rollup parallel batch index integration test

[paul-rogers]

The IT test perfect rollup parallel batch index integration test failed. The PR in question did not change the indexer (or indeed, any existing code; it adds new, as-yet-unused code.) So, the failure is likely flaky.

2022-06-18T22:33:03,506 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up.  Failing.
java.lang.reflect.InvocationTargetException: null
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_332]
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_332]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_332]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_332]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-core-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-core-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-core-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:165) [druid-services-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:306) [druid-services-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.cli.Main.main(Main.java:114) [druid-services-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
Caused by: java.io.UncheckedIOException: /tmp/intermediary-segments/index_parallel_wikipedia_parallel_druid_input_source_index_test %Россия 한국 中国!?_ndiecicn_2022-06-18T22:26:38.659Z
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2153) ~[commons-io-2.11.0.jar:2.11.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:200) ~[druid-indexing-service-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) ~[?:1.8.0_332]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:197) ~[druid-indexing-service-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:134) ~[druid-indexing-service-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	... 10 more
Caused by: java.nio.file.NoSuchFileException: /tmp/intermediary-segments/index_parallel_wikipedia_parallel_druid_input_source_index_test %Россия 한국 中国!?_ndiecicn_2022-06-18T22:26:38.659Z
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) ~[?:1.8.0_332]

...

2022-06-18T22:38:25,438 ERROR [main] org.apache.druid.testing.utils.LoggerListener - Failed [org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData] with parameters [SingleDimensionPartitionsSpec{targetRowsPerSegment=2, maxRowsPerSegment=null, partitionDimension='namespace', assumeGrouped=false, resolvedMaxRowPerSegment=3}]
org.apache.druid.java.util.common.ISE: Error while making request to indexer [404 Not Found No task reports were found for this task. The task may not exist, or it may not have completed yet.]
	at org.apache.druid.testing.clients.OverlordResourceTestClient.makeRequest(OverlordResourceTestClient.java:727) ~[druid-integration-tests-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.testing.clients.OverlordResourceTestClient.getTaskReport(OverlordResourceTestClient.java:263) ~[druid-integration-tests-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.testing.clients.OverlordResourceTestClient.getTaskErrorMessage(OverlordResourceTestClient.java:246) ~[druid-integration-tests-0.24.0-SNAPSHOT.jar:0.24.0-SNAPSHOT]
	at org.apache.druid.tests.indexer.AbstractIndexerTest.unloadAndKillData(AbstractIndexerTest.java:91) ~[test-classes/:?]
	at org.apache.druid.tests.indexer.AbstractIndexerTest.lambda$unloader$0(AbstractIndexerTest.java:77) ~[test-classes/:?]
	at org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData(ITPerfectRollupParallelIndexTest.java:127) ~[test-classes/:?]

...

[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,815.156 s <<< FAILURE! - in TestSuite
[ERROR] testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest)  Time elapsed: 667.606 s
[ERROR] testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest)  Time elapsed: 576.529 s
[ERROR] testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest)  Time elapsed: 474.567 s  <<< FAILURE!
org.apache.druid.java.util.common.ISE: Error while making request to indexer [404 Not Found No task reports were found for this task. The task may not exist, or it may not have completed yet.]
	at org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData(ITPerfectRollupParallelIndexTest.java:127)
...

[ERROR] Failures: 
[ERROR] org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest.testIndexData(org.apache.druid.tests.indexer.ITPerfectRollupParallelIndexTest)
[INFO]   Run 1: PASS
[INFO]   Run 2: PASS
[ERROR]   Run 3: ITPerfectRollupParallelIndexTest.testIndexData:127->AbstractIndexerTest.lambda$unloader$0:77->AbstractIndexerTest.unloadAndKillData:91 » ISE

[abhishekagarwal87]

As I was debugging this, I found two problems with LocalIntermediaryDataManager.

• Both middle-manager and peons start LocalIntermediaryDataManager. There is no guard to protect from race conditions as multiple services on the same machine try to fiddle with local files. E.g. here is a potential race condition
  • Task Y starts iterating through all the supervisor task directories on local disk. There is no need to walk through all the supervisor task directories but LocalIntermediaryDataManager does not know whether it is running inside a peon or a middle manager. Task Y detects that there is a directory for Task X. It is about to iterate through the contents of X's dir.
  • Middle manager detects that X already completed and is expired. So it will delete the contents of X's dir.
  • Task Y gets an error during startup because X's dir cannot be found.
• Though looking further, it turns out the default expiration period for deleting intermediate data is 1 day. The test certainly did not take a day. so why did the MM delete the intermediate data at all? Weird. Another dead-end? Well, thankfully not. Poking around the code further, you find this line of code in LocalIntermediaryDataManager#deleteExpiredSuprevisorTaskPartitionsIfNotRunning

if (checkTime.isAfter(now)) {
       expiredSupervisorTasks.add(supervisorTaskId);
     }

checkTime is regularly updated and is usually set to lastAccess + expiryTime. The condition checkTime.isAfter(now) will always be true when expiryTime is a day. The above code snippet should really be as follows

if (now.isAfter(checkTime)) {
       expiredSupervisorTasks.add(supervisorTaskId);
     }

This bug has been around for ages now. It appears to be a simple fix but fixing it has implications. Because of this bug, stale intermediate data gets deleted in 5 minutes once the supervisor is completed. 5 minutes since that's the frequency at which expired data is checked. If we fix this bug, on all the druid installations out there, intermediate data will lie around for a day. That can be a problem. Especially on the clusters that have a lot of short-running batch tasks. We should definitely fix the peons to ignore the contents belonging to a different task group. For the expiration bug, we should fix it and reduce the default expiration time to 5 minutes in code.

[kfaraz]

Great find, @abhishekagarwal87 ! I agree with the fix for the expiration bug.

We should definitely fix the peons to ignore the contents belonging to a different task group.

This would make sense but there is an issue. The peon needs to be aware of the contents to know if it can reserve a location or not. In other words, the instance of StorageLocation must be updated with all the files that exist so that it knows how much space it can allocate for new partition files.

---

I thought of breaking up the IntermediaryDataManager into two parts, one which is used by the peon, solely to addSegment() and the other used by indexer/middle manager to perform discovery and cleanup. But there are two issues with this:

• As before, StorageLocation.reserve() would not work correctly for peon
• IntermediaryDataManager is an extension point, so it would not be desirable to change this contract

Also, there are some race conditions possible even outside the ones you mentioned. In particular, between two invocations of the discovery method, a task would have reserved space and exhausted the storage location which might cause another task to fail. But since the default discovery period is small enough (60s), it has not been an issue, I guess.

---

Proposal:
Expose a reserve API on the ShuffleResource (used by middle manager/indexer) and peons would call that to reserve a location.

• eliminates race conditions as middle manager becomes the sole storage resource manager
• peon logic would become much simpler
  • try to reserve using ShuffleResource API
  • if succeeds, add segment
  • if fails, try next location
  • does not need to keep track of anything else
  • does not perform cleanup or discovery
• we can continue to use the same IntermediaryDataManager contract

Cons:

• introduces new API call between peon and middle manager whenever pushing segment (not very frequent)
• the new reserve API is relevant only to local storage, as deep storage need not perform a reserve functionality (could be a noop for deep storage)

[paul-rogers]

@abhishekagarwal87, thanks much for diagnosing this issue!

Failed again in this build: 37272.52 (Compile=openjdk8, Run=openjdk8) perfect rollup parallel batch index integration test with Indexer

[layoaster]

This is still an issue in 2026, I'm on Druid v34.0.0 and my periodic reindexing task are also failing upon initialization because the MMs delete the intermediate data of previously run compaction tasks.

task_failed_logs.txt
task_failed_logs2.txt

I've attached the full log files for two affected tasks, but this is the error:

026-04-14T20:05:06,866 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Starting lifecycle [DiscoveryServiceLocator#start]
2026-04-14T20:05:06,867 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler - Starting lifecycle [LocalIntermediaryDataManager#start]
2026-04-14T20:05:06,869 ERROR [main] org.apache.druid.cli.CliPeon - Error when starting up.  Failing.
java.lang.reflect.InvocationTargetException: null
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.start(Lifecycle.java:446) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.java.util.common.lifecycle.Lifecycle.start(Lifecycle.java:341) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.guice.LifecycleModule$2.start(LifecycleModule.java:152) ~[druid-processing-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:137) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.GuiceRunnable.initLifecycle(GuiceRunnable.java:94) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.CliPeon.run(CliPeon.java:390) [druid-services-34.0.0.jar:34.0.0]
	at org.apache.druid.cli.Main.main(Main.java:112) [druid-services-34.0.0.jar:34.0.0]
Caused by: java.io.UncheckedIOException: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at org.apache.commons.io.function.Uncheck.wrap(Uncheck.java:339) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:199) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more
Caused by: java.nio.file.NoSuchFileException: /opt/druid/var/tmp/intermediary-segments/coordinator-issued_compact_sdk-emea-realtime_eipjffkn_2026-04-14T19:54:42.252Z
	at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]
	at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]
	at java.base/sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:55) ~[?:?]
	at java.base/sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:148) ~[?:?]
	at java.base/sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99) ~[?:?]
	at java.base/java.nio.file.Files.readAttributes(Files.java:1851) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:226) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:277) ~[?:?]
	at java.base/java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:323) ~[?:?]
	at java.base/java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:71) ~[?:?]
	at java.base/java.nio.file.Files.walk(Files.java:3918) ~[?:?]
	at org.apache.commons.io.file.PathUtils.walk(PathUtils.java:1847) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.streamFiles(FileUtils.java:2964) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.lambda$listFiles$10(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.function.Uncheck.get(Uncheck.java:197) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.commons.io.FileUtils.listFiles(FileUtils.java:2325) ~[commons-io-2.17.0.jar:2.17.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.lambda$discoverSupervisorTaskPartitions$3(LocalIntermediaryDataManager.java:198) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1708) ~[?:?]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.discoverSupervisorTaskPartitions(LocalIntermediaryDataManager.java:195) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	at org.apache.druid.indexing.worker.shuffle.LocalIntermediaryDataManager.start(LocalIntermediaryDataManager.java:135) ~[druid-indexing-service-34.0.0.jar:34.0.0]
	... 11 more