Coordinator Segment Handoff - Is it possible to prioritize new segments from ingestion tasks?

[pchang388]

Apache Druid Version: 0.23.0

Problem:
Sometimes when we have infrastructure issues, we have to decommission some Historicals and temporarily lower retention period (to not run out of disk on running ones). When we bring back the Historicals and re-apply our old retention period and reload the dropped data, our ingestion tasks wait for handoff much longer and sometimes fail because they did not handoff completely in the completionTimeout phase even if that is set to a higher number.

Is it possible to adjust Coordinator behavior to prioritize new segment handoffs from currently running ingestion tasks over reloading dropped data so that running ingestion tasks are not affected and still have some segments being reloaded?

[abhishekagarwal87]

this seems like a genuine problem. @kfaraz is working on a proposal to improve the segment balancing - #12881. maybe this issue can be taken up too as part of that proposal.

[pchang388]

Another issue we noticed during this week of infrastructure issues. When we bring down the historicals from one site, the others get their data loaded. We also switched retention period to a lower number (365 -> 90 days) to not run out of disks on the working historicals right before decommissioning the bad historicals.

But it appears the Coordinator will load the data first from the other historicals that were decommissioned before dropping the existing data that is now out of the new retention period:

2022-08-17T20:01:01,893 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Skipping replica assignment for tier [_default_tier]
2022-08-17T20:01:01,893 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Loading in progress, skipping drop until loading is complete

This means that the historicals will fill up first before the drop for retention goes through since it tries to load the data first. Is there any way we can avoid this?

It's weird, doesn't always seem to happen. Might be an order of operations thing or timing. But restarting the coordinator and historicals seems to get it to realize that it needs to finally drop segments instead of just load

[pchang388]

For the original problem, here's an example showing a rebalancing going on after historicals are brought back up in this case after infrastructure issues are resolved and retention period is changed back from PT90D to PT365D in our case:

Task example shows exceeding the default completionTimeout (PT30M) and failing due to unable to handoff during segment rebalancing

2022-08-18T19:39:17,730 INFO [[index_kafka_REDACT_72177abfbb559eb_anenendi]-publish] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Saved sequence metadata to disk: []
2022-08-18T19:39:54,131 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:40:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:41:54,131 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:42:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:43:54,134 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:44:54,138 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:45:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:46:54,134 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:47:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:48:54,130 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:49:54,135 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:50:54,130 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:51:54,134 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:52:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:53:54,131 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:54:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:55:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:56:54,371 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:57:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:58:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T19:59:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:00:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:01:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:02:54,130 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:03:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:04:54,135 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:05:54,133 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:06:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:07:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:08:54,132 WARN [coordinator_handoff_scheduled_0] org.apache.druid.segment.handoff.CoordinatorBasedSegmentHandoffNotifier - Still waiting for Handoff for [2] Segments
2022-08-18T20:09:00,332 INFO [parent-monitor-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Triggering JVM shutdown.

Exact order of operations - Hourly ingestion tasks are running through this time:

1. Pause Coordinator = True in Dynamic Config
2. Bring the recently decomissioned Historicals back up
3. Change Retention period from PT90D to PT365D (365 is our normal SLA/promise for retention. It was lowered to 90D during the infrastructure issues in one site to not fill up disks on the remaining site)
4. Pause Coordinator = False in Dynamic Config, begin rebalancing and handoffs again

Hope this provides more information/motivation, thank you again to @abhishekagarwal87 and @kfaraz for looking into this

[danprince1]

We've had this running in a big cluster in production for a long time and it's been solid. I couldn't get traction on it when I submitted it; maybe I'll try again. #12013

[abhishekagarwal87]

@pchang388 @danprince1 - You could check out #13367 to see if that will help with the issues you are facing.

[kfaraz]

@danprince1 , @pchang388 , thanks for reporting the issue.
There are a set of segment loading changes in #13197 that are currently in review. That PR addresses this and several other issues with segment loading and balancing.

That said, the original issue described here should not really be a problem since newer segments are already prioritized over older segments while loading even in the current code.
References:

druid/server/src/main/java/org/apache/druid/server/coordinator/HttpLoadQueuePeon.java

Line 86 in 309cae7

DruidCoordinator.SEGMENT_COMPARATOR_RECENT_FIRST

druid/server/src/main/java/org/apache/druid/server/coordinator/DruidCoordinatorRuntimeParams.java

Line 52 in 309cae7

private static TreeSet<DataSegment> createUsedSegmentsSet(Iterable<DataSegment> usedSegments)

My guess is that the prioritization is not the issue, the root cause is perhaps that the coordinator is just trying to load everything that is missing and taking very long to do that. If the coordinator gets stuck in a cycle, it doesn't know of new segments that the tasks are publishing and fails to make them available quickly. This problem is being addressed in the mentioned set of changes.

Also, as @abhishekagarwal87 mentioned, #13367 is meant to speed up assignments. Since it is a coordinator dynamic config, you can safely enable and disable it during coordinator operations. The suggested approach would be to enable it before spawning a new historical so that assignments happen quickly.
Once normal operation has resumed, you can choose to disable the round-robin or let it run if your historicals don't seem too out-of-balance.

[kfaraz]

But it appears the Coordinator will load the data first from the other historicals that were decommissioned before dropping the existing data that is now out of the new retention period:

This is also a known issue proposed in #12881 and being addressed in #13197 .