Segments are not cleaned up properly after unannouncing them in HTTP-based segment loading if numLoadingThreads > 1

[jihoonson]

Affected Version

0.12, 0.13, 0.14

Description

When numLoadingThread > 1, the same segment can be loaded by the same historical more than once. This can cause a race between threads. Multiple threads can load the same segment and announce it in different ZooKeeper paths, but the historical removes only one of the ZooKeeper paths.

Here is a historical log which shows the issue.

@400000005c76725b1f639b3c.s:2019-02-27T09:01:30,979 INFO [SimpleDataSegmentChangeHandler-5] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[ads_events_2019
-02-01T07:00:00.000Z_2019-02-01T08:00:00.000Z_2019-02-01T08:52:15.391Z] at existing path[/druid/segments/ip-172-16-27-17.ec2.internal:8283/ip-172-16-27-17.ec2.internal:8283_historical_tier2_2
019-01-19T22:15:15.984Z_3383e66491e14925a778e438000fa25c4]
@400000005c76725b1f639b3c.s:2019-02-27T09:01:30,980 INFO [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Announcing segment[ads_events_2019
-02-01T07:00:00.000Z_2019-02-01T08:00:00.000Z_2019-02-01T08:52:15.391Z] at existing path[/druid/segments/ip-172-16-27-17.ec2.internal:8283/ip-172-16-27-17.ec2.internal:8283_historical_tier2_2
019-01-19T22:16:21.423Z_cb3afea2cf0a4c04bc4f47220510d6385]
@400000005c76725b1f639b3c.s:2019-02-27T09:02:17,134 INFO [qtp353841915-200] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - Unannouncing segment[ads_events_2019-02-01T07:00:00.000Z_2019-02-01T08:00:00.000Z_2019-02-01T08:52:15.391Z] at path[/druid/segments/ip-172-16-27-17.ec2.internal:8283/ip-172-16-27-17.ec2.internal:8283_historical_tier2_2019-01-19T22:16:21.423Z_cb3afea2cf0a4c04bc4f47220510d6385]

I think the root problem is that different threads can load the same segment at the same time which we should fix. But, it would also be good if the historical cleans up properly even if this kind of error happens because it's not easy to even notice the problem exists until we check query results carefully.

[jihoonson]

This would be fixed in 0.15 because HTTP-based segment loading is an experimental feature.

[jihoonson]

I raised #7203 to fix the problem of only historical side. I'm still not sure how to improve the segment assigning protocol to avoid duplicates.

[jihoonson]

I don't think this bug is a blocker for 0.15.0 release because it was partially fixed in #7203. I'm rescheduling this to 0.16.0.

[jihoonson]

This bug is partially fixed and I don't think it should be a release blocker. Taking the milestone out of this issue.

[himanshug]

It is indeed possible for HttpLoadQueuePeon to retry if request to load segment by historical failed for whatever reason (but historical got the request and started loading the segment). but it maybe possible that there is some bug as well. That said, Retrying needs to be part of protocol and expectation is that historical would be able to take care of duplication which I think is handled by #7203

Multiple threads can load the same segment and announce it in different ZooKeeper paths

that looks like a bug in BatchDataSegmentAnnouncer which does appear to have necessary synchronization and deduplication code.

that said, above problem does not exist at all if HTTP based segment discovery is used . historicals could be told to not announce any segments in zookeeper then ( druid.announcer.skipSegmentAnnouncementOnZk=false )

I think we should just switch to HTTP for both segment load/drop and discovery when we switch.

sidenote: I have been using HTTP based segment management, discovery and HTTP based task management (with patch #8427 ) on my cluster since long time.