Kill segments by versions by abhishekrb19 · Pull Request #15994 · apache/druid

abhishekrb19 · 2024-02-28T08:49:21Z

Summary:

Prior to this patch, kill tasks will delete all versions of unused segments within the specified interval. With this patch, users can now delete specific versions of unused data, while retaining the rest by specifying an optional list of versions in the kill task payload. If left unspecified, the default behavior remains unchanged, i.e., delete all versions of unused segments in the interval.

Motivation:

A user may want to use this functionality for data compliance reasons - an ingestion job that created some bad data that needs to be deleted right away, while keeping other versions of the data around for sometime.
Manage storage costs: Keep only the last 'n' versions of unused segments in the deep storage.

Note that adding an optional list of versions support to /markAsUsed and /markAsUnused segment management APIs would be a complementary addition. I didn't make those changes in this PR to keep it simple to review - I will follow up on that later.

Release note

Kill task accepts an optional list of unused segment versions to delete.

Key changed/added classes in this PR

KillUnusedSegmentsTask.java
IndexerSQLMetadataStorageCoordinator.java
RetrieveUnusedSegmentsAction.java
KillUnusedSegmentsTaskTest.java
IndexerSQLMetadataStorageCoordinatorTest.java
RetrieveSegmentsActionsTest.java

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
been tested in a test Druid cluster.

Kill tasks by default kill all versions of unused segments in the specified interval. Users wanting to delete specific versions (for example, data compliance reasons) and keep rest of the versions can specify the optional version in the kill task payload.

Sort of like method-level parameterized tests.

Support multiple versions.

kfaraz

Left some comments. On the whole, I would prefer that we override the methods where possible to add a new flavour that accepts versions. Otherwise, we have to pass a lot of nulls around which is a little confusing.

Retain the old interface method and make it default and route it to the method with nullable versions variant. Update usages to use the default method where versions doesn't matter.

kfaraz

Left a couple of final comments, otherwise all looks good.
Haven't gone through the tests in detail. But should we also add a test to verify that we don't kill a segment whose load spec is currently being used by some other version segment?

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

…sed load spec. Checkpoint.

zachjsh · 2024-03-12T03:22:29Z

+      final String versionsStr = versions.stream()
+                                          .map(version -> "'" + version + "'")
+                                          .collect(Collectors.joining(","));
+      sb.append(StringUtils.format(" AND version IN (%s)", versionsStr));


should we enforce some limit on the number of versions here? I think in practice most users wouldnt be specifying too many versions at once.

Good point. However, we don't enforce an upper bound on some related parameters like limit and batchSize, so I'm unsure if we want to do it for versions size. Also, the maximum size for IN would also depend on the underlying metadata store itself, so my suggestion would be to roll this out without any size restriction right out of the bat, and revisit this later if needed

abhishekrb19 · 2024-03-12T09:19:27Z

+    final DateTime maxUsedStatusLastUpdatedTime1 = DateTimes.nowUtc();
+
+    // Delay for 1s, mark the segments as unused and then capture the last updated time cutoff again
+    Thread.sleep(1000);


I think we can actually remove the sleep here and in the other existing tests by directly marking the segments as unused using the test connector. That way, the test would have control over the last updated time and we can set it to whatever time. I can clean up this pattern along with other miscellaneous testing stuff in a follow-up patch

kfaraz

Thanks for the changes, @abhishekrb19 !
The test improvements can be done in a follow up.

Out of curiosity, do you plan to make similar changes for markUsed / markUnused APIs too?

kfaraz · 2024-03-13T03:18:55Z

+        null,
+        null,
+        null
        );


Indentation seems off when looking at the args in the previous lines.

kfaraz · 2024-03-13T03:22:16Z

+    final DateTime now = DateTimes.nowUtc();
+    final String v1 = now.toString();
+    final String v2 = now.minusHours(2).toString();
+    final String v3 = now.minusHours(3).toString();
+
+    final DataSegment segment1 = newSegment(Intervals.of("2019-01-01/2019-02-01"), v1, ImmutableMap.of("foo", "1"));
+    final DataSegment segment2 = newSegment(Intervals.of("2019-02-01/2019-03-01"), v2, ImmutableMap.of("foo", "1"));
+    final DataSegment segment3 = newSegment(Intervals.of("2019-03-01/2019-04-01"), v3, ImmutableMap.of("foo", "1"));


Many of the tests seem to be using similar segments, versions, etc. Do you think some of this can go into the setup method?

kfaraz · 2024-03-13T03:22:52Z

+            umbrellaInterval,
+            null,
+            null
            );


Indentation seems off. Closing brace should have smaller indentation than the preceding args.

abhishekrb19 · 2024-03-13T04:07:12Z

Thanks for the thorough reviews, @kfaraz!

Thanks for the changes, @abhishekrb19 !
The test improvements can be done in a follow up.
Out of curiosity, do you plan to make similar changes for markUsed / markUnused APIs too?

Yes, I'm going to do similar changes to the markUsed / markUnused APIs next. I will also follow up on the test improvements in a separate patch.

github-actions Bot added Area - Documentation Area - Batch Ingestion Area - Ingestion labels Feb 28, 2024

abhishekrb19 commented Feb 28, 2024

View reviewed changes

Comment thread ...xing-service/src/main/java/org/apache/druid/indexing/common/task/KillUnusedSegmentsTask.java Outdated

Kill task version support.

b49d7f9

Kill tasks by default kill all versions of unused segments in the specified interval. Users wanting to delete specific versions (for example, data compliance reasons) and keep rest of the versions can specify the optional version in the kill task payload.

abhishekrb19 force-pushed the kill_by_version branch from 5c22821 to b49d7f9 Compare February 28, 2024 08:52

abhishekrb19 changed the title ~~Kill task version support~~ Kill segments by version Feb 28, 2024

abhishekrb19 added 2 commits February 28, 2024 18:59

Formatting changes.

3cecbd8

Multi version tests in RetrieveSegmentsActionsTest

1970e52

Sort of like method-level parameterized tests.

abhishekrb19 requested review from AmatyaAvadhanula, kfaraz and zachjsh and removed request for kfaraz and zachjsh February 28, 2024 18:08