Remove task action audit logging and druid_taskLog metadata table by kfaraz · Pull Request #16309 · apache/druid

kfaraz · 2024-04-18T15:59:11Z

Note

This PR does not pertain to the audit logging system used by Druid for auditing all major update actions such as run a task, update rules, update dynamic configs, create a supervisor, etc. That information is persisted in druid_audit metadata table (if druid.audit.manager.type=sql) or simply logged.
Instead, it deals with the audit logging used only for task actions i.e. the druid_taskLog metadata table.

Description

Task action audit logging was first deprecated and disabled by default in Druid 0.13, #6368.

As called out in the original discussion #5859, there are several drawbacks to persisting task action audit logs.

Only usage of the task audit logs is to serve the API /indexer/v1/task/{taskId}/segments
which returns the list of segments created by a task.
The use case is really narrow and no prod clusters really use this information.
There can be better ways of obtaining this information, such as the metric
segment/added/bytes which reports both the segment ID and task ID
when a segment is committed by a task. We could also include committed segment IDs in task reports.
A task persisting several segments would bloat up the audit logs table putting unnecessary strain
on metadata storage.

Changes

Remove TaskAuditLogConfig
Remove method TaskAction.isAudited(). No task action is audited anymore.
Remove SegmentInsertAction as it is not used anymore. SegmentTransactionalInsertAction is the new incarnation which has been in use for a while.
Deprecate MetadataStorageActionHandler.addLog() and getLogs(). These are not used anymore but need to be retained for backward compatibility of extensions.
Do not create druid_taskLog metadata table anymore.

Release notes

Task action audit logging was deprecated in Druid 0.13 and is being completely removed in this release.
The API /indexer/v1/task/{taskId}/segments is not supported anymore and will give a 404 NOT FOUND response.
Druid will not write to or read from the metadata table druid_taskLog anymore.
The property druid.indexer.auditlog.enabled will be ignored by Druid.
The metric task/action/log/time will not be emitted anymore.

Extension dev notes

The changes in this PR are backward compatible with all existing metadata storage extensions.
The methods addLog and getLogs of MetadataStorageActionHandler are now deprecated
and not used by the Druid code.
Any new metadata storage extension need not implement these methods.

Rolling upgrade concerns

No upgrade concerns as none of the tasks use the SegmentInsertAction.

Future solutions

Which task created a segment?

A more preferable approach would be to simply add a task_id column in the segments table.
Something similar has been recently done for pending segments in #16144.

Alternatively, it could also be possible to determine the list of segments committed by a task by inspecting
the reports of the task or emitted metrics.

Which user created a segment?

Task submission is already logged and/or persisted depending on configuration by the Druid audit system.
Once we can associate segments to task IDs, we would also be able to identify which user created a given
segment.

This PR has:

…ion_audit

+  {
+    OverlordResource overlordResource =
+        new OverlordResource(null, null, null, null, null, null, null, null, null, null);
+    final Response response = overlordResource.getTaskSegments("taskId");


-        handler.getLogs("non_exist_entry")
+    Exception exception = Assert.assertThrows(
+        UnsupportedOperationException.class,
+        () -> handler.addLog("abcd", ImmutableMap.of("logentry", "created"))


+  {
+    Exception exception = Assert.assertThrows(
+        UnsupportedOperationException.class,
+        () -> handler.getLogs("abcd")


+  {
+    Exception exception = Assert.assertThrows(
+        DruidException.class,
+        () -> handler.addLog("abcd", "logentry")


+  {
+    Exception exception = Assert.assertThrows(
+        DruidException.class,
+        () -> handler.getLogs("abcd")


kfaraz · 2024-07-17T11:39:14Z

Thanks for the review, @clintropolis !

…ache#16309) Description: Task action audit logging was first deprecated and disabled by default in Druid 0.13, apache#6368. As called out in the original discussion apache#5859, there are several drawbacks to persisting task action audit logs. - Only usage of the task audit logs is to serve the API `/indexer/v1/task/{taskId}/segments` which returns the list of segments created by a task. - The use case is really narrow and no prod clusters really use this information. - There can be better ways of obtaining this information, such as the metric `segment/added/bytes` which reports both the segment ID and task ID when a segment is committed by a task. We could also include committed segment IDs in task reports. - A task persisting several segments would bloat up the audit logs table putting unnecessary strain on metadata storage. Changes: - Remove `TaskAuditLogConfig` - Remove method `TaskAction.isAudited()`. No task action is audited anymore. - Remove `SegmentInsertAction` as it is not used anymore. `SegmentTransactionalInsertAction` is the new incarnation which has been in use for a while. - Deprecate `MetadataStorageActionHandler.addLog()` and `getLogs()`. These are not used anymore but need to be retained for backward compatibility of extensions. - Do not create `druid_taskLog` metadata table anymore.

Description: The `MetadataStorageActionHandler` deals with persistence of `Task` and related objects. But those model classes exist in `indexing-service` module which forces the `MetadataStorageActionHandler` to handle generics unnecessarily. This makes the code cluttered and difficult to understand at times. Changes: - Move `MetadataStorageActionHandler` and factory classes to `indexing-service` module - Move implementations to respective extensions - Simplify class `TaskInfo` by removing type parameters - Add class `TaskIdStatus` - Remove class `TaskStuff` - Add `DerbyTaskStorageModule` to bind the `MetadataStorageActionHandler` to its default impl on `CliOverlord` - Remove references to the tasklog table as it had been disabled in #16309

Remove task audit logging

26fdb96

github-actions Bot added the Area - Ingestion label Apr 18, 2024

kfaraz requested a review from gianm April 18, 2024 16:00

Remove backward incompat changes

3fc539f

kfaraz changed the title ~~Remove task audit logging~~ Remove task action audit logging Apr 19, 2024

Merge branch 'master' of github.com:apache/druid into remove_task_act…

662053e

…ion_audit

kfaraz requested a review from abhishekrb19 April 25, 2024 05:16

kfaraz changed the title ~~Remove task action audit logging~~ Remove task action audit logging and druid_taskLog metadata table Jun 6, 2024

kfaraz added 6 commits June 6, 2024 10:48

Merge branch 'master' of github.com:apache/druid into remove_task_act…

8898379

…ion_audit

Merge branch 'master' of github.com:apache/druid into remove_task_act…

31734b7

…ion_audit

Merge branch 'master' of github.com:apache/druid into remove_task_act…

b3ae351

…ion_audit

Revert changes to StorageActionHandler classes

d377aef

Remove backward incompatible changes

2fedfb5

Fix new action

7c2e254

kfaraz requested review from clintropolis and removed request for abhishekrb19 June 26, 2024 07:33

kfaraz added 2 commits June 26, 2024 13:05

Do not create tasklogs table anymore

66ead86

Fix up tests

ad7cede

clintropolis reviewed Jun 26, 2024

View reviewed changes

Comment thread indexing-service/src/main/java/org/apache/druid/indexing/overlord/http/OverlordResource.java

Fix tests

5bf3485

github-advanced-security AI found potential problems Jun 26, 2024

View reviewed changes

More fixes

7b9d053

kfaraz requested a review from clintropolis June 26, 2024 11:38

Update docs

332a4d6

github-actions Bot added the Area - Documentation label Jun 27, 2024

Add coverage for MetadataStorageACtionHandler

b2146c7

github-advanced-security AI found potential problems Jun 27, 2024

View reviewed changes

Remove method MetadataStorageActionHandlerTypes.getLogType()

4bda9ab

kfaraz added Release Notes Area - Metadata labels Jul 12, 2024

clintropolis approved these changes Jul 17, 2024

View reviewed changes

kfaraz merged commit 9f6ce6d into apache:master Jul 17, 2024

kfaraz deleted the remove_task_action_audit branch July 17, 2024 11:39

kfaraz mentioned this pull request Jul 17, 2024

Fix bug in TaskStorageQueryAdapter #16750

Merged

10 tasks

kfaraz added this to the 31.0.0 milestone Oct 4, 2024

kfaraz mentioned this pull request Oct 11, 2024

[DRAFT] 31.0.0 Release Notes #17332

Closed

kfaraz mentioned this pull request Sep 4, 2025

Refactor: Remove type params from MetadataStorageActionHandler #18479

Merged

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove task action audit logging and druid_taskLog metadata table#16309

Remove task action audit logging and druid_taskLog metadata table#16309
kfaraz merged 16 commits intoapache:masterfrom
kfaraz:remove_task_action_audit

kfaraz commented Apr 18, 2024 •

edited

Loading

Uh oh!

Uh oh!

Check notice

Check notice

Check notice

Check notice

Check notice

kfaraz commented Jul 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

kfaraz commented Apr 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note

Description

Changes

Release notes

Extension dev notes

Rolling upgrade concerns

Future solutions

Which task created a segment?

Which user created a segment?

Uh oh!

Uh oh!

Check notice

Check notice

Check notice

Check notice

Check notice

kfaraz commented Jul 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kfaraz commented Apr 18, 2024 •

edited

Loading