Skip to content

Fix compaction status API response#17006

Merged
kfaraz merged 3 commits intoapache:masterfrom
kfaraz:fix_compaction_apis
Sep 5, 2024
Merged

Fix compaction status API response#17006
kfaraz merged 3 commits intoapache:masterfrom
kfaraz:fix_compaction_apis

Conversation

@kfaraz
Copy link
Copy Markdown
Contributor

@kfaraz kfaraz commented Sep 4, 2024

Description

#16768 introduces new compaction APIs on the Overlord /compact/status and /compact/progress.
But the corresponding OverlordClient methods do not return an object compatible with the actual endpoints defined in OverlordCompactionResource.

This patch ensures that the objects are compatible.

Changes

  • Add CompactionStatusResponse and CompactionProgressResponse
  • Use these as the return type in OverlordClient methods and as the response entity in OverlordCompactionResource
  • Add SupervisorCleanupModule bound on the Coordinator to perform cleanup of supervisors. Without this module, Coordinator cannot deserialize compaction supervisors.

Testing

  • Added some UTs
  • Local Druid cluster:
    • verified that compaction status APIs are redirected correctly when running compaction supervisor
    • verified that compaction supervisors are cleaned up successfully by the coordinator

@kfaraz kfaraz added this to the 31.0.0 milestone Sep 5, 2024
snapshots = Collections.singleton(autoCompactionSnapshot);
}
return Response.ok(Collections.singletonMap("latestStatus", snapshots)).build();
return Response.ok(new CompactionStatusResponse(snapshots)).build();
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason that raw map isn't compatible but a class works? I believe both would be serialised in the same manner.

Copy link
Copy Markdown
Contributor Author

@kfaraz kfaraz Sep 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's true. The problem wasn't that I had used a map instead of a dedicated POJO. Instead, I had forgotten to use anything at all 😛 .

The Overlord REST endpoint returns a response that looks like

{
    "latestStatus": [{snapshot1object}, {snapshot2object}, {snapshot3object}]
}

but in the Overlord client, I was just reading the internal list

[{snapshot1object}, {snapshot2object}, {snapshot3object}]

instead of the whole object.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to avoid such mistakes in the future and to keep the code simple, I decided to use a model POJO instead of maps everywhere.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree. Makes it so much cleaner.

@kfaraz kfaraz merged commit ba6f804 into apache:master Sep 5, 2024
@kfaraz kfaraz deleted the fix_compaction_apis branch September 5, 2024 17:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants