Skip to content

Fix numSegmentsKilled metric#16103

Merged
abhishekrb19 merged 1 commit intoapache:masterfrom
abhishekrb19:kill_metric_bug
Mar 12, 2024
Merged

Fix numSegmentsKilled metric#16103
abhishekrb19 merged 1 commit intoapache:masterfrom
abhishekrb19:kill_metric_bug

Conversation

@abhishekrb19
Copy link
Copy Markdown
Contributor

@abhishekrb19 abhishekrb19 commented Mar 11, 2024

Prior to this patch, the KillTaskReport contained the numSegmentsKilled metric, representing the size of unused segments nuked from the metadata store, rather than the actual number of segments killed from deep storage. For instance, there may be cases where an unused segment won't be killed from deep storage if its load spec is in use by another segment. In such cases, only the metadata will be cleaned up. This patch contains a test for this scenario.

Alternatively and at the very least to avoid any confusion, we can capture both the number of nuked segments from the metadata store and the number of segments killed from deep storage as separate metrics in the KillTaskReport.


This PR has:

  • been self-reviewed.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • been tested in a test Druid cluster.

@zachjsh
Copy link
Copy Markdown
Contributor

zachjsh commented Mar 12, 2024

LGTM, question: in this case, how does the orphaned segment whose metadata was deleted but data remaings in deep storage get cleaned up from deep storage, does it ever? Or is this a bug?

@abhishekrb19
Copy link
Copy Markdown
Contributor Author

LGTM, question: in this case, how does the orphaned segment whose metadata was deleted but data remaings in deep storage get cleaned up from deep storage, does it ever? Or is this a bug?

AFAIK, the orphan segments need to be manually cleaned up from the deep storage.

@abhishekrb19 abhishekrb19 merged commit 0a615f1 into apache:master Mar 12, 2024
@abhishekrb19 abhishekrb19 deleted the kill_metric_bug branch March 12, 2024 04:32
@adarshsanjeev adarshsanjeev added this to the 30.0.0 milestone May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants