RunWorkOrder: Account for two simultaneous statistics collectors.#17216
Merged
abhishekagarwal87 merged 4 commits intoapache:masterfrom Oct 3, 2024
Merged
RunWorkOrder: Account for two simultaneous statistics collectors.#17216abhishekagarwal87 merged 4 commits intoapache:masterfrom
abhishekagarwal87 merged 4 commits intoapache:masterfrom
Conversation
As a follow up to apache#17057, divide the amount of partitionStatsMemory by two, to account for the fact that there are at some times going to be two copies of the full collector. First there will be one for processors and one for the accumulated collector. Then, after the processor ones are GCed, a snapshot of the accumulated collector will be created. Also includes an optimization to "addAll" for the two KeyCollectors, for the case where we're adding into an empty collector. This is always going to happen once per stage due to the "withAccumulation" call.
github-actions
Bot
added
Area - Batch Ingestion
Area - MSQ
Contributor
|
I ran into this bug :-p |
adarshsanjeev
approved these changes
Oct 3, 2024
kfaraz
pushed a commit
to kfaraz/druid
that referenced
this pull request
Oct 4, 2024
…ache#17216) * RunWorkOrder: Account for two simultaneous statistics collectors. As a follow up to apache#17057, divide the amount of partitionStatsMemory by two, to account for the fact that there are at some times going to be two copies of the full collector. First there will be one for processors and one for the accumulated collector. Then, after the processor ones are GCed, a snapshot of the accumulated collector will be created. Also includes an optimization to "addAll" for the two KeyCollectors, for the case where we're adding into an empty collector. This is always going to happen once per stage due to the "withAccumulation" call. * Fix missing variable. * Don't divide by numProcessingThreads twice. * Fix test.
kfaraz
added a commit
that referenced
this pull request
Oct 5, 2024
…) (#17251) * SQL: Use regular filters for time filtering in subqueries. (#17173) * RunWorkOrder: Account for two simultaneous statistics collectors. (#17216) * DartTableInputSpecSlicer: Fix for TLS workers. (#17224) * Upgrade avro - minor version (#17230) * SuperSorter: Don't set allDone if it's already set. (#17238) * Decoupled planning: improve join support (#17039) --------- Co-authored-by: Gian Merlino <gianmerlino@gmail.com> Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com> Co-authored-by: Zoltan Haindrich <kirk@rxd.hu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
As a follow up to #17057, divide the amount of partitionStatsMemory by two, to account for the fact that there are at some times going to be two copies of the full collector. First there will be one for processors and one for the accumulated collector. Then, after the processor ones are GCed, a snapshot of the accumulated collector will be created.
Also includes an optimization to "addAll" for the two KeyCollectors, for the case where we're adding into an empty collector. This is always going to happen once per stage due to the "withAccumulation" call.
This fixes an OOME @vogievetsky noticed during local testing with
bin/start-druid.