[Backport] Patches (#17039) (#17173) (#17216) (#17224) (#17230) (#17238)#17251
Merged
kfaraz merged 6 commits intoapache:31.0.0from Oct 5, 2024
Merged
[Backport] Patches (#17039) (#17173) (#17216) (#17224) (#17230) (#17238)#17251kfaraz merged 6 commits intoapache:31.0.0from
kfaraz merged 6 commits intoapache:31.0.0from
Conversation
) * SQL: Use regular filters for time filtering in subqueries. Using the "intervals" feature on subqueries, or any non-table, should be avoided because it isn't a meaningful optimization in those cases, and it's simpler for runtime implementations if they can assume all filters are located in the regular filter object. Two changes: 1) Fix the logic in DruidQuery.canUseIntervalFiltering. It was intended to return false for QueryDataSource, but actually returned true. 2) Add a validation to ScanQueryFrameProcessor to ensure that when running on an input channel (which would include any subquery), the query has "intervals" set to ONLY_ETERNITY. Prior to this patch, the new test case in testTimeFilterOnSubquery would throw a "Can only handle a single interval" error in the native engine, and "QueryNotSupported" in the MSQ engine. * Mark new case as having extra columns in decoupled mode. * Adjust test.
…ache#17216) * RunWorkOrder: Account for two simultaneous statistics collectors. As a follow up to apache#17057, divide the amount of partitionStatsMemory by two, to account for the fact that there are at some times going to be two copies of the full collector. First there will be one for processors and one for the accumulated collector. Then, after the processor ones are GCed, a snapshot of the accumulated collector will be created. Also includes an optimization to "addAll" for the two KeyCollectors, for the case where we're adding into an empty collector. This is always going to happen once per stage due to the "withAccumulation" call. * Fix missing variable. * Don't divide by numProcessingThreads twice. * Fix test.
We should use getHost(), which returns TLS if configured or plaintext otherwise. getHostAndPort() returns plaintext only.
This fixes a race where, if there is no output at all, setAllDoneIfPossible could be called twice (once when the output partitions future resolves, and once when the batcher finishes). If the calls happen in that order, it would try to create nil output channels both times, resulting in a "Channel already set" error.
github-actions
Bot
added
Area - Batch Ingestion
Area - Querying
Area - Dependencies
Area - MSQ
There were some problematic cases join branches are run with finalize=false instead of finalize=true like normal subqueries this inconsistency is not good - but fixing it is a bigger thing ensure that right hand sides of joins are always subqueries - or accessible globally To achieve the above: operand indexes were needed for the upstream reltree nodes in the generator source unwrapping now takes the join situation into account as well
kfaraz
changed the title
Contributor
Author
|
IT failure is unrelated, merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport the following patches:
#17039
#17173
#17216
#17224
#17230
#17238