SQL: Use regular filters for time filtering in subqueries.#17173
Merged
abhishekagarwal87 merged 3 commits intoapache:masterfrom Sep 27, 2024
Merged
SQL: Use regular filters for time filtering in subqueries.#17173abhishekagarwal87 merged 3 commits intoapache:masterfrom
abhishekagarwal87 merged 3 commits intoapache:masterfrom
Conversation
Using the "intervals" feature on subqueries, or any non-table, should be avoided because it isn't a meaningful optimization in those cases, and it's simpler for runtime implementations if they can assume all filters are located in the regular filter object. Two changes: 1) Fix the logic in DruidQuery.canUseIntervalFiltering. It was intended to return false for QueryDataSource, but actually returned true. 2) Add a validation to ScanQueryFrameProcessor to ensure that when running on an input channel (which would include any subquery), the query has "intervals" set to ONLY_ETERNITY. Prior to this patch, the new test case in testTimeFilterOnSubquery would throw a "Can only handle a single interval" error in the native engine, and "QueryNotSupported" in the MSQ engine.
github-actions
Bot
added
Area - Batch Ingestion
Area - Querying
Area - MSQ
clintropolis
approved these changes
Sep 26, 2024
kfaraz
pushed a commit
to kfaraz/druid
that referenced
this pull request
Oct 4, 2024
) * SQL: Use regular filters for time filtering in subqueries. Using the "intervals" feature on subqueries, or any non-table, should be avoided because it isn't a meaningful optimization in those cases, and it's simpler for runtime implementations if they can assume all filters are located in the regular filter object. Two changes: 1) Fix the logic in DruidQuery.canUseIntervalFiltering. It was intended to return false for QueryDataSource, but actually returned true. 2) Add a validation to ScanQueryFrameProcessor to ensure that when running on an input channel (which would include any subquery), the query has "intervals" set to ONLY_ETERNITY. Prior to this patch, the new test case in testTimeFilterOnSubquery would throw a "Can only handle a single interval" error in the native engine, and "QueryNotSupported" in the MSQ engine. * Mark new case as having extra columns in decoupled mode. * Adjust test.
kfaraz
added a commit
that referenced
this pull request
Oct 5, 2024
…) (#17251) * SQL: Use regular filters for time filtering in subqueries. (#17173) * RunWorkOrder: Account for two simultaneous statistics collectors. (#17216) * DartTableInputSpecSlicer: Fix for TLS workers. (#17224) * Upgrade avro - minor version (#17230) * SuperSorter: Don't set allDone if it's already set. (#17238) * Decoupled planning: improve join support (#17039) --------- Co-authored-by: Gian Merlino <gianmerlino@gmail.com> Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com> Co-authored-by: Zoltan Haindrich <kirk@rxd.hu>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Using the "intervals" feature on subqueries, or any non-table, should be avoided because it isn't a meaningful optimization in those cases, and it's simpler for runtime implementations if they can assume all filters are located in the regular filter object.
Two changes:
Fix the logic in DruidQuery.canUseIntervalFiltering. It was intended
to return false for QueryDataSource, but actually returned true.
Add a validation to ScanQueryFrameProcessor to ensure that when running
on an input channel (which would include any subquery), the query has
"intervals" set to ONLY_ETERNITY.
Prior to this patch, the new test case in testTimeFilterOnSubquery would throw a "Can only handle a single interval" error in the native engine, and "QueryNotSupported" in the MSQ engine.