-
Notifications
You must be signed in to change notification settings - Fork 3.8k
document useFilterCNF query context parameter #9647
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
clintropolis
merged 5 commits into
apache:master
from
clintropolis:doc-useFilterCNF-context-parameter
Apr 17, 2020
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
2cbf0a2
document useFilterCNF query context parameter
clintropolis 5c2c087
move context key to QueryContexts
clintropolis ee9f6e9
Update .spelling
clintropolis 8a920d3
Merge remote-tracking branch 'upstream/master' into doc-useFilterCNF-…
clintropolis 19cc541
Merge remote-tracking branch 'upstream/master' into doc-useFilterCNF-…
clintropolis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -52,6 +52,7 @@ public class QueryContexts | |
| public static final String JOIN_FILTER_REWRITE_ENABLE_KEY = "enableJoinFilterRewrite"; | ||
| public static final String JOIN_FILTER_REWRITE_VALUE_COLUMN_FILTERS_ENABLE_KEY = "enableJoinFilterRewriteValueColumnFilters"; | ||
| public static final String JOIN_FILTER_REWRITE_MAX_SIZE_KEY = "joinFilterRewriteMaxSize"; | ||
| public static final String USE_FILTER_CNF_KEY = "useFilterCNF"; | ||
|
|
||
| public static final boolean DEFAULT_BY_SEGMENT = false; | ||
| public static final boolean DEFAULT_POPULATE_CACHE = true; | ||
|
|
@@ -67,7 +68,8 @@ public class QueryContexts | |
| public static final boolean DEFAULT_ENABLE_JOIN_FILTER_PUSH_DOWN = true; | ||
| public static final boolean DEFAULT_ENABLE_JOIN_FILTER_REWRITE = true; | ||
| public static final boolean DEFAULT_ENABLE_JOIN_FILTER_REWRITE_VALUE_COLUMN_FILTERS = false; | ||
| public static final long DEFAULT_ENABLE_JOIN_FILTER_REWRITE_MAX_SIZE_KEY = 10000; | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 👍 |
||
| public static final long DEFAULT_ENABLE_JOIN_FILTER_REWRITE_MAX_SIZE = 10000; | ||
| public static final boolean DEFAULT_USE_FILTER_CNF = false; | ||
|
|
||
| @SuppressWarnings("unused") // Used by Jackson serialization | ||
| public enum Vectorize | ||
|
|
@@ -249,7 +251,7 @@ public static <T> boolean getEnableJoinFilterRewriteValueColumnFilters(Query<T> | |
|
|
||
| public static <T> long getJoinFilterRewriteMaxSize(Query<T> query) | ||
| { | ||
| return parseLong(query, JOIN_FILTER_REWRITE_MAX_SIZE_KEY, DEFAULT_ENABLE_JOIN_FILTER_REWRITE_MAX_SIZE_KEY); | ||
| return parseLong(query, JOIN_FILTER_REWRITE_MAX_SIZE_KEY, DEFAULT_ENABLE_JOIN_FILTER_REWRITE_MAX_SIZE); | ||
| } | ||
|
|
||
| public static <T> boolean getEnableJoinFilterPushDown(Query<T> query) | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggest providing a few examples to clarify:
A || BwhereAcan be resolved using bitmap indexes butBcannot will prevent the whole OR filter from being considered for pre-filteringA && Binstead,Awould be considered for pre-filtering butBwould not.A && (C || D)whereCandDcan be resolved using bitmap indexes, then the whole filter can be considered for pre-filteringA && (B || C)onlyAwill be considered for pre-filteringThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, I think how filters work with query processing and how mechanically filters are split into pre and post filters should be documented somewhere, but I don't think this setting is quite the correct avenue. Additionally, since which filters can and can't use bitmaps isn't exactly documented anywhere, I'm not sure how much the examples would help.
If we added this general description of query processing and how filters are involved could link to this setting, and also could link to documentation we would add for the
filterTuningadded in #8209, as ways the user can help influence how filter processing behaves. Maybesegments.mdwould be an appropriate place since it mentions bitmaps and their role in filtering, orsegment-optimization.mdsince it involves how to tune segment sizes? Or perhaps we need anadvanced-tuning.mdto put this and other stuff that users shouldn't really mess with unless they are prepared to roll up their sleeves and experimentally verify the settings to fine tune to their workload?Should this be part of this PR, or done as a follow-up? It sort of blows up the scope a bit of what I was looking to do as part of this PR, but it also seems useful so I'm fine either way.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the filter tuning guide could be done in a follow-up, it sounds like something that would be much larger than this PR, this PR LGTM.
(There's a merge conflict in the spelling exclusions now)