add druid.expressions.allowVectorizeFallback and default to false#17248
Merged
abhishekagarwal87 merged 4 commits intoapache:masterfrom Oct 5, 2024
Merged
Conversation
changes: * adds ExpressionProcessing.allowVectorizeFallback() and ExpressionProcessingConfig.allowVectorizeFallback(), defaulting to false * add cannotVectorizeUnlessFallback functions to make it easy to toggle the default of this config, and easy to know what to delete when we remove it in the future
gianm
approved these changes
Oct 5, 2024
| |Key|Description|Default| | ||
| |---|-----------|-------| | ||
| |`druid.expressions.useStrictBooleans`|Controls the behavior of Druid boolean operators and functions, if set to `true` all boolean values are either `1` or `0`. This configuration has been deprecated and will be removed in a future release, taking on the `true` behavior. See [expression documentation](../querying/math-expr.md#logical-operator-modes) for more information.|true| | ||
| |`druid.expressions.allowNestedArrays`|If enabled, Druid array expressions can create nested arrays. This configuration has been deprecated and will be removed in a future release, taking on the `true` behavior.|true| |
Contributor
There was a problem hiding this comment.
I guess this had already been removed? I don't see references to it in the code.
Member
Author
There was a problem hiding this comment.
yea, i forgot to delete the docs for it
abhishekagarwal87
pushed a commit
to abhishekagarwal87/druid
that referenced
this pull request
Oct 5, 2024
…ache#17248) changes: adds ExpressionProcessing.allowVectorizeFallback() and ExpressionProcessingConfig.allowVectorizeFallback(), defaulting to false until few remaining bugs can be fixed (mostly complex types and some odd interactions with mixed types) add cannotVectorizeUnlessFallback functions to make it easy to toggle the default of this config, and easy to know what to delete when we remove it in the future
kfaraz
pushed a commit
to kfaraz/druid
that referenced
this pull request
Oct 6, 2024
…ache#17248) changes: adds ExpressionProcessing.allowVectorizeFallback() and ExpressionProcessingConfig.allowVectorizeFallback(), defaulting to false until few remaining bugs can be fixed (mostly complex types and some odd interactions with mixed types) add cannotVectorizeUnlessFallback functions to make it easy to toggle the default of this config, and easy to know what to delete when we remove it in the future
abhishekagarwal87
added a commit
that referenced
this pull request
Oct 6, 2024
…7248) (#17260) changes: adds ExpressionProcessing.allowVectorizeFallback() and ExpressionProcessingConfig.allowVectorizeFallback(), defaulting to false until few remaining bugs can be fixed (mostly complex types and some odd interactions with mixed types) add cannotVectorizeUnlessFallback functions to make it easy to toggle the default of this config, and easy to know what to delete when we remove it in the future Co-authored-by: Clint Wylie <cwylie@apache.org>
gianm
added a commit
to gianm/druid
that referenced
this pull request
Sep 9, 2025
PR apache#16366 originally added fallback vectorization, a mechanism for making all expressions vectorizable. Later, apache#17098 fixed some issues that arose and apache#17248 disabled fallback vectorization in the out-of-box configuration. This patch fixes various remaining issues with inconsistent type handling between the vectorized and nonvectorized expr implementations. It does not yet re-enable fallback vectorization out of the box, due to remaining inconsistencies with conditional exprs like "case_searched", "case_simple", and "if". 1) Aligns the behavior of missing columns and literal nulls so they are always treated as null longs. This was already the case for vectorized identifiers, but non-vectorized identifiers and literal nulls were still represented as strings. 2) Replaces all occurrences of "ExprEval.of(null)" with either an explicit type, or a call to "ExprEval.ofMissing()". ofMissing is a new function for situations where an eval represents a null value of unknown type. It is equivalent to "ExprEval.ofLong(null)", but is a separate function for clarity at the call site. 3) Update "cast" to return the target type even for null values. 4) Update "greatest", "least", and "array" so they eval to types that match what is reported by "getOutputType". 5) Update "scalb" to coerce input strings as numbers, to better allow for type evolution and missing columns. 6) Update "reverse" to coerce inputs to strings, to better allow for type evolution and missing columns.
gianm
added a commit
that referenced
this pull request
Sep 18, 2025
* Additional expr type alignment. PR #16366 originally added fallback vectorization, a mechanism for making all expressions vectorizable. Later, #17098 fixed some issues that arose and #17248 disabled fallback vectorization in the out-of-box configuration. This patch fixes various remaining issues with inconsistent type handling between the vectorized and nonvectorized expr implementations. It does not yet re-enable fallback vectorization out of the box, due to remaining inconsistencies with conditional exprs like "case_searched", "case_simple", and "if". 1) Aligns the behavior of missing columns and literal nulls so they are always treated as null longs. This was already the case for vectorized identifiers, but non-vectorized identifiers and literal nulls were still represented as strings. 2) Replaces all occurrences of "ExprEval.of(null)" with either an explicit type, or a call to "ExprEval.ofMissing()". ofMissing is a new function for situations where an eval represents a null value of unknown type. It is equivalent to "ExprEval.ofLong(null)", but is a separate function for clarity at the call site. 3) Update "cast" to return the target type even for null values. 4) Update "greatest", "least", and "array" so they eval to types that match what is reported by "getOutputType". 5) Update "scalb" to coerce input strings as numbers, to better allow for type evolution and missing columns. 6) Update "reverse" to coerce inputs to strings, to better allow for type evolution and missing columns. * Restore fallback in testArrayFns. * Fix issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
changes:
ExpressionProcessing.allowVectorizeFallback()andExpressionProcessingConfig.allowVectorizeFallback(), defaulting to false until few remaining bugs can be fixed (mostly complex types and some odd interactions with mixed types)cannotVectorizeUnlessFallbackfunctions to make it easy to toggle the default of this config, and easy to know what to delete when we remove it in the future