[improvement](mtmv) Support to use current_date() when create async mv #36111
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39959 ms |
TPC-DS: Total hot run time: 169159 ms |
ClickBench: Total hot run time: 30.71 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40702 ms |
TPC-DS: Total hot run time: 173055 ms |
ClickBench: Total hot run time: 30.72 s |
Contributor
|
support all nondeterministic function ? |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39864 ms |
TPC-DS: Total hot run time: 170309 ms |
ClickBench: Total hot run time: 30.39 s |
morrySnow
reviewed
Jun 12, 2024
| * Identify the function is deterministic or not, such as UnixTimestamp, when it's children is not empty | ||
| * it's deterministic | ||
| */ | ||
| boolean isDeterministic(); |
Contributor
There was a problem hiding this comment.
add this interface to expression with default implement check all check isDeterministic. Nondeterministic default return false. unixTimestamp do not implement Nondeterministic anymore. all check isntance of Nondeterministic change to check expression.isDeterministic().
| * the expression in whiteFunctionSet would not be collected | ||
| */ | ||
| public static class FunctionCollectContext { | ||
| private final List<Expression> collectedExpressions = new LinkedList<>(); |
Contributor
There was a problem hiding this comment.
do not use java's linkedList in any case. it has poor perf with no adv with ArrayList
| return collectedExpressions; | ||
| } | ||
| expressions.forEach(expression -> { | ||
| collectedExpressions.addAll(expression.collect(Nondeterministic.class::isInstance)); |
Contributor
There was a problem hiding this comment.
i think u could collected isDeterministic == true after move this interface into expression
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39478 ms |
TPC-DS: Total hot run time: 174267 ms |
ClickBench: Total hot run time: 30.69 s |
morrySnow
reviewed
Jun 13, 2024
| * Identify the expression is deterministic or not | ||
| */ | ||
| default boolean isDeterministic() { | ||
| return true; |
Contributor
There was a problem hiding this comment.
recursive call children's isDeterministic
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40459 ms |
morrySnow
reviewed
Jun 17, 2024
| boolean containsNondeterministic = !((ExpressionTrait) expr).isDeterministic(); | ||
| if (!collectContext.getCollectExpressionTypes().isEmpty()) { | ||
| containsNondeterministic &= collectContext.getCollectExpressionTypes().stream() | ||
| .anyMatch(type -> type.isAssignableFrom(expr.getClass())); |
Contributor
There was a problem hiding this comment.
why need type.isAssignableFrom(expr.getClass())
ClickBench: Total hot run time: 30.18 s |
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
Contributor
Author
|
run feut |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39522 ms |
TPC-DS: Total hot run time: 173457 ms |
ClickBench: Total hot run time: 30.8 s |
zddr
approved these changes
Jun 20, 2024
Contributor
zddr
left a comment
zddr
left a comment
There was a problem hiding this comment.
It may be strange to set enable_dondeterministic_function to false through the alter statement after creating a materialized view, but it does not affect the business logic
Contributor
|
PR approved by anyone and no changes requested. |
morrySnow
approved these changes
Jun 21, 2024
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
iszhangpch
pushed a commit
to iszhangpch/doris-p
that referenced
this pull request
Jun 21, 2024
…async mv (apache#36111) Support to use current_date() when create async materialized view by adding 'enable_nondeterministic_function' = 'true' in properties when create materialized view. `enable_nondeterministic_function` is default false. Here is a example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp(k3, '%Y-%m-%d %H:%i-%s') from ${tableName} where current_date() > k3; Note: unix_timestamp is nondeterministic when has no params. it is deterministic when has params which means format column k3 date another example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp() from ${tableName} where current_date() > k3; though unix_timestamp() is nondeterministic, we add 'enable_date_nondeterministic_function' = 'true' in properties
dataroaring
pushed a commit
that referenced
this pull request
Jun 21, 2024
…async mv (#36111) Support to use current_date() when create async materialized view by adding 'enable_nondeterministic_function' = 'true' in properties when create materialized view. `enable_nondeterministic_function` is default false. Here is a example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp(k3, '%Y-%m-%d %H:%i-%s') from ${tableName} where current_date() > k3; Note: unix_timestamp is nondeterministic when has no params. it is deterministic when has params which means format column k3 date another example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp() from ${tableName} where current_date() > k3; though unix_timestamp() is nondeterministic, we add 'enable_date_nondeterministic_function' = 'true' in properties
seawinde
added a commit
to seawinde/doris
that referenced
this pull request
Jul 11, 2024
…async mv (apache#36111) Support to use current_date() when create async materialized view by adding 'enable_nondeterministic_function' = 'true' in properties when create materialized view. `enable_nondeterministic_function` is default false. Here is a example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp(k3, '%Y-%m-%d %H:%i-%s') from ${tableName} where current_date() > k3; Note: unix_timestamp is nondeterministic when has no params. it is deterministic when has params which means format column k3 date another example, it will success > CREATE MATERIALIZED VIEW mv_name > BUILD DEFERRED REFRESH AUTO ON MANUAL > DISTRIBUTED BY RANDOM BUCKETS 2 > PROPERTIES ( > 'replication_num' = '1', > 'enable_nondeterministic_function' = 'true' > ) > AS > SELECT *, unix_timestamp() from ${tableName} where current_date() > k3; though unix_timestamp() is nondeterministic, we add 'enable_date_nondeterministic_function' = 'true' in properties
morrySnow
pushed a commit
that referenced
this pull request
Jul 12, 2024
starocean999
pushed a commit
that referenced
this pull request
Aug 26, 2024
…a expression is nondeterministic or not (#39801) ## Proposed changes In #36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
seawinde
added a commit
to seawinde/doris
that referenced
this pull request
Aug 29, 2024
…a expression is nondeterministic or not (apache#39801) ## Proposed changes In apache#36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
yiguolei
pushed a commit
to seawinde/doris
that referenced
this pull request
Aug 29, 2024
…a expression is nondeterministic or not (apache#39801) ## Proposed changes In apache#36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
dataroaring
pushed a commit
that referenced
this pull request
Sep 26, 2024
…a expression is nondeterministic or not (#39801) ## Proposed changes In #36111, we add `isDeterministic` method in class `ExpressionTrait` to identify the expression is deterministic or not. But `unix_timestamp` doesn't extend Nondeterministic, but it is not deterministic when it's children is empty. and is not deterministic when children is not empty. If we use` instanceOf Nondeterministic `to indentify if expression is is not deterministic, that is confused. So we do something as fllowing: 1. Remove Nondeterministic class, and use `isDeterministic` to indentify it's deterministic. 2. Add `containsNondeterministic` method in `ExpressionTrait` to identify it contains nondeterministic expression or not. 3. `isDeterministic` only identify current expression is deterministic or not. would identify if contains nondeterministic or not
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Support to use current_date() when create async materialized view by adding
'enable_nondeterministic_function' = 'true'in properties when create materialized view.enable_nondeterministic_functionis default false.Here is a example, it will success
Note:
unix_timestampis nondeterministic when has no params. it is deterministic when has params which means format column k3 dateanother example, it will success
though
unix_timestamp()is nondeterministic, we add'enable_date_nondeterministic_function' = 'true'in properties