Skip to content

Conversation

@keanji-x
Copy link
Contributor

@keanji-x keanji-x commented Oct 9, 2023

Proposed changes

covert predicate to sargable

  1. support format like 1 - a
  2. support rearrange year/month/week/day/minutes/seconds_sub/add function

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@keanji-x
Copy link
Contributor Author

keanji-x commented Oct 9, 2023

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.65 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162353504 Bytes

@keanji-x keanji-x force-pushed the covert_to_sargable branch from e3f134d to 4debe46 Compare October 9, 2023 12:00
@keanji-x
Copy link
Contributor Author

keanji-x commented Oct 9, 2023

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.49 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162213904 Bytes

@keanji-x keanji-x force-pushed the covert_to_sargable branch from 4debe46 to 068eaf6 Compare October 9, 2023 12:57
@keanji-x
Copy link
Contributor Author

keanji-x commented Oct 9, 2023

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.84 seconds
stream load tsv: 562 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.1 seconds inserted 10000000 Rows, about 343K ops/s
storage size: 17162385676 Bytes

@keanji-x keanji-x force-pushed the covert_to_sargable branch 2 times, most recently from 819737b to 3fbe33f Compare October 10, 2023 08:06
@keanji-x
Copy link
Contributor Author

run buildall

@keanji-x
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.85 seconds
stream load tsv: 573 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162176995 Bytes

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.78 seconds
stream load tsv: 578 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162698280 Bytes

@keanji-x
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.93 seconds
stream load tsv: 557 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162225962 Bytes

public Expression visitLessThanEqual(LessThanEqual lessThanEqual, ExpressionRewriteContext context) {
return process(lessThanEqual);
// Ensure that the second child must be Literal, such as
private @Nullable ComparisonPredicate normalize(ComparisonPredicate comparison) {
Copy link
Contributor

@englefly englefly Oct 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be better to make it public, because we could use it to generate more effecient Runtime filter.
PostProcessor need this function

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@keanji-x keanji-x changed the title [feature](Nereids) covert predicate to sargable [feature](Nereids) covert predicate to SARGABLE Oct 12, 2023
@jackwener jackwener merged commit d6ff974 into apache:master Oct 12, 2023
dutyu pushed a commit to dutyu/doris that referenced this pull request Oct 28, 2023
covert predicate to SARGABLE 
1. support format like `1 - a`
2. support rearrange `year/month/week/day/minutes/seconds_sub/add` function
924060929 added a commit that referenced this pull request Nov 28, 2024
…e wrong result because SimplifyArithmeticComparisonRule (#44725)

fix months_add/ months_sub/ years_add/years_sub compute wrong result
because SimplifyArithmeticComparisonRule
introduced by #25180
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Nov 30, 2024
…e wrong result because SimplifyArithmeticComparisonRule (apache#44725)

fix months_add/ months_sub/ years_add/years_sub compute wrong result
because SimplifyArithmeticComparisonRule
introduced by apache#25180

(cherry picked from commit 1ae91b8)
924060929 added a commit to 924060929/incubator-doris that referenced this pull request Nov 30, 2024
…e wrong result because SimplifyArithmeticComparisonRule (apache#44725)

fix months_add/ months_sub/ years_add/years_sub compute wrong result
because SimplifyArithmeticComparisonRule
introduced by apache#25180

(cherry picked from commit 1ae91b8)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants