Skip to content

[Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions #23874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
JNSimba merged 1 commit into from
Sep 5, 2023
Merged

[Fix](Plan)StreamLoad cannot be parsed correctly when it contains complex where conditions #23874

JNSimba merged 1 commit into from
Sep 5, 2023

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Sep 4, 2023

Steps to reproduce：

 curl --location-trusted  -H "Expect: 100-continue" -X PUT  -u root:  -H "format:json" -H "where:d = 'doris' or d = 'nb'"  -d '{"a":"3","b":"2","c":"389","d":"nb"}' http://localhost:8030/api/kris/test/_stream_load

java.lang.IllegalStateException: null
        at com.google.common.base.Preconditions.checkState(Preconditions.java:492) ~[guava-30.0-jre.jar:?]
        at org.apache.doris.analysis.SlotRef.getTableName(SlotRef.java:264) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.rewriteOrToIn(ExtractCommonFactorsRule.java:532) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.makeCompoundRemaining(ExtractCommonFactorsRule.java:455) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.extractCommonFactors(ExtractCommonFactorsRule.java:205) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExtractCommonFactorsRule.apply(ExtractCommonFactorsRule.java:80) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExprRewriter.applyRuleOnce(ExprRewriter.java:146) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.rewrite.ExprRewriter.rewrite(ExprRewriter.java:139) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.initWhereExpr(ExternalFileScanNode.java:312) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.initAndSetWhereExpr(ExternalFileScanNode.java:293) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.initParamCreateContexts(ExternalFileScanNode.java:274) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.external.ExternalFileScanNode.init(ExternalFileScanNode.java:207) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.StreamLoadPlanner.plan(StreamLoadPlanner.java:201) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.load.routineload.RoutineLoadJob.plan(RoutineLoadJob.java:804) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.load.routineload.KafkaTaskInfo.rePlan(KafkaTaskInfo.java:117) ~[doris-fe.jar:1.2-SNAPSHOT]

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Sep 4, 2023

run buildall

1 similar comment
@CalvinKirs
Copy link
Member Author

CalvinKirs commented Sep 4, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Sep 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.12 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17161949833 Bytes

@doris-robot
Copy link

doris-robot commented Sep 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.66 seconds
stream load tsv: 532 seconds loaded 74807831229 Bytes, about 134 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.9 seconds inserted 10000000 Rows, about 346K ops/s
storage size: 17162029829 Bytes

@CalvinKirs
Copy link
Member Author

CalvinKirs commented Sep 4, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Sep 4, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 50.18 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17162054916 Bytes

@CalvinKirs CalvinKirs force-pushed the streamload-where-cause branch from e11a76f to 8e35f43 Compare September 5, 2023 01:29
@CalvinKirs
Copy link
Member Author

CalvinKirs commented Sep 5, 2023

run buildall

@doris-robot
Copy link

doris-robot commented Sep 5, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.67 seconds
stream load tsv: 534 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162194766 Bytes

dataroaring
dataroaring approved these changes Sep 5, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 5, 2023

PR approved by anyone and no changes requested.

JNSimba
JNSimba approved these changes Sep 5, 2023
View reviewed changes
Copy link
Member

@JNSimba JNSimba left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JNSimba JNSimba merged commit 4dac2d3 into apache:master Sep 5, 2023
@CalvinKirs CalvinKirs deleted the streamload-where-cause branch September 5, 2023 03:28
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Nov 1, 2023
@CalvinKirs
[2.0-branch][Fix](Plan)StreamLoad cannot be parsed correctly when it …
2495a1e 
…contains complex where conditions apache#23874
@CalvinKirs CalvinKirs mentioned this pull request Nov 1, 2023
Merged
xiaokang pushed a commit that referenced this pull request Nov 1, 2023
@CalvinKirs
[2.0-branch][Fix](Plan)StreamLoad cannot be parsed correctly when it …
a321f63 
…contains complex where conditions #23874 (#26212)
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
@CalvinKirs
[2.0-branch][Fix](Plan)StreamLoad cannot be parsed correctly when it …
fb6182c 
…contains complex where conditions apache#23874 (apache#26212)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JNSimba JNSimba JNSimba approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/1.2.9 dev/2.0.3-merged merge_conflict reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@CalvinKirs @doris-robot @JNSimba @dataroaring @xiaokang @morningman