Skip to content

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Feb 18, 2024

Proposed changes

exception thrown

org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: null
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:588) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:453) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:443) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:584) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:841) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
Caused by: java.lang.UnsupportedOperationException
        at com.google.common.collect.ImmutableCollection.add(ImmutableCollection.java:268) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.planner.PlanFragment.setTargetRuntimeFilterIds(PlanFragment.java:231) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.RuntimeFilter.assignToPlanNodes(RuntimeFilter.java:648) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.RuntimeFilterTranslator.finalize(RuntimeFilterTranslator.java:180) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.RuntimeFilterTranslator.createLegacyRuntimeFilter(RuntimeFilterTranslator.java:174) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.lambda$null$40(PhysicalPlanTranslator.java:1233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_312]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.lambda$visitPhysicalHashJoin$41(PhysicalPlanTranslator.java:1233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_312]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashJoin(PhysicalPlanTranslator.java:1231) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashJoin(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.accept(PhysicalHashJoin.java:128) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:1595) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalProject.accept(PhysicalProject.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:270) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute.accept(PhysicalDistribute.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:355) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:70) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:243) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:142) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:584) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 9 more

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@morrySnow
Copy link
Contributor Author

run buildall

@morrySnow morrySnow requested a review from englefly February 18, 2024 03:26
@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Feb 18, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 18, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 50384 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67d83c05033a7b4c0ee7390ed69b3792b55df80c, data reload: false

------ Round 1 ----------------------------------
q1	17534	4309	4323	4309
q2	2026	146	149	146
q3	10273	1939	1930	1930
q4	10097	1250	1336	1250
q5	8468	3957	3940	3940
q6	224	122	122	122
q7	2024	1575	1622	1575
q8	9273	2714	2790	2714
q9	10926	11197	10852	10852
q10	8645	3565	3545	3545
q11	411	233	233	233
q12	466	293	304	293
q13	18352	3969	4051	3969
q14	361	337	316	316
q15	501	459	454	454
q16	680	597	596	596
q17	1132	947	990	947
q18	7287	6962	6868	6868
q19	1699	1572	1542	1542
q20	542	322	290	290
q21	4470	4125	4097	4097
q22	509	396	402	396
Total cold run time: 115900 ms
Total hot run time: 50384 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4344	4287	4281	4281
q2	322	222	221	221
q3	4143	4156	4125	4125
q4	2750	2748	2759	2748
q5	7354	7217	7192	7192
q6	238	122	119	119
q7	3206	2818	2853	2818
q8	4333	4465	4570	4465
q9	17059	16867	17007	16867
q10	4246	4263	4269	4263
q11	753	710	722	710
q12	1025	835	861	835
q13	5038	3725	3742	3725
q14	446	436	416	416
q15	517	446	459	446
q16	753	698	702	698
q17	3881	3843	3830	3830
q18	8768	8922	8814	8814
q19	1705	1726	1697	1697
q20	2416	2193	2126	2126
q21	8513	8517	8583	8517
q22	1028	970	962	962
Total cold run time: 82838 ms
Total hot run time: 79875 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 238486 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67d83c05033a7b4c0ee7390ed69b3792b55df80c, data reload: false

query1	912	391	378	378
query2	6519	2263	2247	2247
query3	6917	207	202	202
query4	20714	18057	17959	17959
query5	19698	6528	6439	6439
query6	279	215	234	215
query7	4157	311	306	306
query8	240	252	227	227
query9	3121	2746	2620	2620
query10	415	285	296	285
query11	11297	10678	10782	10678
query12	124	72	72	72
query13	5580	648	645	645
query14	18087	13754	13229	13229
query15	356	233	244	233
query16	6427	260	260	260
query17	1766	1458	868	868
query18	2301	408	404	404
query19	208	153	152	152
query20	74	78	72	72
query21	187	98	94	94
query22	5247	4995	5028	4995
query23	32548	31857	31933	31857
query24	6914	6458	6497	6458
query25	514	441	411	411
query26	535	162	157	157
query27	1869	291	292	291
query28	6000	2227	2191	2191
query29	2958	2851	2857	2851
query30	241	164	163	163
query31	911	727	720	720
query32	72	60	60	60
query33	389	245	243	243
query34	852	475	459	459
query35	1119	931	937	931
query36	1516	1569	1368	1368
query37	86	62	61	61
query38	3107	2904	2947	2904
query39	1373	1320	1330	1320
query40	209	99	92	92
query41	35	32	30	30
query42	89	98	85	85
query43	572	562	708	562
query44	1130	721	720	720
query45	242	228	230	228
query46	1239	957	970	957
query47	1921	1811	1731	1731
query48	983	669	664	664
query49	606	372	370	370
query50	860	602	646	602
query51	5521	5493	5392	5392
query52	106	92	78	78
query53	436	328	312	312
query54	2663	2477	2467	2467
query55	85	69	74	69
query56	200	206	199	199
query57	1281	1122	1109	1109
query58	222	196	187	187
query59	3334	3205	3194	3194
query60	206	180	205	180
query61	89	81	82	81
query62	857	513	508	508
query63	478	337	332	332
query64	2557	1549	1433	1433
query65	3636	3550	3559	3550
query66	800	365	369	365
query67	15810	15733	15940	15733
query68	8399	663	673	663
query69	553	347	338	338
query70	1745	1729	1605	1605
query71	396	305	318	305
query72	6438	3383	3423	3383
query73	718	349	332	332
query74	6389	5893	5826	5826
query75	5180	3817	3750	3750
query76	5078	1135	1202	1135
query77	795	259	252	252
query78	32173	53558	47011	47011
query79	16519	632	639	632
query80	5364	382	387	382
query81	560	232	233	232
query82	951	94	95	94
query83	446	142	135	135
query84	259	72	66	66
query85	2631	279	274	274
query86	486	365	418	365
query87	3247	3014	2932	2932
query88	6867	2376	2327	2327
query89	462	278	275	275
query90	2517	200	206	200
query91	159	133	115	115
query92	61	51	49	49
query93	6233	585	560	560
query94	1854	203	203	203
query95	1089	1059	1071	1059
query96	642	324	327	324
query97	6583	6360	6364	6360
query98	188	169	174	169
query99	3932	928	878	878
Total cold run time: 350711 ms
Total hot run time: 238486 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 67d83c05033a7b4c0ee7390ed69b3792b55df80c, data reload: false

query1	0.02	0.02	0.02
query2	0.05	0.02	0.02
query3	0.24	0.06	0.04
query4	1.81	0.06	0.07
query5	0.53	0.53	0.52
query6	1.31	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.53	0.48	0.48
query10	0.54	0.54	0.56
query11	0.11	0.09	0.08
query12	0.12	0.09	0.09
query13	0.62	0.61	0.60
query14	0.77	0.79	0.80
query15	0.78	0.76	0.78
query16	0.39	0.37	0.36
query17	1.02	1.01	1.00
query18	0.24	0.24	0.25
query19	1.91	1.87	1.87
query20	0.02	0.01	0.00
query21	15.48	0.57	0.57
query22	1.94	2.04	1.73
query23	17.65	0.95	0.94
query24	7.79	1.38	0.94
query25	1.53	0.11	0.12
query26	0.36	0.15	0.14
query27	0.12	0.11	0.11
query28	6.13	0.72	0.74
query29	12.63	2.33	2.32
query30	0.60	0.53	0.48
query31	2.80	0.38	0.38
query32	3.39	0.48	0.51
query33	3.07	3.08	3.05
query34	15.26	4.81	4.76
query35	4.83	4.82	4.85
query36	1.06	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 106.07 s
Total hot run time: 30.92 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 67d83c05033a7b4c0ee7390ed69b3792b55df80c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@morrySnow morrySnow merged commit 17a9998 into apache:branch-2.0 Feb 18, 2024
@morrySnow morrySnow deleted the fix_add_rf_failed branch February 18, 2024 06:31
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…pache#31038)

exception thrown
```
org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = Unexpected exception: null
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:588) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:453) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:443) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:435) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.dispatch(ConnectProcessor.java:584) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.processOnce(ConnectProcessor.java:841) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_312]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_312]
Caused by: java.lang.UnsupportedOperationException
        at com.google.common.collect.ImmutableCollection.add(ImmutableCollection.java:268) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.planner.PlanFragment.setTargetRuntimeFilterIds(PlanFragment.java:231) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.RuntimeFilter.assignToPlanNodes(RuntimeFilter.java:648) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.RuntimeFilterTranslator.finalize(RuntimeFilterTranslator.java:180) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.RuntimeFilterTranslator.createLegacyRuntimeFilter(RuntimeFilterTranslator.java:174) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.lambda$null$40(PhysicalPlanTranslator.java:1233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_312]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.lambda$visitPhysicalHashJoin$41(PhysicalPlanTranslator.java:1233) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_312]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashJoin(PhysicalPlanTranslator.java:1231) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalHashJoin(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalHashJoin.accept(PhysicalHashJoin.java:128) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:1595) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalProject(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalProject.accept(PhysicalProject.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:270) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalDistribute(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalDistribute.accept(PhysicalDistribute.java:87) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:355) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.visitPhysicalResultSink(PhysicalPlanTranslator.java:216) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.physical.PhysicalResultSink.accept(PhysicalResultSink.java:70) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.glue.translator.PhysicalPlanTranslator.translatePlan(PhysicalPlanTranslator.java:243) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:142) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:584) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 9 more
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. area/planner Issues or PRs related to the query planner reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants