Skip to content

Conversation

@morningman
Copy link
Contributor

If trigger "fetching partition in batch mode" feature, the explain verbose for external table may cause NPE.
Because in batch mode, the scan range is not initialized in plan phase.

org.apache.doris.common.NereidsException: errCode = 2, detailMessage = java.lang.NullPointerException
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:591) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:554) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:544) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:323) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:234) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = java.lang.NullPointerException
        ... 13 more
Caused by: java.lang.NullPointerException
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.collect.AbstractMultimap.putAll(AbstractMultimap.java:83) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.collect.ArrayListMultimap.putAll(ArrayListMultimap.java:62) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.datasource.FileScanNode.getNodeExplainString(FileScanNode.java:141) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.getExplainString(PlanNode.java:548) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanFragment.getExplainString(PlanFragment.java:369) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.Planner.getExplainString(Planner.java:93) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.getExplainString(NereidsPlanner.java:627) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:736) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41589 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e956fc9ff63212e4ed7b5cf7dc9c03e7d335d45b, data reload: false

------ Round 1 ----------------------------------
q1	17600	7378	7289	7289
q2	2033	160	157	157
q3	10583	1131	1233	1131
q4	10226	780	782	780
q5	7750	3169	3114	3114
q6	234	147	150	147
q7	1025	633	618	618
q8	9685	2074	2038	2038
q9	8528	6436	6497	6436
q10	7094	2292	2303	2292
q11	434	253	256	253
q12	529	218	220	218
q13	17791	2985	2989	2985
q14	236	211	222	211
q15	555	537	524	524
q16	680	625	644	625
q17	1002	842	835	835
q18	7269	6836	6652	6652
q19	1406	1032	1021	1021
q20	587	293	292	292
q21	4041	3154	2967	2967
q22	1093	1013	1004	1004
Total cold run time: 110381 ms
Total hot run time: 41589 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7215	7196	7188	7188
q2	329	226	231	226
q3	2921	2793	2757	2757
q4	1968	1740	1687	1687
q5	5412	5429	5480	5429
q6	226	141	142	141
q7	2117	1732	1733	1732
q8	3200	3339	3336	3336
q9	8497	8461	8479	8461
q10	3376	3364	3378	3364
q11	578	474	472	472
q12	780	572	567	567
q13	9561	3015	2984	2984
q14	304	267	256	256
q15	561	509	509	509
q16	709	652	668	652
q17	1782	1590	1543	1543
q18	7673	7530	7455	7455
q19	1676	1614	1320	1320
q20	2068	1796	1831	1796
q21	5296	5263	5213	5213
q22	1090	1016	987	987
Total cold run time: 67339 ms
Total hot run time: 58075 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194686 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e956fc9ff63212e4ed7b5cf7dc9c03e7d335d45b, data reload: false

query1	901	387	376	376
query2	4772	2091	2032	2032
query3	3428	213	230	213
query4	31804	23428	23391	23391
query5	2669	477	480	477
query6	225	160	162	160
query7	3976	300	305	300
query8	266	207	217	207
query9	9206	2741	2715	2715
query10	473	305	271	271
query11	18203	15273	15153	15153
query12	160	100	109	100
query13	1649	407	396	396
query14	9277	6909	7317	6909
query15	268	170	180	170
query16	6798	454	471	454
query17	1358	578	554	554
query18	1865	305	309	305
query19	267	149	145	145
query20	117	108	113	108
query21	223	105	104	104
query22	4599	4225	4305	4225
query23	34675	33876	33689	33689
query24	8240	3003	2922	2922
query25	636	397	386	386
query26	1022	162	161	161
query27	2178	292	288	288
query28	6269	2502	2489	2489
query29	797	427	438	427
query30	320	159	154	154
query31	1019	795	855	795
query32	93	58	60	58
query33	674	312	297	297
query34	909	494	482	482
query35	841	752	740	740
query36	1085	940	931	931
query37	152	85	85	85
query38	4027	3905	4041	3905
query39	1481	1417	1433	1417
query40	202	98	100	98
query41	52	50	50	50
query42	117	98	96	96
query43	529	489	486	486
query44	1162	836	803	803
query45	194	166	169	166
query46	1135	735	751	735
query47	1902	1821	1837	1821
query48	451	356	359	356
query49	931	424	399	399
query50	815	408	405	405
query51	6969	6915	7044	6915
query52	103	86	94	86
query53	261	187	181	181
query54	993	465	493	465
query55	78	75	78	75
query56	289	257	271	257
query57	1220	1100	1099	1099
query58	249	241	238	238
query59	3305	3075	3245	3075
query60	304	278	273	273
query61	122	123	149	123
query62	779	655	668	655
query63	227	189	180	180
query64	3643	628	621	621
query65	3284	3249	3159	3159
query66	868	300	289	289
query67	16219	15930	15716	15716
query68	2916	871	857	857
query69	451	338	348	338
query70	1236	1139	1202	1139
query71	330	320	336	320
query72	4999	3444	3341	3341
query73	594	588	593	588
query74	9369	8909	9034	8909
query75	3091	2899	2901	2899
query76	2059	882	890	882
query77	409	362	366	362
query78	9432	9983	9306	9306
query79	947	906	865	865
query80	587	591	571	571
query81	461	250	250	250
query82	233	235	230	230
query83	158	162	168	162
query84	247	103	107	103
query85	673	364	354	354
query86	301	312	314	312
query87	4453	4419	4311	4311
query88	4716	4099	4072	4072
query89	368	355	364	355
query90	844	322	315	315
query91	161	170	169	169
query92	77	71	72	71
query93	914	928	917	917
query94	365	398	350	350
query95	425	407	407	407
query96	490	492	494	492
query97	3139	3093	3107	3093
query98	223	227	220	220
query99	1327	1346	1313	1313
Total cold run time: 269320 ms
Total hot run time: 194686 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e956fc9ff63212e4ed7b5cf7dc9c03e7d335d45b, data reload: false

query1	0.04	0.05	0.04
query2	0.07	0.03	0.03
query3	0.22	0.06	0.06
query4	1.64	0.10	0.10
query5	0.50	0.52	0.51
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.50
query10	0.57	0.56	0.55
query11	0.14	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.61	0.59
query14	2.99	2.96	3.02
query15	0.91	0.82	0.81
query16	0.38	0.38	0.38
query17	1.02	1.03	1.06
query18	0.19	0.19	0.18
query19	1.94	1.90	1.94
query20	0.01	0.01	0.01
query21	15.36	0.59	0.56
query22	2.74	3.26	1.84
query23	17.47	0.88	0.70
query24	2.85	1.26	1.48
query25	0.20	0.18	0.06
query26	0.44	0.15	0.14
query27	0.04	0.05	0.05
query28	10.27	1.09	1.06
query29	12.56	3.26	3.24
query30	0.25	0.06	0.06
query31	2.84	0.39	0.38
query32	3.29	0.46	0.46
query33	2.96	3.01	3.04
query34	16.52	4.36	4.40
query35	4.38	4.38	4.46
query36	0.67	0.49	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 106.42 s
Total hot run time: 32.97 s

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 20, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit fb40871 into apache:master Sep 24, 2024
morningman added a commit to morningman/doris that referenced this pull request Sep 24, 2024
…apache#40969)

If trigger "fetching partition in batch mode" feature, the `explain
verbose` for external table may cause NPE.
Because in batch mode, the scan range is not initialized in plan phase.

```
org.apache.doris.common.NereidsException: errCode = 2, detailMessage = java.lang.NullPointerException
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:591) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:554) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:544) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:323) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:234) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = java.lang.NullPointerException
        ... 13 more
Caused by: java.lang.NullPointerException
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.collect.AbstractMultimap.putAll(AbstractMultimap.java:83) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.collect.ArrayListMultimap.putAll(ArrayListMultimap.java:62) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.datasource.FileScanNode.getNodeExplainString(FileScanNode.java:141) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.getExplainString(PlanNode.java:548) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanFragment.getExplainString(PlanFragment.java:369) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.Planner.getExplainString(Planner.java:93) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.getExplainString(NereidsPlanner.java:627) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:736) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more
```
dataroaring pushed a commit that referenced this pull request Sep 26, 2024
…#40969)

If trigger "fetching partition in batch mode" feature, the `explain
verbose` for external table may cause NPE.
Because in batch mode, the scan range is not initialized in plan phase.

```
org.apache.doris.common.NereidsException: errCode = 2, detailMessage = java.lang.NullPointerException
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:767) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:591) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.queryRetry(StmtExecutor.java:554) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:544) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:323) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:234) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:194) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:222) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:281) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: org.apache.doris.common.AnalysisException: errCode = 2, detailMessage = java.lang.NullPointerException
        ... 13 more
Caused by: java.lang.NullPointerException
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:903) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.collect.AbstractMultimap.putAll(AbstractMultimap.java:83) ~[guava-32.1.2-jre.jar:?]
        at com.google.common.collect.ArrayListMultimap.putAll(ArrayListMultimap.java:62) ~[guava-32.1.2-jre.jar:?]
        at org.apache.doris.datasource.FileScanNode.getNodeExplainString(FileScanNode.java:141) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanNode.getExplainString(PlanNode.java:548) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.PlanFragment.getExplainString(PlanFragment.java:369) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.planner.Planner.getExplainString(Planner.java:93) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.getExplainString(NereidsPlanner.java:627) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.trees.plans.commands.ExplainCommand.run(ExplainCommand.java:96) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:736) ~[doris-fe.jar:1.2-SNAPSHOT]
        ... 12 more
```
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Sep 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.2-merged p0_b reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants