Skip to content

Conversation

@CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Oct 15, 2024

bp #41391

## Proposed changes

The JOB's execution SQL is currently defined by an older CUP file, which
causes some issues with lexical analysis in the new optimizer as it
doesn't pass under the old optimizer. Since the JOB's underlying
execution already uses the new optimizer, we're planning to fully
migrate to ANTLR4 for consistency.

(cherry picked from commit 334b473)
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39539 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e4c3ee3213b77f6a03e65db0af7d16c72ce8a8c, data reload: false

------ Round 1 ----------------------------------
q1	17636	7368	7231	7231
q2	2032	153	178	153
q3	10772	1046	1096	1046
q4	10548	736	712	712
q5	7742	2789	2702	2702
q6	228	145	145	145
q7	996	630	604	604
q8	9566	1870	1935	1870
q9	7754	6343	6377	6343
q10	7068	2283	2333	2283
q11	441	256	246	246
q12	403	221	215	215
q13	17784	2951	2987	2951
q14	232	216	207	207
q15	561	504	521	504
q16	510	402	404	402
q17	962	556	489	489
q18	7087	6603	6478	6478
q19	3514	983	948	948
q20	549	265	271	265
q21	3920	3101	2770	2770
q22	1059	986	975	975
Total cold run time: 111364 ms
Total hot run time: 39539 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7321	7221	7211	7211
q2	319	221	235	221
q3	2892	2826	2843	2826
q4	1991	1780	1800	1780
q5	5683	5675	5673	5673
q6	225	145	145	145
q7	2195	1782	1804	1782
q8	3331	3550	3458	3458
q9	8717	8771	8713	8713
q10	3500	3484	3511	3484
q11	573	491	481	481
q12	790	637	597	597
q13	16441	3134	3172	3134
q14	301	286	274	274
q15	572	523	524	523
q16	520	454	460	454
q17	1848	1600	1601	1600
q18	8178	7541	7667	7541
q19	5857	1559	1435	1435
q20	2070	1845	1818	1818
q21	5337	5213	5047	5047
q22	1089	1073	1012	1012
Total cold run time: 79750 ms
Total hot run time: 59209 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193497 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e4c3ee3213b77f6a03e65db0af7d16c72ce8a8c, data reload: false

query1	1223	876	858	858
query2	6278	2069	1987	1987
query3	10780	3749	3777	3749
query4	64069	29147	23466	23466
query5	4998	444	457	444
query6	387	158	160	158
query7	5407	305	299	299
query8	279	201	204	201
query9	8499	2672	2650	2650
query10	459	283	261	261
query11	16941	15131	15580	15131
query12	160	100	108	100
query13	1565	454	436	436
query14	10822	7448	7263	7263
query15	222	192	172	172
query16	7099	506	497	497
query17	1164	600	598	598
query18	1733	312	305	305
query19	211	173	154	154
query20	121	108	110	108
query21	215	104	103	103
query22	4556	4387	4470	4387
query23	34722	33938	33755	33755
query24	6004	2896	2800	2800
query25	513	400	393	393
query26	652	167	158	158
query27	1648	300	285	285
query28	3908	2175	2157	2157
query29	675	435	433	433
query30	226	157	151	151
query31	926	771	782	771
query32	70	53	54	53
query33	387	285	299	285
query34	877	499	488	488
query35	878	747	743	743
query36	1067	931	949	931
query37	130	80	73	73
query38	3900	3832	3988	3832
query39	1474	1418	1425	1418
query40	202	98	99	98
query41	43	42	45	42
query42	113	97	93	93
query43	518	484	472	472
query44	1148	765	763	763
query45	193	164	166	164
query46	1116	710	711	710
query47	1922	1812	1818	1812
query48	422	337	324	324
query49	700	364	368	364
query50	811	410	392	392
query51	7519	6856	6826	6826
query52	101	88	86	86
query53	255	183	179	179
query54	582	456	441	441
query55	74	77	76	76
query56	264	248	248	248
query57	1259	1136	1164	1136
query58	231	226	235	226
query59	2991	2851	3073	2851
query60	289	263	259	259
query61	103	98	95	95
query62	751	639	665	639
query63	210	180	178	178
query64	1310	631	582	582
query65	3246	3147	3207	3147
query66	656	301	301	301
query67	15725	15263	15325	15263
query68	4056	541	529	529
query69	765	289	287	287
query70	1065	1118	1162	1118
query71	487	273	278	273
query72	8084	3974	4002	3974
query73	785	336	339	336
query74	10420	8914	9048	8914
query75	4621	2656	2631	2631
query76	3899	916	922	916
query77	784	286	295	286
query78	9976	9234	9130	9130
query79	4646	567	590	567
query80	2461	458	463	458
query81	565	223	227	223
query82	1386	134	131	131
query83	312	138	136	136
query84	302	77	76	76
query85	1145	291	284	284
query86	445	290	302	290
query87	4564	4232	4287	4232
query88	5419	2367	2390	2367
query89	406	285	292	285
query90	2264	186	181	181
query91	132	104	103	103
query92	57	46	45	45
query93	5436	529	524	524
query94	1079	283	284	283
query95	343	249	248	248
query96	616	271	277	271
query97	3274	3160	3080	3080
query98	208	194	200	194
query99	1824	1314	1305	1305
Total cold run time: 329825 ms
Total hot run time: 193497 ms

@CalvinKirs
Copy link
Member Author

run feut

@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40179 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9e4c3ee3213b77f6a03e65db0af7d16c72ce8a8c, data reload: false

------ Round 1 ----------------------------------
q1	17594	7439	7267	7267
q2	2036	151	154	151
q3	10783	1040	1123	1040
q4	10543	764	748	748
q5	7733	2840	2721	2721
q6	231	148	148	148
q7	996	644	624	624
q8	9377	1951	1981	1951
q9	6553	6440	6403	6403
q10	7001	2281	2299	2281
q11	443	251	237	237
q12	394	223	211	211
q13	17773	2930	2961	2930
q14	231	208	206	206
q15	553	507	513	507
q16	495	414	413	413
q17	963	576	511	511
q18	7353	6607	6658	6607
q19	2434	1025	1133	1025
q20	572	272	261	261
q21	3913	3180	2969	2969
q22	1090	1012	968	968
Total cold run time: 109061 ms
Total hot run time: 40179 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7339	7251	7199	7199
q2	314	225	223	223
q3	2835	2832	2858	2832
q4	2033	1818	1760	1760
q5	5664	5722	5708	5708
q6	220	145	143	143
q7	2226	1749	1767	1749
q8	3321	3527	3478	3478
q9	8738	8844	8858	8844
q10	3515	3481	3518	3481
q11	589	489	485	485
q12	769	586	578	578
q13	16451	3090	3107	3090
q14	318	281	278	278
q15	561	524	518	518
q16	541	473	486	473
q17	1834	1605	1624	1605
q18	8120	7784	7455	7455
q19	1648	1455	1462	1455
q20	2092	1888	1882	1882
q21	5474	5202	5172	5172
q22	1176	1023	1032	1023
Total cold run time: 75778 ms
Total hot run time: 59431 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193047 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9e4c3ee3213b77f6a03e65db0af7d16c72ce8a8c, data reload: false

query1	1268	926	870	870
query2	6273	1930	1911	1911
query3	10812	3846	3790	3790
query4	68161	25470	23325	23325
query5	5447	462	438	438
query6	484	176	162	162
query7	6328	303	304	303
query8	315	203	206	203
query9	9546	2671	2659	2659
query10	512	276	270	270
query11	17992	15063	15813	15063
query12	160	100	103	100
query13	1622	452	448	448
query14	11393	7293	7340	7293
query15	224	179	179	179
query16	7362	476	485	476
query17	1064	559	550	550
query18	1693	297	304	297
query19	201	148	154	148
query20	112	110	103	103
query21	210	105	97	97
query22	4378	4380	4155	4155
query23	34057	33512	33733	33512
query24	5627	2892	2917	2892
query25	501	417	394	394
query26	685	160	164	160
query27	1692	295	302	295
query28	4079	2217	2210	2210
query29	659	438	424	424
query30	239	155	151	151
query31	1000	783	787	783
query32	69	54	55	54
query33	444	286	285	285
query34	907	486	493	486
query35	870	733	717	717
query36	1069	930	950	930
query37	138	79	78	78
query38	3916	3918	3876	3876
query39	1468	1423	1400	1400
query40	206	95	102	95
query41	45	43	42	42
query42	115	98	95	95
query43	519	492	477	477
query44	1132	773	776	773
query45	194	164	163	163
query46	1122	718	720	718
query47	1918	1807	1784	1784
query48	424	346	335	335
query49	706	389	384	384
query50	812	404	411	404
query51	7021	6963	6953	6953
query52	124	90	88	88
query53	256	177	182	177
query54	570	451	458	451
query55	76	78	74	74
query56	282	258	247	247
query57	1235	1141	1127	1127
query58	228	224	239	224
query59	2999	3014	2878	2878
query60	294	263	253	253
query61	105	104	100	100
query62	781	653	654	653
query63	214	184	187	184
query64	1696	643	616	616
query65	3243	3165	3145	3145
query66	738	301	324	301
query67	15597	15314	15231	15231
query68	4569	547	546	546
query69	508	278	289	278
query70	1137	1131	1145	1131
query71	419	259	268	259
query72	6707	3887	3868	3868
query73	762	334	336	334
query74	9917	8960	8964	8960
query75	3376	2642	2634	2634
query76	2857	932	890	890
query77	524	286	282	282
query78	11414	9708	9116	9116
query79	7703	592	584	584
query80	1094	433	441	433
query81	526	223	222	222
query82	828	128	124	124
query83	266	140	137	137
query84	292	78	75	75
query85	990	296	294	294
query86	455	277	285	277
query87	4472	4207	4317	4207
query88	4606	2388	2408	2388
query89	472	298	288	288
query90	2019	181	185	181
query91	141	105	103	103
query92	60	46	45	45
query93	4629	537	526	526
query94	854	286	289	286
query95	336	242	252	242
query96	627	277	278	277
query97	3330	3139	3123	3123
query98	212	195	199	195
query99	1576	1287	1283	1283
Total cold run time: 333307 ms
Total hot run time: 193047 ms

@morningman morningman merged commit 1a61cb4 into apache:branch-3.0 Oct 18, 2024
@CalvinKirs CalvinKirs deleted the branch-3.0-41391 branch October 18, 2024 03:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants