Skip to content

[opt](Nereids) support more pattern in DeferMaterializeTopNResult #48747

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Mar 20, 2025
Merged

[opt](Nereids) support more pattern in DeferMaterializeTopNResult #48747

morrySnow merged 1 commit into from
Mar 20, 2025

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Mar 6, 2025

What problem does this PR solve?

Related PR: #47661, #22487

Problem Summary:

after this PR we support:

  1. TopN(Scan)
  2. Project(TopN(Project(Scan)))
  3. TopN(Filter(Scan))
  4. TopN(Project(Filter(Scan)))
  5. Project(TopN(Project(Filter(Scan))))

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 6, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morrySnow
Copy link
Contributor Author

morrySnow commented Mar 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 6, 2025

TPC-H: Total hot run time: 32859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 034a9a071310183d5b3172857461d161474c658a, data reload: false

------ Round 1 ----------------------------------
q1	17627	5280	5217	5217
q2	2054	302	172	172
q3	10584	1385	759	759
q4	10314	1061	592	592
q5	8790	2402	2402	2402
q6	208	175	137	137
q7	916	750	606	606
q8	9313	1254	1084	1084
q9	5352	4818	4815	4815
q10	6822	2317	1910	1910
q11	485	287	262	262
q12	356	354	224	224
q13	17790	3678	3072	3072
q14	232	221	216	216
q15	552	503	508	503
q16	627	619	585	585
q17	590	874	336	336
q18	7234	6401	6270	6270
q19	1558	956	557	557
q20	320	332	201	201
q21	2885	2198	1981	1981
q22	1044	987	958	958
Total cold run time: 105653 ms
Total hot run time: 32859 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5248	5193	5486	5193
q2	240	339	233	233
q3	2169	2696	2286	2286
q4	1468	1856	1432	1432
q5	4287	4114	4175	4114
q6	234	172	129	129
q7	2019	1937	1756	1756
q8	2602	2641	2582	2582
q9	7188	7216	7239	7216
q10	2991	3192	2762	2762
q11	590	510	485	485
q12	675	780	632	632
q13	3606	3917	3240	3240
q14	271	288	274	274
q15	525	474	472	472
q16	654	690	648	648
q17	1183	1618	1307	1307
q18	7725	7575	7429	7429
q19	851	916	1020	916
q20	1987	2048	1969	1969
q21	5533	4900	4867	4867
q22	1086	1050	997	997
Total cold run time: 53132 ms
Total hot run time: 50939 ms

@doris-robot
Copy link

doris-robot commented Mar 6, 2025

TPC-DS: Total hot run time: 186022 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 034a9a071310183d5b3172857461d161474c658a, data reload: false

query1	981	382	392	382
query2	6532	1891	1913	1891
query3	6788	213	217	213
query4	26148	23533	23410	23410
query5	4311	658	473	473
query6	302	206	206	206
query7	4603	507	293	293
query8	296	249	237	237
query9	8645	2547	2578	2547
query10	456	324	253	253
query11	16080	15196	14860	14860
query12	171	117	107	107
query13	1698	528	397	397
query14	10093	6379	6658	6379
query15	218	187	172	172
query16	7668	658	492	492
query17	1201	746	577	577
query18	1990	425	305	305
query19	201	188	159	159
query20	128	117	116	116
query21	208	125	105	105
query22	4168	4423	4294	4294
query23	34047	33104	33027	33027
query24	7671	2378	2413	2378
query25	538	463	397	397
query26	1225	272	151	151
query27	2130	501	327	327
query28	3947	2394	2382	2382
query29	716	563	417	417
query30	286	218	187	187
query31	972	866	781	781
query32	78	65	60	60
query33	556	352	308	308
query34	782	862	499	499
query35	802	825	741	741
query36	977	985	903	903
query37	119	101	76	76
query38	4092	4249	4264	4249
query39	1440	1418	1370	1370
query40	226	115	103	103
query41	55	57	52	52
query42	119	109	106	106
query43	510	504	480	480
query44	1296	790	790	790
query45	172	171	164	164
query46	836	1039	627	627
query47	1761	1819	1701	1701
query48	366	418	296	296
query49	789	521	415	415
query50	672	750	417	417
query51	4211	4137	4113	4113
query52	108	116	94	94
query53	233	268	189	189
query54	485	504	411	411
query55	81	82	81	81
query56	287	277	250	250
query57	1144	1136	1067	1067
query58	249	262	253	253
query59	2619	2672	2569	2569
query60	283	264	280	264
query61	123	117	119	117
query62	820	709	648	648
query63	233	193	199	193
query64	4280	988	675	675
query65	4434	4303	4297	4297
query66	1053	399	301	301
query67	15741	15496	15313	15313
query68	8204	885	508	508
query69	460	316	272	272
query70	1202	1135	1113	1113
query71	456	292	269	269
query72	5543	3742	3720	3720
query73	773	739	345	345
query74	9157	9162	8865	8865
query75	3761	3192	2717	2717
query76	3680	1202	778	778
query77	798	395	283	283
query78	9949	10294	9317	9317
query79	1907	898	594	594
query80	674	520	456	456
query81	482	256	222	222
query82	669	176	98	98
query83	180	173	160	160
query84	244	93	74	74
query85	786	363	327	327
query86	332	275	277	275
query87	4477	4595	4451	4451
query88	2831	2219	2277	2219
query89	397	316	294	294
query90	1930	212	211	211
query91	152	148	107	107
query92	71	58	59	58
query93	1166	1062	591	591
query94	665	420	306	306
query95	354	269	265	265
query96	486	570	273	273
query97	3337	3437	3261	3261
query98	222	213	195	195
query99	1463	1412	1291	1291
Total cold run time: 273258 ms
Total hot run time: 186022 ms

@doris-robot
Copy link

doris-robot commented Mar 6, 2025

ClickBench: Total hot run time: 31.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 034a9a071310183d5b3172857461d161474c658a, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.54	0.57	0.56
query6	1.19	0.73	0.74
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.59	0.52	0.53
query10	0.57	0.58	0.57
query11	0.15	0.11	0.10
query12	0.14	0.12	0.12
query13	0.63	0.62	0.60
query14	2.82	2.67	2.74
query15	0.95	0.86	0.86
query16	0.39	0.40	0.40
query17	1.06	1.05	1.07
query18	0.21	0.21	0.20
query19	1.95	1.92	1.77
query20	0.01	0.01	0.01
query21	15.35	0.95	0.54
query22	0.77	1.26	0.73
query23	14.77	1.39	0.65
query24	7.35	1.38	0.85
query25	0.50	0.18	0.09
query26	0.65	0.17	0.14
query27	0.05	0.05	0.05
query28	9.21	0.89	0.44
query29	12.60	4.06	3.38
query30	0.25	0.09	0.07
query31	2.81	0.60	0.40
query32	3.22	0.57	0.48
query33	3.11	3.14	3.22
query34	15.57	5.19	4.55
query35	4.58	4.55	4.58
query36	0.68	0.51	0.50
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.2 s
Total hot run time: 31.32 s

@morrySnow morrySnow force-pushed the support_more_defer branch 2 times, most recently from 9e80ce4 to 50a2e4b Compare March 17, 2025 07:21
@morrySnow
Copy link
Contributor Author

morrySnow commented Mar 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-H: Total hot run time: 32555 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 50a2e4badcda9cac29318f11273f2ea1426cfbb7, data reload: false

------ Round 1 ----------------------------------
q1	24761	5060	5130	5060
q2	2062	322	174	174
q3	10359	1260	707	707
q4	10233	1016	539	539
q5	7561	2464	2379	2379
q6	189	165	136	136
q7	954	741	613	613
q8	9305	1385	1144	1144
q9	5035	4637	4686	4637
q10	6811	2330	1898	1898
q11	475	273	250	250
q12	361	359	222	222
q13	17779	3727	3055	3055
q14	243	233	212	212
q15	541	484	482	482
q16	619	639	594	594
q17	581	880	356	356
q18	6874	6423	6465	6423
q19	1572	954	566	566
q20	334	359	201	201
q21	2885	2133	1927	1927
q22	1024	982	980	980
Total cold run time: 110558 ms
Total hot run time: 32555 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5620	5159	5133	5133
q2	238	323	226	226
q3	2169	2645	2342	2342
q4	1420	1806	1408	1408
q5	4317	4203	4506	4203
q6	230	174	133	133
q7	2019	1898	1805	1805
q8	2673	2695	2583	2583
q9	7444	7298	7022	7022
q10	2993	3218	2748	2748
q11	576	515	491	491
q12	702	774	617	617
q13	3517	3910	3331	3331
q14	288	316	288	288
q15	538	489	470	470
q16	630	670	684	670
q17	1185	1613	1392	1392
q18	7826	7649	7500	7500
q19	840	843	868	843
q20	2006	1965	1793	1793
q21	5361	4938	4982	4938
q22	1126	1101	1016	1016
Total cold run time: 53718 ms
Total hot run time: 50952 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

TPC-DS: Total hot run time: 193027 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 50a2e4badcda9cac29318f11273f2ea1426cfbb7, data reload: false

query1	1388	1090	1021	1021
query2	6219	1943	1966	1943
query3	11145	4559	4570	4559
query4	25256	23619	23483	23483
query5	4710	651	488	488
query6	309	201	190	190
query7	3999	499	295	295
query8	318	252	247	247
query9	8522	2652	2654	2652
query10	505	323	256	256
query11	15578	15048	14775	14775
query12	171	111	103	103
query13	1569	525	398	398
query14	9470	6344	6697	6344
query15	205	186	178	178
query16	7665	671	482	482
query17	1248	762	591	591
query18	2011	423	339	339
query19	201	189	173	173
query20	126	122	122	122
query21	206	127	115	115
query22	4512	4631	4416	4416
query23	34298	33577	33617	33577
query24	7856	2451	2448	2448
query25	529	481	389	389
query26	1196	279	166	166
query27	2274	509	347	347
query28	4397	2476	2444	2444
query29	712	567	435	435
query30	291	228	201	201
query31	941	881	830	830
query32	72	59	66	59
query33	540	376	313	313
query34	802	868	510	510
query35	801	850	775	775
query36	1005	1011	942	942
query37	130	99	73	73
query38	4208	4326	4285	4285
query39	1523	1446	1448	1446
query40	206	116	106	106
query41	51	53	50	50
query42	121	106	109	106
query43	511	517	489	489
query44	1315	807	815	807
query45	181	175	171	171
query46	866	1054	645	645
query47	1847	1876	1774	1774
query48	399	420	327	327
query49	781	504	436	436
query50	695	756	412	412
query51	4340	4351	4259	4259
query52	132	102	101	101
query53	246	270	192	192
query54	521	506	424	424
query55	84	81	88	81
query56	280	254	261	254
query57	1166	1159	1110	1110
query58	242	260	252	252
query59	2886	2985	2736	2736
query60	286	285	279	279
query61	128	123	135	123
query62	784	778	766	766
query63	235	195	194	194
query64	4183	1037	782	782
query65	4434	4317	4337	4317
query66	920	402	294	294
query67	15641	15495	15285	15285
query68	8825	885	503	503
query69	451	291	259	259
query70	1240	1107	1138	1107
query71	444	286	263	263
query72	5561	3557	3783	3557
query73	644	724	344	344
query74	9129	9227	8835	8835
query75	4030	3130	2668	2668
query76	3235	1181	747	747
query77	920	365	271	271
query78	10176	10254	9405	9405
query79	2323	820	595	595
query80	624	531	481	481
query81	497	256	230	230
query82	718	128	98	98
query83	187	178	149	149
query84	239	91	70	70
query85	778	364	311	311
query86	341	314	288	288
query87	4731	4580	4451	4451
query88	3662	2289	2278	2278
query89	388	320	277	277
query90	1868	220	219	219
query91	141	139	113	113
query92	77	61	63	61
query93	1567	1079	589	589
query94	660	404	308	308
query95	360	266	260	260
query96	492	574	281	281
query97	3395	3407	3357	3357
query98	228	202	203	202
query99	1395	1411	1268	1268
Total cold run time: 279715 ms
Total hot run time: 193027 ms

@doris-robot
Copy link

doris-robot commented Mar 17, 2025

ClickBench: Total hot run time: 30.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 50a2e4badcda9cac29318f11273f2ea1426cfbb7, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.59	0.19	0.20
query5	0.59	0.59	0.60
query6	1.18	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.54
query10	0.57	0.59	0.60
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.61	0.60	0.60
query14	2.79	2.75	2.83
query15	0.92	0.84	0.87
query16	0.37	0.39	0.38
query17	1.01	1.03	1.01
query18	0.22	0.20	0.20
query19	1.87	1.97	1.78
query20	0.01	0.01	0.01
query21	15.35	0.87	0.55
query22	0.76	1.21	0.60
query23	15.00	1.40	0.60
query24	7.94	1.73	0.36
query25	0.30	0.22	0.14
query26	0.67	0.16	0.13
query27	0.06	0.05	0.05
query28	8.97	0.86	0.45
query29	12.53	4.01	3.30
query30	0.25	0.08	0.06
query31	2.81	0.58	0.40
query32	3.23	0.54	0.46
query33	3.04	2.98	3.01
query34	15.85	5.17	4.51
query35	4.58	4.56	4.57
query36	0.65	0.50	0.46
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.58 s
Total hot run time: 30.7 s

@morrySnow
Copy link
Contributor Author

morrySnow commented Mar 18, 2025

run compile

@morrySnow morrySnow force-pushed the support_more_defer branch from 50a2e4b to 8076f9c Compare March 18, 2025 05:55
@morrySnow
Copy link
Contributor Author

morrySnow commented Mar 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-H: Total hot run time: 32873 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8076f9cb2855a952323afb365818ca9fc4b76353, data reload: false

------ Round 1 ----------------------------------
q1	24443	5082	5066	5066
q2	2041	297	180	180
q3	10393	1281	718	718
q4	10212	1027	597	597
q5	7498	2414	2342	2342
q6	190	163	133	133
q7	939	738	600	600
q8	9309	1304	1300	1300
q9	5178	4630	4717	4630
q10	6804	2323	1893	1893
q11	473	284	261	261
q12	350	368	228	228
q13	17772	3772	3107	3107
q14	222	231	211	211
q15	534	490	489	489
q16	611	618	592	592
q17	570	887	365	365
q18	6870	6545	6392	6392
q19	1223	953	559	559
q20	313	343	203	203
q21	2833	2322	2014	2014
q22	1027	1024	993	993
Total cold run time: 109805 ms
Total hot run time: 32873 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5184	5179	5252	5179
q2	248	335	229	229
q3	2201	2651	2273	2273
q4	1445	1785	1388	1388
q5	4258	4132	4514	4132
q6	206	168	127	127
q7	2018	1939	1787	1787
q8	2638	2550	2651	2550
q9	7059	7061	7004	7004
q10	2988	3202	2796	2796
q11	564	527	494	494
q12	691	761	595	595
q13	3537	3908	3247	3247
q14	299	328	267	267
q15	523	479	490	479
q16	627	677	646	646
q17	1183	1611	1334	1334
q18	7647	7448	7512	7448
q19	807	815	848	815
q20	2031	2018	1860	1860
q21	5533	4889	4738	4738
q22	1063	1077	1014	1014
Total cold run time: 52750 ms
Total hot run time: 50402 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-DS: Total hot run time: 184931 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8076f9cb2855a952323afb365818ca9fc4b76353, data reload: false

query1	984	471	463	463
query2	6557	1913	1845	1845
query3	6791	235	220	220
query4	26604	23795	22989	22989
query5	4403	657	472	472
query6	296	189	174	174
query7	4593	497	286	286
query8	292	232	219	219
query9	8567	2518	2539	2518
query10	491	314	257	257
query11	15448	15085	15022	15022
query12	161	113	103	103
query13	1639	530	379	379
query14	9210	7031	6500	6500
query15	201	181	170	170
query16	7147	641	411	411
query17	913	723	566	566
query18	1955	412	321	321
query19	193	183	154	154
query20	121	113	112	112
query21	210	125	112	112
query22	4052	4280	4129	4129
query23	33979	33155	32870	32870
query24	7679	2362	2405	2362
query25	547	449	390	390
query26	1234	272	155	155
query27	2126	492	325	325
query28	3912	2376	2383	2376
query29	741	561	411	411
query30	290	222	231	222
query31	955	847	779	779
query32	74	66	66	66
query33	571	367	321	321
query34	811	846	490	490
query35	777	834	732	732
query36	943	986	885	885
query37	120	101	77	77
query38	4115	4208	4098	4098
query39	1444	1397	1560	1397
query40	212	114	100	100
query41	54	51	50	50
query42	118	106	103	103
query43	489	508	457	457
query44	1323	784	799	784
query45	172	170	163	163
query46	830	1032	623	623
query47	1738	1772	1720	1720
query48	393	405	302	302
query49	785	491	430	430
query50	704	732	425	425
query51	4169	4256	4123	4123
query52	106	110	96	96
query53	240	260	188	188
query54	499	506	440	440
query55	79	78	85	78
query56	253	267	262	262
query57	1132	1136	1047	1047
query58	253	236	244	236
query59	2572	2637	2633	2633
query60	294	267	258	258
query61	125	120	120	120
query62	793	712	675	675
query63	240	196	192	192
query64	4390	1001	669	669
query65	4426	4348	4333	4333
query66	1147	401	328	328
query67	15798	15557	15224	15224
query68	7255	887	520	520
query69	483	312	271	271
query70	1144	1130	1102	1102
query71	416	316	266	266
query72	5785	3590	3764	3590
query73	744	722	342	342
query74	9084	8931	8690	8690
query75	3167	3182	2724	2724
query76	3184	1184	845	845
query77	500	373	283	283
query78	9949	10103	9229	9229
query79	2383	821	592	592
query80	697	522	432	432
query81	508	254	220	220
query82	482	125	91	91
query83	169	217	155	155
query84	289	96	69	69
query85	783	356	308	308
query86	373	283	287	283
query87	4596	4431	4496	4431
query88	3933	2226	2239	2226
query89	384	313	279	279
query90	1886	207	213	207
query91	139	134	115	115
query92	71	63	56	56
query93	2183	1029	577	577
query94	694	416	308	308
query95	355	267	256	256
query96	478	571	273	273
query97	3335	3378	3297	3297
query98	231	215	201	201
query99	1339	1375	1256	1256
Total cold run time: 271499 ms
Total hot run time: 184931 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8076f9cb2855a952323afb365818ca9fc4b76353, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.24	0.19	0.19
query4	1.58	0.19	0.20
query5	0.60	0.58	0.59
query6	1.19	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.59	0.52	0.51
query10	0.58	0.59	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	2.68	2.82	2.70
query15	0.92	0.85	0.86
query16	0.37	0.38	0.37
query17	1.05	1.06	1.00
query18	0.21	0.19	0.20
query19	1.91	1.86	1.90
query20	0.02	0.01	0.02
query21	15.35	0.93	0.55
query22	0.76	1.17	0.71
query23	14.90	1.37	0.63
query24	7.18	1.89	0.56
query25	0.54	0.23	0.13
query26	0.64	0.16	0.14
query27	0.05	0.05	0.05
query28	9.24	0.89	0.44
query29	12.56	3.95	3.34
query30	0.25	0.09	0.07
query31	2.83	0.60	0.39
query32	3.23	0.57	0.46
query33	2.98	3.04	3.03
query34	15.73	5.14	4.51
query35	4.49	4.50	4.47
query36	0.67	0.49	0.48
query37	0.09	0.07	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.09	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.04
Total cold run time: 104.96 s
Total hot run time: 31.02 s

@morrySnow
[opt](Nereids) support more pattern in DeferMaterializeTopNResult
5b306d4 
after this PR we support:

1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
@morrySnow morrySnow force-pushed the support_more_defer branch from 8076f9c to 5b306d4 Compare March 18, 2025 06:51
@morrySnow
Copy link
Contributor Author

morrySnow commented Mar 18, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-H: Total hot run time: 32406 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b306d4014bfb5d8947724722ec5653db5d1d456, data reload: false

------ Round 1 ----------------------------------
q1	24136	5127	5009	5009
q2	2039	313	185	185
q3	10434	1229	695	695
q4	10210	1021	525	525
q5	7593	2414	2362	2362
q6	182	162	131	131
q7	919	747	606	606
q8	9310	1323	1097	1097
q9	5025	4678	4676	4676
q10	6818	2291	1903	1903
q11	476	270	252	252
q12	351	349	214	214
q13	17798	3677	3099	3099
q14	247	223	221	221
q15	530	482	481	481
q16	623	622	572	572
q17	579	875	336	336
q18	6772	6408	6384	6384
q19	1596	973	552	552
q20	310	329	188	188
q21	2742	2222	1935	1935
q22	1078	1043	983	983
Total cold run time: 109768 ms
Total hot run time: 32406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5189	5096	5091	5091
q2	246	333	232	232
q3	2181	2651	2274	2274
q4	1431	1822	1369	1369
q5	4232	4162	4371	4162
q6	221	174	126	126
q7	1984	1915	1781	1781
q8	2647	2746	2642	2642
q9	7343	7191	7245	7191
q10	3010	3168	2648	2648
q11	573	525	493	493
q12	689	815	632	632
q13	3505	3933	3284	3284
q14	280	295	272	272
q15	525	479	490	479
q16	652	679	642	642
q17	1143	1575	1385	1385
q18	7756	7578	7544	7544
q19	863	933	1013	933
q20	2033	2070	1912	1912
q21	5468	4818	4934	4818
q22	1170	1062	1051	1051
Total cold run time: 53141 ms
Total hot run time: 50961 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

TPC-DS: Total hot run time: 191576 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b306d4014bfb5d8947724722ec5653db5d1d456, data reload: false

query1	1407	1029	1017	1017
query2	6418	1939	1923	1923
query3	11159	4535	4632	4535
query4	25976	23868	23033	23033
query5	4817	670	466	466
query6	302	196	198	196
query7	3986	505	292	292
query8	305	269	263	263
query9	8525	2599	2577	2577
query10	496	309	268	268
query11	15824	15108	15040	15040
query12	165	108	102	102
query13	1562	533	388	388
query14	9691	5980	6431	5980
query15	204	187	162	162
query16	7574	637	494	494
query17	1086	735	574	574
query18	2019	408	318	318
query19	186	184	155	155
query20	145	116	126	116
query21	211	132	109	109
query22	4636	4879	4477	4477
query23	34305	33439	33332	33332
query24	8085	2435	2422	2422
query25	487	453	395	395
query26	1174	284	155	155
query27	2310	492	329	329
query28	4205	2467	2410	2410
query29	718	599	431	431
query30	276	220	190	190
query31	928	876	783	783
query32	71	62	60	60
query33	562	394	299	299
query34	789	866	516	516
query35	817	853	756	756
query36	1007	1034	911	911
query37	116	97	74	74
query38	4144	4223	4063	4063
query39	1487	1488	1464	1464
query40	210	121	102	102
query41	52	51	49	49
query42	122	110	105	105
query43	504	514	482	482
query44	1316	803	795	795
query45	195	175	167	167
query46	862	1054	656	656
query47	1845	1882	1825	1825
query48	413	432	307	307
query49	750	506	427	427
query50	687	734	405	405
query51	4297	4288	4333	4288
query52	103	109	100	100
query53	233	259	189	189
query54	491	486	411	411
query55	79	77	80	77
query56	292	270	258	258
query57	1166	1191	1138	1138
query58	247	251	234	234
query59	2736	2890	2721	2721
query60	294	286	266	266
query61	120	119	145	119
query62	776	757	676	676
query63	226	203	188	188
query64	4108	1142	857	857
query65	4571	4428	4454	4428
query66	961	392	290	290
query67	16179	15539	15507	15507
query68	9848	873	504	504
query69	475	315	258	258
query70	1198	1106	1121	1106
query71	489	285	266	266
query72	5118	3524	3789	3524
query73	795	722	355	355
query74	8963	8941	8965	8941
query75	4191	3119	2674	2674
query76	5095	1179	733	733
query77	993	371	270	270
query78	9944	10091	9323	9323
query79	4379	774	562	562
query80	582	516	435	435
query81	464	250	221	221
query82	249	133	97	97
query83	164	165	154	154
query84	287	92	72	72
query85	751	357	302	302
query86	332	321	292	292
query87	4507	4449	4352	4352
query88	2891	2225	2217	2217
query89	429	314	286	286
query90	2129	205	222	205
query91	137	134	106	106
query92	69	56	55	55
query93	2610	1026	570	570
query94	698	400	303	303
query95	349	266	261	261
query96	481	566	271	271
query97	3222	3362	3267	3267
query98	218	212	201	201
query99	1366	1431	1251	1251
Total cold run time: 285433 ms
Total hot run time: 191576 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2025

ClickBench: Total hot run time: 31.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b306d4014bfb5d8947724722ec5653db5d1d456, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.58	0.18	0.18
query5	0.57	0.57	0.60
query6	1.21	0.71	0.71
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.59	0.59	0.56
query11	0.15	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.69	2.66	2.68
query15	0.91	0.84	0.85
query16	0.38	0.38	0.38
query17	1.05	1.03	1.07
query18	0.22	0.19	0.20
query19	1.92	1.92	1.90
query20	0.01	0.02	0.01
query21	15.36	0.90	0.54
query22	0.77	1.14	0.77
query23	14.83	1.37	0.64
query24	6.84	1.95	0.69
query25	0.50	0.27	0.16
query26	0.58	0.16	0.13
query27	0.06	0.05	0.05
query28	9.69	0.83	0.43
query29	12.58	3.90	3.27
query30	0.25	0.09	0.06
query31	2.83	0.57	0.38
query32	3.22	0.54	0.46
query33	3.05	3.06	3.08
query34	15.66	5.17	4.58
query35	4.60	4.52	4.53
query36	0.66	0.48	0.48
query37	0.08	0.06	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.05 s
Total hot run time: 31.25 s

aytrack
aytrack approved these changes Mar 19, 2025
View reviewed changes
Copy link

@aytrack aytrack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 20, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 20, 2025

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 947463d into apache:master Mar 20, 2025
24 checks passed
@morrySnow morrySnow deleted the support_more_defer branch March 20, 2025 03:08
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@morrySnow @koarz
[opt](Nereids) support more pattern in DeferMaterializeTopNResult (ap…
bd2a11f 
…ache#48747)

### What problem does this PR solve?

Related PR: apache#47661, apache#22487

Problem Summary:

after this PR we support:

1. TopN(Scan)
2. Project(TopN(Project(Scan)))
3. TopN(Filter(Scan))
4. TopN(Project(Filter(Scan)))
5. Project(TopN(Project(Filter(Scan))))
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project
a214d49 
pick from #47661 #47975 #48747
@morrySnow morrySnow mentioned this pull request Jun 20, 2025
Merged
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project
97eda6a 
pick from #47661 #47975 #48747
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project
9371d86 
pick from #47661 #47975 #48747
morrySnow added a commit that referenced this pull request Jun 20, 2025
@morrySnow
branch-3.1: [opt](Nereids) support defer materialization with project (…
171c067 
…#52078)

pick from #47661 #47975 #48747
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Jun 30, 2025
@morrySnow
branch-3.0: [opt](Nereids) support defer materialization with project
fedaeea 
pick from apache#47661 apache#47975 apache#48747
@morrySnow morrySnow mentioned this pull request Jun 30, 2025
Merged
dataroaring pushed a commit to morrySnow/incubator-doris that referenced this pull request Jul 9, 2025
@morrySnow @dataroaring
branch-3.0: [opt](Nereids) support defer materialization with project
81f12f5 
pick from apache#47661 apache#47975 apache#48747
dataroaring pushed a commit that referenced this pull request Jul 10, 2025
@morrySnow
branch-3.0: [opt](Nereids) support defer materialization with project (…
1357b68 
…#52522)

pick from #47661 #47975 #48747
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@englefly englefly englefly approved these changes

+1 more reviewer

@aytrack aytrack aytrack approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morrySnow @Thearas @doris-robot @englefly @aytrack @dataroaring