Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #48782

… struct infos are huge (#48782)

### What problem does this PR solve?

Opt materialized view rewrite performance when the num of struct infos
are huge

Optimize the recursive algorithm to reduce the number of recursive
calls. If a group has already been refreshed, skip subsequent refreshes.
@github-actions github-actions bot requested a review from dataroaring as a code owner March 10, 2025 07:03
@Thearas
Copy link
Contributor

Thearas commented Mar 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Mar 10, 2025
@Thearas
Copy link
Contributor

Thearas commented Mar 10, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40500 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c1383a3fb6aa9ee962e87befafe96d3ecd5f4d9, data reload: false

------ Round 1 ----------------------------------
q1	17584	7537	6710	6710
q2	2083	181	175	175
q3	10607	1111	1167	1111
q4	10479	754	812	754
q5	7771	2892	2782	2782
q6	229	136	137	136
q7	982	619	617	617
q8	9359	1958	2053	1958
q9	6588	6444	6456	6444
q10	7041	2276	2317	2276
q11	472	269	261	261
q12	409	218	212	212
q13	17806	3032	3020	3020
q14	238	217	204	204
q15	494	475	477	475
q16	677	611	573	573
q17	1007	637	545	545
q18	7341	6787	6831	6787
q19	1401	1053	1131	1053
q20	497	212	208	208
q21	3951	3260	3235	3235
q22	1081	976	964	964
Total cold run time: 108097 ms
Total hot run time: 40500 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6689	6654	6857	6654
q2	334	234	227	227
q3	2881	2891	2957	2891
q4	2072	1817	1859	1817
q5	5769	5791	5752	5752
q6	207	129	133	129
q7	2222	1846	1818	1818
q8	3429	3604	3570	3570
q9	8899	8964	8867	8867
q10	3574	3549	3549	3549
q11	614	507	505	505
q12	854	601	599	599
q13	10593	3181	3151	3151
q14	322	268	282	268
q15	524	471	475	471
q16	688	633	652	633
q17	1863	1628	1617	1617
q18	8440	7793	7708	7708
q19	1694	1609	1572	1572
q20	2068	1879	1869	1869
q21	5497	5600	5429	5429
q22	1165	1065	1040	1040
Total cold run time: 70398 ms
Total hot run time: 60136 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197152 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c1383a3fb6aa9ee962e87befafe96d3ecd5f4d9, data reload: false

query1	1296	922	884	884
query2	6265	2055	2041	2041
query3	10825	4408	4198	4198
query4	60941	29200	23659	23659
query5	5193	459	434	434
query6	396	189	188	188
query7	5479	312	309	309
query8	297	224	218	218
query9	8735	2603	2582	2582
query10	470	274	259	259
query11	17689	15268	15426	15268
query12	156	104	105	104
query13	1432	461	462	461
query14	10440	7604	7197	7197
query15	208	175	186	175
query16	7170	465	443	443
query17	1252	595	594	594
query18	1803	324	349	324
query19	210	160	160	160
query20	123	115	111	111
query21	208	108	108	108
query22	4754	4471	4834	4471
query23	34671	34009	34791	34009
query24	6165	2941	2971	2941
query25	543	430	443	430
query26	681	174	175	174
query27	1949	360	359	359
query28	3678	2509	2421	2421
query29	704	482	446	446
query30	236	170	168	168
query31	1004	794	836	794
query32	69	56	55	55
query33	419	277	280	277
query34	935	498	514	498
query35	860	740	722	722
query36	1114	973	957	957
query37	115	63	67	63
query38	4151	4017	3993	3993
query39	1514	1556	1465	1465
query40	206	96	97	96
query41	50	51	49	49
query42	114	107	106	106
query43	525	494	479	479
query44	1189	832	820	820
query45	190	179	169	169
query46	1141	722	719	719
query47	2040	1895	1926	1895
query48	471	365	392	365
query49	745	381	382	381
query50	820	435	422	422
query51	7306	7367	7135	7135
query52	102	91	90	90
query53	261	188	190	188
query54	574	452	448	448
query55	77	80	76	76
query56	268	253	254	253
query57	1279	1181	1162	1162
query58	211	203	211	203
query59	3204	2863	2843	2843
query60	271	260	244	244
query61	110	106	106	106
query62	758	666	658	658
query63	212	184	193	184
query64	1401	665	635	635
query65	3269	3198	3180	3180
query66	712	314	299	299
query67	15948	15563	15850	15563
query68	4067	570	547	547
query69	423	271	261	261
query70	1178	1125	1124	1124
query71	355	260	259	259
query72	6054	4050	3987	3987
query73	756	339	344	339
query74	10362	8998	9017	8998
query75	3333	2599	2677	2599
query76	2006	1084	1044	1044
query77	533	290	300	290
query78	10513	9582	9619	9582
query79	1578	578	592	578
query80	1194	436	432	432
query81	521	240	240	240
query82	418	86	85	85
query83	165	148	138	138
query84	288	82	77	77
query85	960	295	283	283
query86	361	294	296	294
query87	4492	4273	4326	4273
query88	4160	2355	2338	2338
query89	416	290	288	288
query90	2017	187	185	185
query91	177	149	148	148
query92	59	46	50	46
query93	1782	549	544	544
query94	755	298	297	297
query95	361	280	262	262
query96	612	276	275	275
query97	3335	3145	3215	3145
query98	207	202	203	202
query99	1569	1285	1264	1264
Total cold run time: 312611 ms
Total hot run time: 197152 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c1383a3fb6aa9ee962e87befafe96d3ecd5f4d9, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.10	0.11
query5	0.53	0.50	0.51
query6	1.14	0.73	0.74
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.55	0.50	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.60
query14	2.73	2.87	2.75
query15	0.90	0.83	0.82
query16	0.37	0.38	0.40
query17	1.04	1.03	1.01
query18	0.25	0.23	0.22
query19	1.95	1.82	2.06
query20	0.02	0.01	0.02
query21	15.35	0.57	0.57
query22	2.62	2.35	2.55
query23	16.88	0.88	0.98
query24	3.42	1.34	2.14
query25	0.19	0.08	0.14
query26	0.64	0.14	0.14
query27	0.04	0.05	0.04
query28	8.86	0.53	0.46
query29	12.60	3.28	3.23
query30	0.24	0.06	0.06
query31	2.85	0.39	0.38
query32	3.25	0.46	0.45
query33	2.96	3.04	2.99
query34	17.12	4.42	4.49
query35	4.57	4.50	4.56
query36	0.69	0.49	0.50
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.7 s
Total hot run time: 33.04 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 484625d into branch-3.0 Mar 15, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48782-branch-3.0 branch March 15, 2025 02:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants