[opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782

seawinde · 2025-03-06T12:03:00Z

What problem does this PR solve?

Opt materialized view rewrite performance when the num of struct infos are huge

Optimize the recursive algorithm to reduce the number of recursive calls. If a group has already been refreshed, skip subsequent refreshes.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

… struct infos are huge

Thearas · 2025-03-06T12:03:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

seawinde · 2025-03-06T12:03:08Z

run buildall

doris-robot · 2025-03-06T12:39:57Z

TPC-H: Total hot run time: 32335 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2f336c78a945a24e2ba37c3860760751e248f58, data reload: false

------ Round 1 ----------------------------------
q1	17625	5151	5046	5046
q2	2045	289	175	175
q3	10424	1249	742	742
q4	10244	1028	531	531
q5	7687	2335	2349	2335
q6	188	167	133	133
q7	869	767	601	601
q8	9301	1301	1032	1032
q9	4869	5014	4629	4629
q10	6831	2314	1890	1890
q11	475	284	275	275
q12	353	351	216	216
q13	17805	3663	3111	3111
q14	232	235	211	211
q15	540	493	481	481
q16	643	614	574	574
q17	593	858	346	346
q18	6935	6326	6365	6326
q19	1352	928	522	522
q20	323	326	191	191
q21	2922	2095	1971	1971
q22	1068	1011	997	997
Total cold run time: 103324 ms
Total hot run time: 32335 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5133	5137	5428	5137
q2	242	328	247	247
q3	2153	2684	2304	2304
q4	1457	1821	1396	1396
q5	4251	4153	4123	4123
q6	217	162	124	124
q7	1886	1913	1776	1776
q8	2613	2695	2591	2591
q9	7266	7163	7230	7163
q10	3023	3230	2786	2786
q11	568	525	490	490
q12	729	786	581	581
q13	3502	3973	3421	3421
q14	314	302	271	271
q15	524	471	492	471
q16	639	687	650	650
q17	1153	1621	1326	1326
q18	7764	7600	7517	7517
q19	824	800	961	800
q20	2005	2005	1877	1877
q21	5364	4996	4720	4720
q22	1125	1061	1052	1052
Total cold run time: 52752 ms
Total hot run time: 50823 ms

doris-robot · 2025-03-06T12:52:11Z

TPC-DS: Total hot run time: 193109 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f2f336c78a945a24e2ba37c3860760751e248f58, data reload: false

query1	1375	1010	1018	1010
query2	6219	1997	1941	1941
query3	11016	4554	4365	4365
query4	54124	25817	23019	23019
query5	5131	529	506	506
query6	360	215	202	202
query7	4914	497	293	293
query8	319	248	249	248
query9	5793	2556	2607	2556
query10	421	306	255	255
query11	15159	15252	15239	15239
query12	160	118	110	110
query13	1063	515	402	402
query14	10871	6811	7113	6811
query15	213	212	175	175
query16	6998	630	499	499
query17	1092	739	594	594
query18	1494	410	329	329
query19	210	201	178	178
query20	129	127	137	127
query21	210	123	110	110
query22	4450	4503	4409	4409
query23	34268	33417	33668	33417
query24	5764	2477	2447	2447
query25	495	543	403	403
query26	723	287	160	160
query27	1846	508	335	335
query28	2851	2440	2490	2440
query29	572	580	429	429
query30	271	225	193	193
query31	883	850	793	793
query32	75	64	65	64
query33	451	358	314	314
query34	778	886	507	507
query35	794	837	751	751
query36	974	993	913	913
query37	129	97	79	79
query38	4343	4319	4336	4319
query39	1497	1415	1436	1415
query40	205	122	106	106
query41	52	51	49	49
query42	127	109	105	105
query43	518	542	507	507
query44	1345	801	818	801
query45	179	213	169	169
query46	854	1025	661	661
query47	1839	1827	1849	1827
query48	403	445	324	324
query49	711	512	441	441
query50	691	731	421	421
query51	4228	4298	4238	4238
query52	105	108	95	95
query53	247	264	185	185
query54	495	506	424	424
query55	86	85	83	83
query56	296	280	260	260
query57	1158	1211	1085	1085
query58	247	259	236	236
query59	2793	2863	2903	2863
query60	286	273	256	256
query61	127	117	119	117
query62	720	742	679	679
query63	222	189	186	186
query64	1745	1064	760	760
query65	4540	4513	4477	4477
query66	738	390	301	301
query67	15670	15384	15301	15301
query68	7349	885	508	508
query69	531	292	317	292
query70	1276	1163	1106	1106
query71	487	305	268	268
query72	5747	3643	3732	3643
query73	1302	731	342	342
query74	9272	9220	8905	8905
query75	3796	3177	2746	2746
query76	4223	1191	785	785
query77	586	355	276	276
query78	10116	10109	9331	9331
query79	2308	830	601	601
query80	645	596	464	464
query81	478	254	229	229
query82	669	133	98	98
query83	171	169	154	154
query84	285	100	77	77
query85	788	365	307	307
query86	400	294	303	294
query87	4470	4650	4309	4309
query88	3340	2243	2223	2223
query89	412	321	284	284
query90	1953	205	208	205
query91	136	140	108	108
query92	74	61	63	61
query93	1438	1162	579	579
query94	681	426	297	297
query95	351	268	266	266
query96	485	563	284	284
query97	3346	3464	3302	3302
query98	254	200	203	200
query99	1429	1430	1251	1251
Total cold run time: 297860 ms
Total hot run time: 193109 ms

doris-robot · 2025-03-06T12:57:30Z

ClickBench: Total hot run time: 30.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f2f336c78a945a24e2ba37c3860760751e248f58, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.25	0.06	0.07
query4	1.60	0.10	0.10
query5	0.55	0.54	0.56
query6	1.21	0.73	0.72
query7	0.03	0.01	0.02
query8	0.04	0.03	0.03
query9	0.60	0.52	0.54
query10	0.57	0.59	0.59
query11	0.16	0.10	0.10
query12	0.15	0.11	0.11
query13	0.63	0.60	0.60
query14	2.66	2.82	2.81
query15	0.93	0.85	0.85
query16	0.38	0.36	0.37
query17	1.05	0.99	1.01
query18	0.21	0.20	0.20
query19	1.90	1.81	1.93
query20	0.01	0.01	0.01
query21	15.37	0.93	0.53
query22	0.76	1.19	0.62
query23	15.00	1.40	0.67
query24	6.77	1.94	0.57
query25	0.52	0.28	0.13
query26	0.61	0.16	0.15
query27	0.06	0.06	0.05
query28	9.31	0.91	0.45
query29	12.60	4.18	3.44
query30	0.24	0.09	0.06
query31	2.80	0.62	0.39
query32	3.23	0.54	0.46
query33	2.97	3.02	3.02
query34	15.68	5.12	4.51
query35	4.48	4.54	4.59
query36	0.69	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.62 s
Total hot run time: 30.87 s

github-actions · 2025-03-10T04:07:56Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-10T04:07:58Z

PR approved by anyone and no changes requested.

zfr9527

LGTM

… struct infos are huge (#48782) ### What problem does this PR solve? Opt materialized view rewrite performance when the num of struct infos are huge Optimize the recursive algorithm to reduce the number of recursive calls. If a group has already been refreshed, skip subsequent refreshes.

…n the num of struct infos are huge #48782 (#48856) Cherry-picked from #48782 Co-authored-by: seawinde <wusi@selectdb.com>

…n the num of struct infos are huge #48782 (#48855) Cherry-picked from #48782 Co-authored-by: seawinde <wusi@selectdb.com>

… struct infos are huge (apache#48782) ### What problem does this PR solve? Opt materialized view rewrite performance when the num of struct infos are huge Optimize the recursive algorithm to reduce the number of recursive calls. If a group has already been refreshed, skip subsequent refreshes.

seawinde added 2 commits March 6, 2025 19:53

[opt](mtmv) Opt materialized view rewrite performance when the num of…

e0aae32

… struct infos are huge

opt code

f2f336c

morrySnow approved these changes Mar 10, 2025

View reviewed changes

morrySnow added p0_b dev/2.1.x dev/3.0.x labels Mar 10, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 10, 2025

github-actions bot added the reviewed label Mar 10, 2025

zfr9527 approved these changes Mar 10, 2025

View reviewed changes

morrySnow merged commit 0c958d2 into apache:master Mar 10, 2025
30 checks passed

github-actions bot mentioned this pull request Mar 10, 2025

branch-3.0: [opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782 #48855

Merged

github-actions bot mentioned this pull request Mar 10, 2025

branch-2.1: [opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782 #48856

Merged

yiguolei pushed a commit that referenced this pull request Mar 11, 2025

branch-2.1: [opt](mtmv) Opt materialized view rewrite performance whe…

bffa084

…n the num of struct infos are huge #48782 (#48856) Cherry-picked from #48782 Co-authored-by: seawinde <wusi@selectdb.com>

yiguolei added the dev/2.1.9-merged label Mar 11, 2025

wm1581066 removed the dev/2.1.x label Mar 11, 2025

dataroaring pushed a commit that referenced this pull request Mar 15, 2025

branch-3.0: [opt](mtmv) Opt materialized view rewrite performance whe…

484625d

…n the num of struct infos are huge #48782 (#48855) Cherry-picked from #48782 Co-authored-by: seawinde <wusi@selectdb.com>

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Mar 15, 2025

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782

[opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782

Uh oh!

seawinde commented Mar 6, 2025

Uh oh!

Thearas commented Mar 6, 2025

Uh oh!

seawinde commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 10, 2025

Uh oh!

github-actions bot commented Mar 10, 2025

Uh oh!

zfr9527 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

[opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782

[opt](mtmv) Opt materialized view rewrite performance when the num of struct infos are huge #48782

Uh oh!

Conversation

seawinde commented Mar 6, 2025

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Mar 6, 2025

Uh oh!

seawinde commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

doris-robot commented Mar 6, 2025

Uh oh!

github-actions bot commented Mar 10, 2025

Uh oh!

github-actions bot commented Mar 10, 2025

Uh oh!

zfr9527 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants