Skip to content

branch-3.1: [fix](mtmv) Fix aggregate function roll up fail when function is always nullable #52960 #54802

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 2 commits into from
Aug 19, 2025
Merged

branch-3.1: [fix](mtmv) Fix aggregate function roll up fail when function is always nullable #52960 #54802

morrySnow merged 2 commits into from
Aug 19, 2025

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Aug 14, 2025 •
edited by morrySnow
Loading

picked from #52960

@seawinde
[fix](mtmv) Fix aggregate function roll up fail when function is alwa…
5228a57 
…ys nullable (apache#52960)

### What problem does this PR solve?

Related PR: apache#36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
@seawinde
fix code usage
456383b
@seawinde seawinde requested a review from morrySnow as a code owner August 14, 2025 11:44
@seawinde seawinde changed the title Fix count distinct mv rewrite fail 31 [fix](mtmv) Fix aggregate function roll up fail when function is always nullable #52960 Aug 14, 2025
@seawinde
Copy link
Contributor Author

seawinde commented Aug 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-H: Total hot run time: 32583 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 456383b8b0581af934457d6c430768182d404cd7, data reload: false

------ Round 1 ----------------------------------
q1	17577	5468	5533	5468
q2	2058	288	177	177
q3	10410	1286	729	729
q4	10209	867	464	464
q5	7663	2322	2112	2112
q6	175	162	131	131
q7	885	732	597	597
q8	9327	1422	1124	1124
q9	5486	4939	4944	4939
q10	6773	2272	1837	1837
q11	478	286	264	264
q12	333	351	211	211
q13	17760	3622	2999	2999
q14	241	220	215	215
q15	533	474	460	460
q16	411	434	379	379
q17	593	859	360	360
q18	7137	6380	6400	6380
q19	1197	955	581	581
q20	329	341	221	221
q21	2911	2236	1954	1954
q22	1042	1011	981	981
Total cold run time: 103528 ms
Total hot run time: 32583 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5538	5564	5507	5507
q2	239	340	232	232
q3	2286	2668	2394	2394
q4	1429	1874	1396	1396
q5	4445	4878	5104	4878
q6	176	162	131	131
q7	2145	2018	1816	1816
q8	2675	2813	2747	2747
q9	7248	7153	7182	7153
q10	3054	3349	2779	2779
q11	582	527	488	488
q12	699	800	594	594
q13	3433	3749	3244	3244
q14	290	314	279	279
q15	524	473	471	471
q16	450	506	451	451
q17	1235	1783	1279	1279
q18	7613	7532	7311	7311
q19	810	1177	1037	1037
q20	1985	2082	1894	1894
q21	5510	5089	4765	4765
q22	1125	1086	1033	1033
Total cold run time: 53491 ms
Total hot run time: 51879 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

TPC-DS: Total hot run time: 190209 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 456383b8b0581af934457d6c430768182d404cd7, data reload: false

query1	964	367	374	367
query2	6546	2054	1930	1930
query3	6710	212	229	212
query4	33880	23802	23247	23247
query5	4377	617	454	454
query6	255	184	179	179
query7	4619	493	308	308
query8	292	241	232	232
query9	9665	2625	2628	2625
query10	489	339	264	264
query11	18405	15848	15282	15282
query12	153	109	101	101
query13	1645	529	411	411
query14	10621	6611	7317	6611
query15	218	192	182	182
query16	7990	693	532	532
query17	1754	788	608	608
query18	2097	437	327	327
query19	211	189	172	172
query20	121	115	114	114
query21	217	128	104	104
query22	4421	4602	4362	4362
query23	34949	34132	33210	33210
query24	7654	2633	2603	2603
query25	491	493	424	424
query26	1220	279	171	171
query27	2049	465	341	341
query28	5181	2168	2131	2131
query29	715	582	468	468
query30	258	186	157	157
query31	980	874	813	813
query32	72	57	55	55
query33	529	362	307	307
query34	731	844	505	505
query35	776	818	743	743
query36	993	1058	940	940
query37	102	96	68	68
query38	3946	3900	3812	3812
query39	1505	1453	1412	1412
query40	206	116	109	109
query41	54	55	53	53
query42	115	105	102	102
query43	511	517	486	486
query44	1296	807	802	802
query45	190	178	172	172
query46	856	1043	661	661
query47	1872	1894	1830	1830
query48	407	421	348	348
query49	811	510	424	424
query50	667	677	419	419
query51	7186	7314	7176	7176
query52	105	100	96	96
query53	230	253	191	191
query54	532	538	479	479
query55	80	78	78	78
query56	272	295	275	275
query57	1240	1219	1157	1157
query58	250	213	222	213
query59	3003	3117	2940	2940
query60	294	284	267	267
query61	128	111	112	111
query62	823	725	659	659
query63	217	192	189	189
query64	4142	993	642	642
query65	3309	3225	3208	3208
query66	1058	401	308	308
query67	15846	15725	15472	15472
query68	6978	816	546	546
query69	486	310	262	262
query70	1176	1144	1135	1135
query71	503	295	260	260
query72	5676	3719	3742	3719
query73	640	736	349	349
query74	10203	9387	9205	9205
query75	3195	3178	2685	2685
query76	3181	1185	769	769
query77	513	354	280	280
query78	10383	10479	9794	9794
query79	3162	874	595	595
query80	772	531	447	447
query81	526	259	224	224
query82	1168	117	91	91
query83	163	167	153	153
query84	245	106	79	79
query85	789	367	298	298
query86	410	332	277	277
query87	4375	4311	4266	4266
query88	5318	2425	2389	2389
query89	403	333	295	295
query90	1845	188	187	187
query91	138	157	109	109
query92	64	55	52	52
query93	2771	907	532	532
query94	753	419	288	288
query95	368	273	259	259
query96	491	613	299	299
query97	3256	3292	3197	3197
query98	236	212	205	205
query99	1599	1407	1312	1312
Total cold run time: 295287 ms
Total hot run time: 190209 ms

@doris-robot
Copy link

doris-robot commented Aug 14, 2025

ClickBench: Total hot run time: 28.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 456383b8b0581af934457d6c430768182d404cd7, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.63	0.11	0.11
query5	0.53	0.51	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.56	0.49	0.51
query10	0.54	0.55	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	0.77	0.84	0.82
query15	0.83	0.84	0.81
query16	0.40	0.39	0.37
query17	1.06	1.09	1.07
query18	0.24	0.22	0.23
query19	1.94	1.92	1.91
query20	0.02	0.01	0.02
query21	15.40	0.89	0.58
query22	0.73	0.86	0.54
query23	15.14	1.42	0.53
query24	3.50	0.74	1.47
query25	0.26	0.12	0.10
query26	0.24	0.17	0.13
query27	0.04	0.05	0.05
query28	13.95	0.94	0.42
query29	12.58	3.94	3.31
query30	0.25	0.10	0.07
query31	2.82	0.60	0.39
query32	3.23	0.55	0.46
query33	3.01	3.06	3.03
query34	16.55	5.13	4.53
query35	4.64	4.56	4.52
query36	0.65	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 104.43 s
Total hot run time: 28.76 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Aug 14, 2025

FE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title [fix](mtmv) Fix aggregate function roll up fail when function is always nullable #52960 branch-3.1: [fix](mtmv) Fix aggregate function roll up fail when function is always nullable #52960 Aug 19, 2025
@morrySnow morrySnow merged commit 4b87c30 into apache:branch-3.1 Aug 19, 2025
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@seawinde @doris-robot @hello-stephen @morrySnow