Skip to content

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Jul 8, 2025

What problem does this PR solve?

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

sum(k2) nullable is true
if query is as following, would rewritten fail with err info 'query aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct

Issue Number: close #xxx

Related PR: #36318

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

seawinde commented Jul 8, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33070 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 225eec070b2b62312b68d70b59dd0e4da131923f, data reload: false

------ Round 1 ----------------------------------
q1	17626	5148	5033	5033
q2	1942	297	182	182
q3	10300	1296	706	706
q4	10235	1039	502	502
q5	7544	2298	2341	2298
q6	171	162	132	132
q7	895	731	587	587
q8	9318	1293	1119	1119
q9	6859	5073	5094	5073
q10	6899	2352	1980	1980
q11	494	287	274	274
q12	335	341	215	215
q13	17768	3651	3064	3064
q14	219	216	217	216
q15	552	481	476	476
q16	418	420	387	387
q17	584	862	357	357
q18	7481	7122	7106	7106
q19	1075	941	545	545
q20	335	340	221	221
q21	3741	3147	2306	2306
q22	351	314	291	291
Total cold run time: 105142 ms
Total hot run time: 33070 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5117	5073	5101	5073
q2	245	323	228	228
q3	2152	2677	2315	2315
q4	1400	1751	1327	1327
q5	4197	4119	4677	4119
q6	220	171	129	129
q7	2043	1965	1858	1858
q8	2608	2551	2510	2510
q9	7434	7272	7296	7272
q10	3076	3282	2803	2803
q11	575	534	506	506
q12	688	775	666	666
q13	3532	3977	3356	3356
q14	276	292	256	256
q15	549	506	475	475
q16	439	497	449	449
q17	1264	1622	1364	1364
q18	7882	7636	7554	7554
q19	793	789	1051	789
q20	1969	2055	1977	1977
q21	4974	4637	4528	4528
q22	623	565	550	550
Total cold run time: 52056 ms
Total hot run time: 50104 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185571 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 225eec070b2b62312b68d70b59dd0e4da131923f, data reload: false

query1	989	391	387	387
query2	6558	1775	1808	1775
query3	6747	214	225	214
query4	26422	23719	23594	23594
query5	4373	565	418	418
query6	308	213	212	212
query7	4630	485	287	287
query8	281	224	213	213
query9	8608	2581	2591	2581
query10	477	333	277	277
query11	15280	15040	14738	14738
query12	154	103	100	100
query13	1639	517	400	400
query14	8445	5561	5550	5550
query15	200	183	172	172
query16	7109	609	451	451
query17	929	712	554	554
query18	1966	411	289	289
query19	191	181	155	155
query20	120	112	116	112
query21	204	132	110	110
query22	4098	4205	4524	4205
query23	34889	33790	33998	33790
query24	8403	2329	2348	2329
query25	511	445	384	384
query26	1233	270	146	146
query27	2737	494	333	333
query28	4357	2123	2101	2101
query29	759	553	419	419
query30	277	202	184	184
query31	872	830	755	755
query32	71	61	75	61
query33	552	360	301	301
query34	784	830	505	505
query35	755	800	735	735
query36	936	997	875	875
query37	105	94	70	70
query38	4165	4134	4027	4027
query39	1492	1406	1403	1403
query40	204	118	105	105
query41	53	58	49	49
query42	127	108	109	108
query43	497	499	495	495
query44	1290	820	814	814
query45	172	167	160	160
query46	863	990	613	613
query47	1770	1804	1711	1711
query48	375	422	327	327
query49	739	462	396	396
query50	623	675	392	392
query51	4165	4267	4134	4134
query52	109	103	93	93
query53	217	246	190	190
query54	569	550	501	501
query55	82	80	86	80
query56	323	272	284	272
query57	1187	1188	1109	1109
query58	261	253	258	253
query59	2723	2778	2606	2606
query60	333	303	312	303
query61	129	124	121	121
query62	775	714	661	661
query63	217	186	183	183
query64	4308	1052	756	756
query65	4266	4193	4220	4193
query66	1162	406	323	323
query67	15792	15565	15321	15321
query68	7444	883	531	531
query69	490	312	276	276
query70	1218	1123	1115	1115
query71	426	316	296	296
query72	5813	4898	4868	4868
query73	688	685	352	352
query74	8960	9234	8694	8694
query75	3165	3153	2693	2693
query76	3181	1127	708	708
query77	480	372	286	286
query78	9989	10302	9307	9307
query79	2385	810	566	566
query80	624	498	438	438
query81	462	267	224	224
query82	429	125	96	96
query83	288	252	223	223
query84	290	94	90	90
query85	797	345	309	309
query86	337	293	284	284
query87	4459	4461	4272	4272
query88	3554	2312	2266	2266
query89	376	323	284	284
query90	1929	200	201	200
query91	137	152	108	108
query92	72	57	53	53
query93	1742	936	582	582
query94	717	415	305	305
query95	381	292	281	281
query96	483	568	272	272
query97	2691	2782	2626	2626
query98	233	213	213	213
query99	1415	1436	1272	1272
Total cold run time: 271612 ms
Total hot run time: 185571 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 225eec070b2b62312b68d70b59dd0e4da131923f, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.04	0.04
query3	0.24	0.08	0.08
query4	1.62	0.11	0.11
query5	0.43	0.42	0.43
query6	1.17	0.67	0.64
query7	0.03	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.52	0.51
query10	0.58	0.57	0.57
query11	0.17	0.11	0.11
query12	0.14	0.11	0.12
query13	0.63	0.62	0.62
query14	0.80	0.83	0.80
query15	0.90	0.88	0.87
query16	0.40	0.38	0.39
query17	1.10	1.09	1.05
query18	0.22	0.22	0.21
query19	1.91	1.79	1.88
query20	0.01	0.01	0.01
query21	15.42	0.92	0.56
query22	0.77	1.21	0.70
query23	14.85	1.36	0.64
query24	6.71	2.49	0.56
query25	0.51	0.22	0.19
query26	0.67	0.16	0.14
query27	0.06	0.06	0.05
query28	9.40	0.87	0.44
query29	12.56	3.95	3.33
query30	0.26	0.09	0.07
query31	2.83	0.61	0.39
query32	3.26	0.58	0.48
query33	3.18	3.05	3.11
query34	16.14	5.46	4.77
query35	4.81	4.85	4.81
query36	0.70	0.49	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.15	0.14
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.04
Total cold run time: 103.72 s
Total hot run time: 29.39 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉
Increment coverage report
Complete coverage report

@seawinde seawinde force-pushed the fix_count_distinct_mv_rewrite_fail branch from 225eec0 to 62ce865 Compare August 6, 2025 02:30
@seawinde
Copy link
Contributor Author

seawinde commented Aug 6, 2025

run buildall

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 13, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 0a5cd95 into apache:master Aug 13, 2025
30 of 31 checks passed
github-actions bot pushed a commit that referenced this pull request Aug 13, 2025
…ys nullable (#52960)

### What problem does this PR solve?

Related PR: #36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
github-actions bot pushed a commit that referenced this pull request Aug 13, 2025
…ys nullable (#52960)

### What problem does this PR solve?

Related PR: #36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
github-actions bot pushed a commit that referenced this pull request Aug 13, 2025
…ys nullable (#52960)

### What problem does this PR solve?

Related PR: #36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
seawinde added a commit to seawinde/doris that referenced this pull request Aug 14, 2025
…ys nullable (apache#52960)

### What problem does this PR solve?

Related PR: apache#36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
seawinde added a commit to seawinde/doris that referenced this pull request Aug 14, 2025
…ys nullable (apache#52960)

### What problem does this PR solve?

Related PR: apache#36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
seawinde added a commit to seawinde/doris that referenced this pull request Aug 14, 2025
…ys nullable (apache#52960)

### What problem does this PR solve?

Related PR: apache#36318

Problem Summary:

materaialized view def is as fllowing:

create materialized view as
select k1, k3, sum(k2), count(k4) from ${tblName} group by k1, k3;

`sum(k2) ` nullable is true
if query is as following, would rewritten fail with err info 'query
aggregate function roll up fail', the pr fix this

select sum(distinct k1) from agg_use_key_direct
morrySnow pushed a commit that referenced this pull request Aug 19, 2025
yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Dec 30, 2025
…ion is always nullable apache#52960 (apache#5650)

## Proposed changes

pr: apache#52960
commitId:
selectdb/selectdb-core@0a5cd95

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x dev/3.1.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants