Skip to content

branch-3.0: [fix](nereids) fix distinct window compute wrong result (#48987) #49008

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
924060929 wants to merge 5 commits into from
Closed

branch-3.0: [fix](nereids) fix distinct window compute wrong result (#48987) #49008

924060929 wants to merge 5 commits into from

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Mar 13, 2025

cherry pick from #48987

@924060929
cherry pick from apache#48987
1390015 
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
  select 100 value, 1 id
  union all
  select 100, 2
)a;

+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100                              |
| 100                              |
+----------------------------------+
```
@924060929 924060929 requested a review from dataroaring as a code owner March 13, 2025 03:48
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

924060929 commented Mar 13, 2025

run buildall

@924060929 924060929 changed the title [fix](nereids) fix distinct window compute wrong result (#48987) branch-3.0: [fix](nereids) fix distinct window compute wrong result (#48987) Mar 13, 2025
@924060929
cherry pick from apache#48987
f670357 
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
  select 100 value, 1 id
  union all
  select 100, 2
)a;

+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100                              |
| 100                              |
+----------------------------------+
```
@924060929
cherry pick from apache#48987
ee5ff07 
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
  select 100 value, 1 id
  union all
  select 100, 2
)a;

+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100                              |
| 100                              |
+----------------------------------+
```
@924060929
Copy link
Contributor Author

924060929 commented Mar 13, 2025

run buildall

@924060929
cherry pick from apache#48987
f6832ee 
fix distinct window compute wrong result, introduced by apache#14397
```sql
select distinct sum(value) over(partition by id)
from (
  select 100 value, 1 id
  union all
  select 100, 2
)a;

+----------------------------------+
| sum(value) over(partition by id) |
+----------------------------------+
| 100                              |
| 100                              |
+----------------------------------+
```
@924060929
Copy link
Contributor Author

924060929 commented Mar 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 14, 2025

TPC-H: Total hot run time: 40261 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f6832ee2f5d3efe55b5ba2b0c091c07883e37e3f, data reload: false

------ Round 1 ----------------------------------
q1	17573	6755	6582	6582
q2	2077	172	187	172
q3	10721	1097	1212	1097
q4	10205	796	694	694
q5	7721	2870	2899	2870
q6	216	138	134	134
q7	986	634	623	623
q8	9363	1963	2053	1963
q9	6604	6384	6390	6384
q10	7041	2237	2300	2237
q11	472	271	265	265
q12	403	217	212	212
q13	17777	2983	3039	2983
q14	244	205	209	205
q15	508	462	459	459
q16	668	594	588	588
q17	986	574	611	574
q18	7246	6714	6806	6714
q19	1397	1068	1081	1068
q20	477	205	198	198
q21	4156	3428	3232	3232
q22	1125	1007	1053	1007
Total cold run time: 107966 ms
Total hot run time: 40261 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6714	6683	6634	6634
q2	333	238	226	226
q3	2929	2814	2969	2814
q4	2097	1808	1806	1806
q5	5837	5809	5801	5801
q6	212	131	128	128
q7	2211	1847	1825	1825
q8	3405	3578	3554	3554
q9	9032	9189	9220	9189
q10	3651	3609	3608	3608
q11	593	508	517	508
q12	882	669	667	667
q13	11945	3263	3290	3263
q14	317	259	267	259
q15	522	483	454	454
q16	687	658	649	649
q17	1842	1638	1620	1620
q18	8230	7743	7866	7743
q19	1707	1658	1572	1572
q20	2055	1824	1806	1806
q21	5646	5492	5324	5324
q22	1138	1058	1057	1057
Total cold run time: 71985 ms
Total hot run time: 60507 ms

@doris-robot
Copy link

doris-robot commented Mar 14, 2025

TPC-DS: Total hot run time: 198169 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f6832ee2f5d3efe55b5ba2b0c091c07883e37e3f, data reload: false

query1	1287	940	900	900
query2	6362	2035	2025	2025
query3	10984	4353	4488	4353
query4	61325	29424	23729	23729
query5	5214	472	446	446
query6	402	170	193	170
query7	5488	324	319	319
query8	319	224	231	224
query9	8540	2640	2646	2640
query10	460	263	258	258
query11	18020	15231	15856	15231
query12	163	105	101	101
query13	1455	458	455	455
query14	11023	7251	7629	7251
query15	199	178	185	178
query16	7150	515	498	498
query17	1135	582	590	582
query18	1831	333	326	326
query19	206	159	158	158
query20	116	108	110	108
query21	209	103	105	103
query22	4781	4357	4801	4357
query23	34447	34256	34711	34256
query24	6192	2945	2983	2945
query25	550	430	446	430
query26	674	178	175	175
query27	1961	359	371	359
query28	4244	2508	2454	2454
query29	735	491	429	429
query30	261	164	162	162
query31	1011	821	851	821
query32	68	52	57	52
query33	391	293	294	293
query34	938	524	537	524
query35	848	733	721	721
query36	1099	963	958	958
query37	113	62	70	62
query38	4126	4059	4057	4057
query39	1540	1456	1470	1456
query40	204	106	101	101
query41	49	49	48	48
query42	119	104	102	102
query43	539	514	495	495
query44	1215	846	841	841
query45	188	171	169	169
query46	1151	750	723	723
query47	2006	1877	1947	1877
query48	483	385	387	385
query49	732	397	397	397
query50	858	448	448	448
query51	7250	7209	7171	7171
query52	102	90	89	89
query53	267	183	194	183
query54	591	461	463	461
query55	78	83	84	83
query56	276	252	259	252
query57	1277	1183	1119	1119
query58	241	209	206	206
query59	3370	2923	2941	2923
query60	274	243	262	243
query61	108	119	110	110
query62	749	667	667	667
query63	217	187	193	187
query64	1408	670	632	632
query65	3277	3196	3234	3196
query66	651	285	300	285
query67	16146	15647	15920	15647
query68	4243	603	583	583
query69	423	269	276	269
query70	1184	1174	1095	1095
query71	339	261	266	261
query72	6351	4094	4021	4021
query73	779	353	359	353
query74	10290	9054	8938	8938
query75	3389	2666	2674	2666
query76	2091	981	1086	981
query77	476	280	287	280
query78	10631	9589	9690	9589
query79	1454	605	608	605
query80	876	439	449	439
query81	526	241	235	235
query82	1296	85	87	85
query83	238	142	151	142
query84	287	80	86	80
query85	871	300	287	287
query86	322	289	302	289
query87	4568	4282	4273	4273
query88	3881	2418	2389	2389
query89	423	289	296	289
query90	1948	187	187	187
query91	184	147	150	147
query92	66	49	49	49
query93	1933	561	556	556
query94	696	300	297	297
query95	361	269	257	257
query96	615	286	288	286
query97	3338	3193	3143	3143
query98	211	203	200	200
query99	1549	1292	1296	1292
Total cold run time: 315532 ms
Total hot run time: 198169 ms

@doris-robot
Copy link

doris-robot commented Mar 14, 2025

ClickBench: Total hot run time: 31.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f6832ee2f5d3efe55b5ba2b0c091c07883e37e3f, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.10
query5	0.52	0.51	0.51
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.05	0.03
query9	0.56	0.50	0.51
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	2.75	2.87	2.88
query15	0.91	0.83	0.82
query16	0.39	0.38	0.41
query17	1.06	1.05	1.04
query18	0.24	0.22	0.22
query19	1.97	1.77	1.99
query20	0.01	0.01	0.01
query21	15.35	0.60	0.55
query22	2.60	1.99	2.10
query23	16.90	0.89	0.73
query24	3.21	0.69	0.67
query25	0.20	0.06	0.06
query26	0.49	0.14	0.14
query27	0.04	0.05	0.04
query28	11.05	0.52	0.45
query29	12.58	3.18	3.22
query30	0.25	0.06	0.07
query31	2.86	0.39	0.37
query32	3.25	0.45	0.46
query33	3.04	3.00	3.04
query34	17.27	4.49	4.57
query35	4.55	4.59	4.53
query36	0.66	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 107.77 s
Total hot run time: 31.97 s

@924060929 924060929 marked this pull request as draft March 14, 2025 07:05
@924060929
Copy link
Contributor Author

924060929 commented Mar 24, 2025

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring Awaiting requested review from dataroaring

@lide-reed lide-reed Awaiting requested review from lide-reed lide-reed will be requested when the pull request is marked ready for review lide-reed is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@924060929 @hello-stephen @doris-robot @dataroaring