[fix](Nereids) remove restrict for count(*) in window #35220

morrySnow · 2024-05-22T10:21:44Z

support count(*) used for window function

CREATE TABLE `t1` (
  `id` INT NULL,
  `dt` TEXT NULL
)
DISTRIBUTED BY HASH(`id`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1" );

select *, count(*) over() from t1;

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-05-22T10:21:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

morrySnow · 2024-05-22T10:22:10Z

run buildall

doris-robot · 2024-05-22T10:48:20Z

TPC-DS: Total hot run time: 171594 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eda1c1d15d12e9b4259aea6825a3b8bd958e1b97, data reload: false

query1	895	374	376	374
query2	6453	2392	2319	2319
query3	6653	208	204	204
query4	19015	17248	17206	17206
query5	4153	417	422	417
query6	245	157	175	157
query7	4590	301	297	297
query8	240	192	187	187
query9	8547	2415	2401	2401
query10	467	294	282	282
query11	10711	10081	10040	10040
query12	140	91	87	87
query13	1643	369	367	367
query14	9336	7706	6904	6904
query15	211	164	166	164
query16	7749	263	250	250
query17	1859	517	523	517
query18	1886	275	278	275
query19	203	170	157	157
query20	97	87	87	87
query21	193	132	124	124
query22	4217	3862	4083	3862
query23	33585	33043	32912	32912
query24	10977	2868	2901	2868
query25	651	357	355	355
query26	1501	160	158	158
query27	2995	320	329	320
query28	7518	2057	2049	2049
query29	982	622	596	596
query30	309	168	180	168
query31	1000	791	726	726
query32	101	54	55	54
query33	767	270	262	262
query34	1014	480	469	469
query35	770	600	595	595
query36	1092	913	930	913
query37	139	72	69	69
query38	2880	2775	2697	2697
query39	831	793	833	793
query40	277	129	124	124
query41	49	43	44	43
query42	103	96	101	96
query43	551	525	550	525
query44	1182	723	754	723
query45	179	164	160	160
query46	1074	717	718	717
query47	1838	1739	1791	1739
query48	379	296	294	294
query49	1117	386	379	379
query50	778	414	402	402
query51	6918	6714	6807	6714
query52	98	96	88	88
query53	353	295	293	293
query54	943	428	418	418
query55	75	74	72	72
query56	254	270	238	238
query57	1107	1077	1036	1036
query58	248	209	207	207
query59	3308	3165	3079	3079
query60	276	259	249	249
query61	88	86	90	86
query62	649	453	453	453
query63	312	287	293	287
query64	9879	2243	1751	1751
query65	3172	3101	3102	3101
query66	1397	332	318	318
query67	15324	14936	14820	14820
query68	4601	544	548	544
query69	452	268	274	268
query70	1152	1056	1085	1056
query71	424	265	274	265
query72	7266	5656	5163	5163
query73	725	322	329	322
query74	6068	5601	5628	5601
query75	3491	2620	2632	2620
query76	3046	1010	1036	1010
query77	618	269	276	269
query78	10551	9868	9708	9708
query79	2134	532	525	525
query80	931	471	498	471
query81	528	245	242	242
query82	639	97	92	92
query83	267	169	168	168
query84	241	86	85	85
query85	1724	265	281	265
query86	511	327	313	313
query87	3295	3126	3134	3126
query88	4042	2307	2317	2307
query89	493	400	387	387
query90	1995	190	188	188
query91	123	94	99	94
query92	59	50	49	49
query93	2303	521	496	496
query94	1237	251	195	195
query95	413	315	318	315
query96	585	268	272	268
query97	3161	3025	3038	3025
query98	234	222	217	217
query99	1115	861	846	846
Total cold run time: 272967 ms
Total hot run time: 171594 ms

doris-robot · 2024-05-22T10:53:51Z

ClickBench: Total hot run time: 30.14 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit eda1c1d15d12e9b4259aea6825a3b8bd958e1b97, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.47	0.49	0.51
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.53
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.60	0.60
query14	0.79	0.77	0.79
query15	0.82	0.82	0.81
query16	0.37	0.37	0.38
query17	0.99	0.99	0.93
query18	0.22	0.24	0.24
query19	1.90	1.70	1.74
query20	0.01	0.01	0.01
query21	15.46	0.70	0.67
query22	4.70	7.58	1.53
query23	18.31	1.47	1.34
query24	1.39	0.41	0.22
query25	0.16	0.07	0.08
query26	0.28	0.16	0.17
query27	0.09	0.07	0.09
query28	13.34	1.02	0.99
query29	12.73	3.28	3.26
query30	0.25	0.06	0.06
query31	2.86	0.39	0.38
query32	3.29	0.47	0.47
query33	2.94	2.86	2.93
query34	17.04	4.43	4.47
query35	4.53	4.48	4.51
query36	0.63	0.46	0.45
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.13	0.15
query41	0.09	0.04	0.05
query42	0.05	0.06	0.05
query43	0.03	0.04	0.04
Total cold run time: 109.5 s
Total hot run time: 30.14 s

support count(*) used for window function CREATE TABLE `t1` ( `id` INT NULL, `dt` TEXT NULL ) DISTRIBUTED BY HASH(`id`) BUCKETS 10 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); select *, count(*) over() from t1;

morrySnow · 2024-05-22T12:25:54Z

run buildall

doris-robot · 2024-05-22T12:59:24Z

TPC-DS: Total hot run time: 170071 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5abc96bfb3142a72849aae3d1c4c7674da1763e2, data reload: false

query1	923	376	364	364
query2	6457	2536	2382	2382
query3	6645	206	207	206
query4	19506	17445	17376	17376
query5	4177	422	436	422
query6	247	157	179	157
query7	4584	297	290	290
query8	241	184	195	184
query9	8491	2379	2365	2365
query10	461	286	272	272
query11	10510	10071	10142	10071
query12	147	92	91	91
query13	1653	367	367	367
query14	10167	7414	7556	7414
query15	232	174	171	171
query16	7775	272	266	266
query17	1837	541	520	520
query18	1814	287	284	284
query19	200	161	165	161
query20	100	87	88	87
query21	195	130	133	130
query22	4341	3908	3853	3853
query23	33734	33143	33119	33119
query24	11156	2935	2854	2854
query25	659	375	443	375
query26	1416	153	151	151
query27	2930	318	308	308
query28	7223	2033	2018	2018
query29	956	600	600	600
query30	306	173	171	171
query31	955	750	747	747
query32	88	55	54	54
query33	785	271	268	268
query34	989	480	477	477
query35	716	611	600	600
query36	1064	887	873	873
query37	130	68	73	68
query38	2901	2795	2743	2743
query39	824	769	801	769
query40	234	128	124	124
query41	46	44	44	44
query42	104	95	94	94
query43	553	541	549	541
query44	1234	723	746	723
query45	183	163	164	163
query46	1073	702	739	702
query47	1854	1770	1763	1763
query48	375	302	292	292
query49	1083	367	374	367
query50	763	405	388	388
query51	6864	6789	6744	6744
query52	105	87	95	87
query53	348	286	282	282
query54	965	431	416	416
query55	74	71	72	71
query56	270	248	241	241
query57	1142	1033	1022	1022
query58	244	211	210	210
query59	3364	3125	3252	3125
query60	273	255	251	251
query61	87	86	87	86
query62	685	451	447	447
query63	302	284	281	281
query64	9946	2227	1744	1744
query65	3174	3104	3126	3104
query66	1384	333	332	332
query67	15435	14933	14890	14890
query68	6203	533	538	533
query69	495	268	265	265
query70	1153	1140	1111	1111
query71	485	275	278	275
query72	7989	2725	2528	2528
query73	799	321	322	321
query74	6157	5697	5659	5659
query75	4029	2643	2632	2632
query76	4325	985	992	985
query77	662	268	267	267
query78	10413	10032	9751	9751
query79	2756	509	508	508
query80	1343	443	444	443
query81	526	250	243	243
query82	1423	105	97	97
query83	208	172	170	170
query84	265	86	87	86
query85	1440	269	262	262
query86	476	330	296	296
query87	3310	3118	3114	3114
query88	3955	2330	2310	2310
query89	480	383	384	383
query90	1969	191	188	188
query91	124	98	98	98
query92	64	48	48	48
query93	2972	521	500	500
query94	1149	192	186	186
query95	389	305	311	305
query96	597	266	270	266
query97	3206	3053	3007	3007
query98	230	224	225	224
query99	1230	886	848	848
Total cold run time: 280521 ms
Total hot run time: 170071 ms

doris-robot · 2024-05-22T13:05:00Z

ClickBench: Total hot run time: 31.3 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5abc96bfb3142a72849aae3d1c4c7674da1763e2, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.11	0.08
query5	0.49	0.50	0.51
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.53	0.49	0.48
query10	0.54	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.13
query13	0.61	0.59	0.60
query14	0.75	0.79	0.78
query15	0.84	0.82	0.82
query16	0.36	0.35	0.37
query17	0.98	1.01	1.00
query18	0.22	0.25	0.27
query19	1.86	1.71	1.76
query20	0.02	0.01	0.01
query21	15.52	0.70	0.69
query22	4.00	7.09	2.55
query23	18.26	1.39	1.23
query24	1.80	0.20	0.26
query25	0.15	0.09	0.08
query26	0.25	0.16	0.16
query27	0.08	0.07	0.08
query28	13.32	1.04	1.00
query29	12.68	3.28	3.29
query30	0.25	0.06	0.06
query31	2.86	0.40	0.39
query32	3.26	0.48	0.47
query33	2.91	2.90	2.91
query34	16.95	4.43	4.42
query35	4.52	4.50	4.51
query36	0.66	0.46	0.46
query37	0.17	0.15	0.16
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.9 s
Total hot run time: 31.3 s

github-actions · 2024-05-23T02:41:55Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-05-23T02:41:57Z

PR approved by anyone and no changes requested.

support count(*) used for window function CREATE TABLE `t1` ( `id` INT NULL, `dt` TEXT NULL ) DISTRIBUTED BY HASH(`id`) BUCKETS 10 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); select *, count(*) over() from t1;

pick from master apache#35220 support count(*) used for window function CREATE TABLE `t1` ( `id` INT NULL, `dt` TEXT NULL ) DISTRIBUTED BY HASH(`id`) BUCKETS 10 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); select *, count(*) over() from t1;

pick from master #35220 support count(*) used for window function CREATE TABLE `t1` ( `id` INT NULL, `dt` TEXT NULL ) DISTRIBUTED BY HASH(`id`) BUCKETS 10 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); select *, count(*) over() from t1;

…pache#35452) pick from master apache#35220 support count(*) used for window function CREATE TABLE `t1` ( `id` INT NULL, `dt` TEXT NULL ) DISTRIBUTED BY HASH(`id`) BUCKETS 10 PROPERTIES ( "replication_allocation" = "tag.location.default: 1" ); select *, count(*) over() from t1;

morrySnow added dev/2.0.x dev/2.1.x labels May 22, 2024

morrySnow force-pushed the support_count_star_window branch from eda1c1d to 5abc96b Compare May 22, 2024 12:23

starocean999 approved these changes May 23, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels May 23, 2024

924060929 approved these changes May 23, 2024

View reviewed changes

morrySnow merged commit d8c39a9 into apache:master May 23, 2024

morrySnow deleted the support_count_star_window branch May 23, 2024 02:46

morrySnow added dev/2.1.4-merged and removed dev/2.1.x labels May 27, 2024

morrySnow mentioned this pull request May 27, 2024

[opt](Nereids) remove restrict for count(*) in window (#35220) #35452

Merged

morrySnow added dev/2.0.11-merged and removed dev/2.0.x labels May 27, 2024

morningman mentioned this pull request Jun 22, 2024

Release Notes 2.1.4 #35752

Closed

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](Nereids) remove restrict for count(*) in window #35220

[fix](Nereids) remove restrict for count(*) in window #35220

Uh oh!

morrySnow commented May 22, 2024 •

edited

Loading

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

morrySnow commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

morrySnow commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

github-actions bot commented May 23, 2024

Uh oh!

github-actions bot commented May 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[fix](Nereids) remove restrict for count(*) in window #35220

[fix](Nereids) remove restrict for count(*) in window #35220

Uh oh!

Conversation

morrySnow commented May 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

morrySnow commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

morrySnow commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

doris-robot commented May 22, 2024

Uh oh!

github-actions bot commented May 23, 2024

Uh oh!

github-actions bot commented May 23, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

morrySnow commented May 22, 2024 •

edited

Loading