Skip to content

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented May 29, 2024

pick from master #35504

if window expression is not required by its parent, we should prune this column.
If all window expressions of window operator are pruned, we remove this window operator directly.

if window expression is not required by its parent,
we should prune this column.
If all window expressions of window operator are pruned,
we remove this window operator directly.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

1 similar comment
@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49453 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d117991f0cae68a46f886806c63ec84e886e3b7, data reload: false

------ Round 1 ----------------------------------
q1	17697	4371	4322	4322
q2	2061	182	146	146
q3	10394	1901	1929	1901
q4	10322	1221	1324	1221
q5	8576	3820	3869	3820
q6	236	124	124	124
q7	2071	1583	1585	1583
q8	9278	2718	2687	2687
q9	10633	10163	10089	10089
q10	8626	3516	3530	3516
q11	427	254	251	251
q12	463	303	300	300
q13	18368	4023	3978	3978
q14	355	323	326	323
q15	512	468	465	465
q16	684	570	569	569
q17	1119	958	985	958
q18	7351	6971	6905	6905
q19	1724	1550	1505	1505
q20	520	323	314	314
q21	4376	4120	4081	4081
q22	482	395	398	395
Total cold run time: 116275 ms
Total hot run time: 49453 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4314	4288	4334	4288
q2	325	219	226	219
q3	4187	4122	4157	4122
q4	2741	2725	2739	2725
q5	7082	7006	7024	7006
q6	237	121	122	121
q7	3189	2854	2775	2775
q8	4338	4498	4472	4472
q9	16857	16752	16756	16752
q10	4199	4238	4272	4238
q11	754	698	730	698
q12	1017	843	874	843
q13	6610	3727	3729	3727
q14	453	437	431	431
q15	498	457	452	452
q16	738	677	671	671
q17	3813	3866	3814	3814
q18	8733	8658	8781	8658
q19	1719	1714	1644	1644
q20	2358	2143	2119	2119
q21	8478	8462	8478	8462
q22	1053	916	939	916
Total cold run time: 83693 ms
Total hot run time: 79153 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 205845 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d117991f0cae68a46f886806c63ec84e886e3b7, data reload: false

query1	943	396	375	375
query2	6590	2932	2748	2748
query3	6927	213	212	212
query4	20884	17974	17820	17820
query5	19742	6510	6464	6464
query6	295	222	230	222
query7	4156	297	303	297
query8	239	248	239	239
query9	3080	2665	2572	2572
query10	414	303	300	300
query11	11399	10686	10744	10686
query12	116	73	81	73
query13	5609	707	684	684
query14	18258	13294	13235	13235
query15	374	218	221	218
query16	6455	295	263	263
query17	1698	1436	878	878
query18	2324	410	408	408
query19	216	153	149	149
query20	80	80	81	80
query21	186	99	94	94
query22	5274	5167	5110	5110
query23	32491	31998	31915	31915
query24	6931	6436	6504	6436
query25	509	423	445	423
query26	499	165	159	159
query27	1782	296	289	289
query28	6097	2379	2318	2318
query29	2847	2837	2772	2772
query30	239	163	166	163
query31	919	769	751	751
query32	70	64	56	56
query33	410	274	251	251
query34	856	471	488	471
query35	1123	921	908	908
query36	1459	1165	1218	1165
query37	90	62	61	61
query38	3094	2940	2950	2940
query39	1373	1319	1313	1313
query40	206	90	89	89
query41	42	38	37	37
query42	93	79	79	79
query43	709	759	739	739
query44	1158	716	708	708
query45	241	228	225	225
query46	1231	964	973	964
query47	1743	1681	1760	1681
query48	1021	713	692	692
query49	619	363	394	363
query50	845	632	598	598
query51	4774	4643	4715	4643
query52	103	81	91	81
query53	442	315	310	310
query54	2660	2450	2432	2432
query55	100	86	90	86
query56	225	211	207	207
query57	1130	1020	1083	1020
query58	215	200	204	200
query59	4179	3723	4027	3723
query60	218	195	192	192
query61	98	95	97	95
query62	836	495	526	495
query63	479	337	333	333
query64	2537	1539	1514	1514
query65	3611	3568	3577	3568
query66	766	372	379	372
query67	17100	19379	16961	16961
query68	8432	660	658	658
query69	567	357	335	335
query70	1721	1395	1266	1266
query71	385	314	307	307
query72	6502	3500	3510	3500
query73	730	316	309	309
query74	6299	5871	5903	5871
query75	4985	3804	3763	3763
query76	5047	1125	1168	1125
query77	731	254	266	254
query78	12684	12812	13234	12812
query79	13103	650	633	633
query80	905	401	403	401
query81	499	234	230	230
query82	761	100	98	98
query83	185	134	138	134
query84	263	69	70	69
query85	722	328	326	326
query86	344	328	337	328
query87	3226	2987	3014	2987
query88	5980	2344	2329	2329
query89	395	282	296	282
query90	1899	219	207	207
query91	185	142	149	142
query92	61	52	54	52
query93	4857	572	560	560
query94	780	215	208	208
query95	1110	1053	1041	1041
query96	646	331	327	327
query97	6443	6354	6417	6354
query98	199	184	171	171
query99	2587	904	841	841
Total cold run time: 316662 ms
Total hot run time: 205845 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d117991f0cae68a46f886806c63ec84e886e3b7, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.80	0.08	0.07
query5	0.54	0.52	0.51
query6	1.27	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.48	0.49
query10	0.54	0.54	0.55
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.62
query14	0.80	0.76	0.78
query15	0.79	0.76	0.74
query16	0.38	0.39	0.37
query17	1.01	1.01	1.04
query18	0.23	0.24	0.24
query19	1.94	1.76	1.81
query20	0.01	0.00	0.00
query21	15.48	0.56	0.56
query22	1.99	2.02	1.48
query23	17.19	0.98	0.93
query24	4.76	0.92	1.68
query25	0.37	0.10	0.05
query26	0.56	0.17	0.16
query27	0.04	0.04	0.03
query28	7.93	0.78	0.70
query29	12.68	2.32	2.30
query30	0.60	0.52	0.52
query31	2.82	0.39	0.37
query32	3.37	0.49	0.49
query33	3.02	3.10	3.08
query34	15.28	4.80	4.78
query35	4.84	4.83	4.83
query36	1.06	1.01	1.00
query37	0.05	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.13
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.02	0.01
Total cold run time: 103.45 s
Total hot run time: 30.43 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 5d117991f0cae68a46f886806c63ec84e886e3b7 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@morrySnow morrySnow merged commit 1647702 into apache:branch-2.0 May 29, 2024
@morrySnow morrySnow deleted the 2.0_25504 branch May 29, 2024 05:59
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…or (apache#35577)

pick from master apache#35504

if window expression is not required by its parent, we should prune this
column.
If all window expressions of window operator are pruned, we remove this
window operator directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants