Skip to content

[fix](nereids)should not push down constant output expr in window expression #35961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 6, 2024
Merged

[fix](nereids)should not push down constant output expr in window expression #35961

morrySnow merged 1 commit into from
Jun 6, 2024

Conversation

@starocean999
Copy link
Contributor

@starocean999 starocean999 commented Jun 6, 2024

pick from master #35948

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Jun 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@starocean999
Copy link
Contributor Author

starocean999 commented Jun 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 6, 2024

TPC-H: Total hot run time: 49449 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 928ac23316aefd46735e58cb5ecc45aa4b9f7fe5, data reload: false

------ Round 1 ----------------------------------
q1	17836	4357	4321	4321
q2	2095	155	150	150
q3	10293	1925	1925	1925
q4	10348	1259	1306	1259
q5	8435	3886	3897	3886
q6	254	125	124	124
q7	2018	1576	1614	1576
q8	9278	2696	2692	2692
q9	10695	10087	10165	10087
q10	8647	3483	3534	3483
q11	417	244	257	244
q12	466	308	306	306
q13	18327	3953	4020	3953
q14	350	333	321	321
q15	503	464	461	461
q16	682	573	570	570
q17	1121	965	986	965
q18	7224	6984	6776	6776
q19	1721	1569	1562	1562
q20	534	297	315	297
q21	4421	4090	4089	4089
q22	487	402	410	402
Total cold run time: 116152 ms
Total hot run time: 49449 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4301	4261	4291	4261
q2	317	225	224	224
q3	4205	4203	4176	4176
q4	2735	2755	2753	2753
q5	7168	7078	7052	7052
q6	234	122	120	120
q7	3217	2863	2836	2836
q8	4377	4435	4478	4435
q9	16850	16889	16785	16785
q10	4277	4284	4277	4277
q11	747	703	683	683
q12	1061	867	854	854
q13	7091	3775	3795	3775
q14	452	422	421	421
q15	533	476	457	457
q16	732	693	679	679
q17	3832	3912	3740	3740
q18	8839	8834	8958	8834
q19	1715	1737	1696	1696
q20	2538	2209	2224	2209
q21	8779	8809	8898	8809
q22	1098	1019	947	947
Total cold run time: 85098 ms
Total hot run time: 80023 ms

@doris-robot
Copy link

doris-robot commented Jun 6, 2024

TPC-DS: Total hot run time: 203692 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 928ac23316aefd46735e58cb5ecc45aa4b9f7fe5, data reload: false

query1	921	419	384	384
query2	6526	2776	2628	2628
query3	6913	217	210	210
query4	20951	18054	18147	18054
query5	19749	6529	6491	6491
query6	430	226	231	226
query7	5098	296	309	296
query8	253	264	242	242
query9	3099	2679	2616	2616
query10	412	297	295	295
query11	11429	10665	10703	10665
query12	126	77	76	76
query13	5602	693	673	673
query14	19042	13368	13792	13368
query15	377	228	233	228
query16	6436	290	270	270
query17	1387	1475	885	885
query18	2282	406	407	406
query19	205	149	147	147
query20	84	79	81	79
query21	191	95	91	91
query22	5225	5112	4971	4971
query23	32456	31898	32077	31898
query24	6849	6457	6504	6457
query25	527	435	409	409
query26	502	162	158	158
query27	1694	298	297	297
query28	6117	2368	2346	2346
query29	2889	2792	2774	2774
query30	241	166	168	166
query31	939	733	745	733
query32	76	62	60	60
query33	411	269	268	268
query34	842	468	468	468
query35	1131	908	929	908
query36	1240	1125	1230	1125
query37	87	62	61	61
query38	3057	2920	2967	2920
query39	1385	1319	1318	1318
query40	206	94	97	94
query41	40	39	37	37
query42	81	86	86	86
query43	727	686	665	665
query44	1134	705	712	705
query45	246	235	238	235
query46	1220	941	963	941
query47	1862	1816	1753	1753
query48	1000	705	707	705
query49	627	376	369	369
query50	862	595	633	595
query51	4805	4614	4651	4614
query52	90	80	72	72
query53	458	321	326	321
query54	2698	2492	2463	2463
query55	103	80	86	80
query56	247	195	219	195
query57	1215	1087	1086	1086
query58	217	203	206	203
query59	4180	4300	4270	4270
query60	210	202	202	202
query61	99	96	98	96
query62	826	484	483	483
query63	507	341	341	341
query64	2347	1533	1495	1495
query65	3632	3584	3580	3580
query66	769	378	384	378
query67	15932	15470	15145	15145
query68	10361	674	654	654
query69	567	363	345	345
query70	2057	1539	1420	1420
query71	416	322	313	313
query72	6527	3466	3465	3465
query73	748	331	328	328
query74	6275	5883	5989	5883
query75	5338	3653	3676	3653
query76	6675	1128	1204	1128
query77	1180	253	256	253
query78	12492	11614	11982	11614
query79	7542	641	628	628
query80	1175	409	407	407
query81	499	241	238	238
query82	1560	96	101	96
query83	172	140	133	133
query84	254	73	72	72
query85	866	325	320	320
query86	337	293	280	280
query87	3210	2992	3064	2992
query88	4506	2335	2348	2335
query89	431	323	281	281
query90	1972	215	215	215
query91	189	143	168	143
query92	66	49	52	49
query93	6541	593	598	593
query94	715	204	207	204
query95	1122	1065	1064	1064
query96	642	334	326	326
query97	6550	6419	6353	6353
query98	190	183	176	176
query99	2998	913	855	855
Total cold run time: 317694 ms
Total hot run time: 203692 ms

@doris-robot
Copy link

doris-robot commented Jun 6, 2024

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 928ac23316aefd46735e58cb5ecc45aa4b9f7fe5, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.78	0.06	0.07
query5	0.56	0.53	0.52
query6	1.23	0.63	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.47	0.48
query10	0.53	0.54	0.54
query11	0.11	0.08	0.08
query12	0.12	0.09	0.09
query13	0.61	0.61	0.62
query14	0.79	0.78	0.79
query15	0.78	0.75	0.76
query16	0.37	0.37	0.37
query17	0.98	1.02	0.99
query18	0.22	0.28	0.26
query19	1.87	1.83	1.90
query20	0.02	0.01	0.00
query21	15.46	0.56	0.54
query22	2.25	2.21	1.84
query23	17.48	0.96	1.01
query24	6.92	0.86	0.92
query25	0.36	0.08	0.08
query26	0.73	0.18	0.17
query27	0.04	0.04	0.03
query28	6.44	0.72	0.70
query29	12.63	2.22	2.32
query30	0.55	0.57	0.56
query31	2.80	0.38	0.37
query32	3.40	0.50	0.50
query33	3.07	3.10	3.04
query34	15.28	4.80	4.78
query35	4.82	4.84	4.84
query36	1.07	1.01	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.02	0.01
Total cold run time: 104.57 s
Total hot run time: 30.82 s

@doris-robot
Copy link

doris-robot commented Jun 6, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 928ac23316aefd46735e58cb5ecc45aa4b9f7fe5 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@morrySnow morrySnow merged commit c9721ff into apache:branch-2.0 Jun 6, 2024
morrySnow pushed a commit that referenced this pull request Jun 11, 2024
@starocean999
[fix](nereids)decimal and datetime literal comparison should compare …
1bc27dd 
…datatype too (#36065)

pick from master #36055

this pr revert #35961 and fix the bug in correct way.

The root cause is decimal literal's comparison only compare value, we should also compare their datatype as well as datetime literal.

suppose
a=0 with type decimal(2,1)
b=0 with type decimal(3,1)
we have:
a.equals(b) returns false
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@starocean999
[fix](nereids)should not push down constant output expr in window exp…
007dbca 
…ression (apache#35961)

pick from master apache#35948

SELECT 0 AS x1, (CASE WHEN t.a IS NOT NULL THEN 1.0 / count(1) OVER (PARTITION BY t.b) ELSE 0 END) AS x2 FROM t

0 AS x1 was pushed down to bottom project. Then it will be used to normalize case when expr, and change it to
(CASE WHEN t.a IS NOT NULL THEN 1.0 / count(1) OVER (PARTITION BY t.b) ELSE x1 END) AS x2

the literal 0 was replaced by x1 accidentally. This pr fix it by keep constant expr in output list
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@starocean999
[fix](nereids)decimal and datetime literal comparison should compare …
123cc04 
…datatype too (apache#36065)

pick from master apache#36055

this pr revert apache#35961 and fix the bug in correct way.

The root cause is decimal literal's comparison only compare value, we should also compare their datatype as well as datetime literal.

suppose
a=0 with type decimal(2,1)
b=0 with type decimal(3,1)
we have:
a.equals(b) returns false
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

area/nereids kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@starocean999 @doris-robot @morrySnow