Skip to content

[Fix](nereids) fix ScalarType.getAssignmentCompatibleType() when deal boolean and decimal #34636

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 3 commits into from
May 15, 2024
Merged

[Fix](nereids) fix ScalarType.getAssignmentCompatibleType() when deal boolean and decimal #34636

morrySnow merged 3 commits into from
May 15, 2024

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented May 10, 2024 •
edited
Loading

cherry-pick to branch-2.0 from #34435

@doris-robot
Copy link

doris-robot commented May 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented May 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 11, 2024

TPC-H: Total hot run time: 49636 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4abbdae626f437e9b4242e9b1601ac4fde7a5b7c, data reload: false

------ Round 1 ----------------------------------
q1	17641	4372	4319	4319
q2	2029	155	142	142
q3	10399	1868	1946	1868
q4	10378	1235	1305	1235
q5	8676	3874	3877	3874
q6	232	127	123	123
q7	2026	1598	1597	1597
q8	9279	2710	2706	2706
q9	10634	10356	10312	10312
q10	8632	3488	3492	3488
q11	432	249	245	245
q12	468	300	302	300
q13	18325	3971	4041	3971
q14	354	323	321	321
q15	495	454	448	448
q16	683	568	566	566
q17	1146	949	975	949
q18	7215	6923	6958	6923
q19	1711	1571	1528	1528
q20	545	318	291	291
q21	4430	4076	4048	4048
q22	514	382	392	382
Total cold run time: 116244 ms
Total hot run time: 49636 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4305	4300	4328	4300
q2	317	218	224	218
q3	4185	4122	4105	4105
q4	2734	2731	2716	2716
q5	7152	7065	7065	7065
q6	233	119	120	119
q7	3229	2827	2834	2827
q8	4347	4401	4432	4401
q9	16934	16606	16649	16606
q10	4235	4253	4271	4253
q11	735	706	705	705
q12	1018	868	835	835
q13	6823	3740	3706	3706
q14	446	423	426	423
q15	494	477	450	450
q16	721	666	686	666
q17	3777	3837	3848	3837
q18	8758	8640	8741	8640
q19	1689	1676	1648	1648
q20	2355	2089	2067	2067
q21	8539	8518	8447	8447
q22	1046	966	938	938
Total cold run time: 84072 ms
Total hot run time: 78972 ms

@doris-robot
Copy link

doris-robot commented May 11, 2024

TPC-DS: Total hot run time: 203336 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4abbdae626f437e9b4242e9b1601ac4fde7a5b7c, data reload: false

query1	934	386	376	376
query2	6543	2841	2740	2740
query3	6921	204	207	204
query4	19987	18059	17940	17940
query5	19734	6467	6445	6445
query6	274	227	235	227
query7	4151	299	305	299
query8	238	240	235	235
query9	3084	2735	2634	2634
query10	400	303	330	303
query11	11255	10570	10634	10570
query12	119	75	74	74
query13	5572	703	653	653
query14	18352	13426	13914	13426
query15	392	229	229	229
query16	6470	277	256	256
query17	1743	1476	874	874
query18	2319	407	398	398
query19	202	147	160	147
query20	80	79	79	79
query21	190	93	95	93
query22	5219	5011	5099	5011
query23	32448	31831	31832	31831
query24	6935	6485	6515	6485
query25	511	436	417	417
query26	532	168	165	165
query27	1876	298	298	298
query28	6144	2360	2348	2348
query29	2936	2725	2769	2725
query30	244	160	162	160
query31	937	733	769	733
query32	69	69	57	57
query33	395	256	256	256
query34	874	480	477	477
query35	1086	923	924	923
query36	1303	1255	1225	1225
query37	91	57	60	57
query38	3036	2949	2905	2905
query39	1369	1320	1318	1318
query40	212	99	91	91
query41	37	35	35	35
query42	88	79	84	79
query43	749	651	668	651
query44	1139	704	715	704
query45	239	230	231	230
query46	1237	956	973	956
query47	1790	1657	1938	1657
query48	1006	715	692	692
query49	632	369	368	368
query50	841	649	566	566
query51	4725	4660	4641	4641
query52	92	77	82	77
query53	455	316	319	316
query54	2681	2431	2451	2431
query55	82	74	73	73
query56	210	199	190	190
query57	1185	1112	1080	1080
query58	207	205	190	190
query59	4114	4037	3638	3638
query60	205	191	190	190
query61	86	84	83	83
query62	839	470	479	470
query63	481	340	347	340
query64	2544	1431	1331	1331
query65	3626	3523	3549	3523
query66	778	362	373	362
query67	15790	16503	16050	16050
query68	6345	646	650	646
query69	548	351	347	347
query70	1510	1473	1342	1342
query71	393	315	302	302
query72	6781	3452	3407	3407
query73	735	327	320	320
query74	6281	5842	5842	5842
query75	4646	3690	3728	3690
query76	4099	1136	1209	1136
query77	534	248	243	243
query78	12384	11787	12758	11787
query79	8653	638	619	619
query80	2482	381	393	381
query81	530	236	231	231
query82	689	93	97	93
query83	185	139	132	132
query84	262	67	69	67
query85	1345	296	286	286
query86	365	292	290	290
query87	3187	3027	3025	3025
query88	4916	2352	2342	2342
query89	372	277	276	276
query90	1774	205	211	205
query91	163	144	131	131
query92	59	50	60	50
query93	4172	592	538	538
query94	887	203	199	199
query95	1112	1056	1038	1038
query96	643	340	333	333
query97	6470	6371	6392	6371
query98	187	171	174	171
query99	2910	961	889	889
Total cold run time: 306654 ms
Total hot run time: 203336 ms

@doris-robot
Copy link

doris-robot commented May 11, 2024

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4abbdae626f437e9b4242e9b1601ac4fde7a5b7c, data reload: false

query1	0.01	0.02	0.02
query2	0.08	0.02	0.02
query3	0.25	0.04	0.04
query4	1.78	0.07	0.08
query5	0.53	0.52	0.52
query6	1.23	0.63	0.63
query7	0.01	0.02	0.01
query8	0.03	0.02	0.02
query9	0.53	0.47	0.49
query10	0.54	0.52	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.63	0.61	0.61
query14	0.79	0.77	0.77
query15	0.78	0.76	0.76
query16	0.36	0.37	0.36
query17	1.00	0.99	1.02
query18	0.22	0.25	0.25
query19	1.88	1.85	1.79
query20	0.01	0.01	0.01
query21	15.46	0.58	0.56
query22	2.15	1.80	1.89
query23	17.07	1.11	0.92
query24	5.34	1.19	0.83
query25	0.36	0.15	0.04
query26	0.54	0.14	0.16
query27	0.05	0.05	0.03
query28	7.69	0.73	0.69
query29	12.60	2.25	2.21
query30	0.58	0.52	0.53
query31	2.81	0.38	0.36
query32	3.37	0.50	0.49
query33	3.06	3.07	3.05
query34	15.27	4.80	4.78
query35	4.85	4.84	4.83
query36	1.05	1.00	1.01
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.01	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.01
Total cold run time: 103.53 s
Total hot run time: 30.53 s

@doris-robot
Copy link

doris-robot commented May 11, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4abbdae626f437e9b4242e9b1601ac4fde7a5b7c with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented May 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 11, 2024

TPC-H: Total hot run time: 49294 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2529b96192b86025a79d42c9b8f842710f7c772, data reload: false

------ Round 1 ----------------------------------
q1	18067	4331	4305	4305
q2	2029	155	141	141
q3	10367	1912	1927	1912
q4	10370	1252	1314	1252
q5	8580	3769	3911	3769
q6	233	127	123	123
q7	2006	1608	1552	1552
q8	9248	2696	2695	2695
q9	10613	10429	10156	10156
q10	8684	3489	3486	3486
q11	423	242	256	242
q12	479	298	310	298
q13	18361	3922	4025	3922
q14	349	320	326	320
q15	517	467	461	461
q16	677	559	579	559
q17	1138	956	965	956
q18	7215	6876	6965	6876
q19	1695	1579	1505	1505
q20	507	309	300	300
q21	4403	4120	4075	4075
q22	491	389	394	389
Total cold run time: 116452 ms
Total hot run time: 49294 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4306	4305	4290	4290
q2	316	223	218	218
q3	4141	4171	4127	4127
q4	2743	2739	2753	2739
q5	7161	7071	7082	7071
q6	235	120	122	120
q7	3209	2868	2886	2868
q8	4345	4489	4479	4479
q9	16771	16628	16767	16628
q10	4262	4247	4248	4247
q11	746	690	679	679
q12	1028	848	873	848
q13	6954	3745	3695	3695
q14	445	418	414	414
q15	501	465	460	460
q16	718	674	687	674
q17	3774	3856	3817	3817
q18	8871	8678	8831	8678
q19	1705	1680	1645	1645
q20	2398	2154	2111	2111
q21	8591	8512	8447	8447
q22	1019	971	964	964
Total cold run time: 84239 ms
Total hot run time: 79219 ms

@doris-robot
Copy link

doris-robot commented May 11, 2024

TPC-DS: Total hot run time: 203508 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2529b96192b86025a79d42c9b8f842710f7c772, data reload: false

query1	927	387	383	383
query2	6546	2686	2862	2686
query3	6924	213	211	211
query4	20293	18011	17941	17941
query5	19725	6461	6540	6461
query6	278	220	228	220
query7	4144	307	323	307
query8	255	260	246	246
query9	3184	2711	2632	2632
query10	421	304	296	296
query11	11432	10795	10752	10752
query12	123	74	70	70
query13	5571	678	662	662
query14	17294	13067	13272	13067
query15	363	221	229	221
query16	6466	280	256	256
query17	1711	1468	866	866
query18	2323	409	397	397
query19	202	146	147	146
query20	78	76	85	76
query21	184	96	91	91
query22	5095	5068	5026	5026
query23	32596	32046	32033	32033
query24	7088	6544	6471	6471
query25	495	436	419	419
query26	535	162	158	158
query27	1896	298	299	298
query28	6190	2349	2314	2314
query29	2959	2691	2835	2691
query30	238	166	163	163
query31	911	790	778	778
query32	67	60	65	60
query33	403	266	258	258
query34	856	462	492	462
query35	1079	946	937	937
query36	1354	1114	1260	1114
query37	93	60	60	60
query38	3070	2913	2928	2913
query39	1397	1318	1332	1318
query40	206	93	94	93
query41	43	36	35	35
query42	90	86	93	86
query43	754	794	834	794
query44	1124	706	712	706
query45	237	227	224	224
query46	1223	971	1056	971
query47	1812	1660	1717	1660
query48	1016	743	706	706
query49	623	371	369	369
query50	865	628	610	610
query51	4749	4759	4675	4675
query52	96	80	93	80
query53	438	319	319	319
query54	2644	2459	2500	2459
query55	93	74	91	74
query56	225	217	213	213
query57	1317	1184	1131	1131
query58	209	184	208	184
query59	4344	3873	4244	3873
query60	224	215	214	214
query61	89	86	86	86
query62	798	482	470	470
query63	490	356	339	339
query64	2538	1471	1485	1471
query65	3635	3559	3572	3559
query66	776	382	384	382
query67	16129	15720	15822	15720
query68	8804	673	642	642
query69	575	358	339	339
query70	1550	1610	1493	1493
query71	405	311	320	311
query72	6435	3472	3457	3457
query73	733	327	321	321
query74	6314	5850	5882	5850
query75	5276	3790	3640	3640
query76	5495	1141	1188	1141
query77	898	251	252	251
query78	12710	11570	11701	11570
query79	8309	641	664	641
query80	1340	392	385	385
query81	488	238	228	228
query82	1676	99	96	96
query83	168	134	129	129
query84	257	67	72	67
query85	847	298	294	294
query86	336	299	310	299
query87	3200	3057	3070	3057
query88	5103	2314	2339	2314
query89	387	314	277	277
query90	1963	209	201	201
query91	173	134	134	134
query92	62	52	51	51
query93	5590	550	530	530
query94	697	207	201	201
query95	1125	1055	1041	1041
query96	644	328	332	328
query97	6538	6325	6248	6248
query98	194	181	171	171
query99	2923	856	838	838
Total cold run time: 312768 ms
Total hot run time: 203508 ms

@doris-robot
Copy link

doris-robot commented May 11, 2024

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2529b96192b86025a79d42c9b8f842710f7c772, data reload: false

query1	0.02	0.02	0.01
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.81	0.07	0.07
query5	0.53	0.53	0.51
query6	1.24	0.61	0.62
query7	0.01	0.00	0.01
query8	0.04	0.02	0.02
query9	0.53	0.49	0.50
query10	0.55	0.54	0.55
query11	0.12	0.09	0.10
query12	0.12	0.09	0.09
query13	0.63	0.62	0.62
query14	0.78	0.79	0.80
query15	0.79	0.75	0.76
query16	0.37	0.37	0.39
query17	1.01	1.00	1.03
query18	0.21	0.24	0.26
query19	1.86	1.89	1.82
query20	0.02	0.01	0.01
query21	15.44	0.57	0.56
query22	1.94	1.70	1.55
query23	17.06	0.99	0.94
query24	4.45	1.32	1.08
query25	0.36	0.12	0.05
query26	0.54	0.15	0.16
query27	0.04	0.05	0.04
query28	8.30	0.76	0.74
query29	12.63	2.36	2.17
query30	0.60	0.52	0.50
query31	2.80	0.39	0.37
query32	3.40	0.50	0.49
query33	3.11	3.05	3.03
query34	15.25	4.80	4.78
query35	4.86	4.83	4.86
query36	1.06	1.00	1.01
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.18 s
Total hot run time: 30.61 s

@doris-robot
Copy link

doris-robot commented May 11, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d2529b96192b86025a79d42c9b8f842710f7c772 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 15, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 15, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 3578ba6 into apache:branch-2.0 May 15, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@feiniaofeiafei
[fix](planner) fix ScalarType.getAssignmentCompatibleType() when deal…
476a65f 
… boolean and decimal (apache#34636)

cherry-pick to branch-2.0 from apache#34435

The legacy planner encounters issues when handling filters such as: c1(boolean type)=0.0(decimalv3).
The literal 0.0 is interpreted as decimalv3(1,1), and the boolean type c1 is coerced to decimalv3(1,1).
decimalv3(1,1) can only retain values in the range [0,1), while the boolean true is represented as 1, exceeding the upper bound, thus causing an overflow problem.
This pull request addresses this issue by considering the boolean type as decimalv3(1,0), making both c1 and 0.0 being cast to decimal(2,1).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@feiniaofeiafei @doris-robot @morrySnow