Skip to content

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Aug 1, 2024

During ReplacePartitionClause.analyze(), the default value of isStrictRange is true. However, ReplacePartitionClause instances constructed by internal code do not call analyze(), so their isStrictRange value is incorrect (e.g., INSERT INTO ... OVERWRITE).

This PR handle the relevant properties when constructing ReplacePartitionClause.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Aug 1, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 1, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41905 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bc3da05d1601eb79b9f899867eeca81e7238910, data reload: false

------ Round 1 ----------------------------------
q1	18889	4272	4131	4131
q2	2021	210	208	208
q3	10444	1323	1313	1313
q4	10171	875	912	875
q5	7673	3022	2992	2992
q6	226	142	142	142
q7	1052	618	615	615
q8	9447	1945	1989	1945
q9	8514	6601	6612	6601
q10	8717	3808	3875	3808
q11	427	257	260	257
q12	410	242	234	234
q13	17760	2941	2946	2941
q14	272	247	239	239
q15	538	493	486	486
q16	493	392	388	388
q17	966	935	947	935
q18	7968	7432	7227	7227
q19	1380	1219	1221	1219
q20	564	329	338	329
q21	5328	4788	4738	4738
q22	363	286	282	282
Total cold run time: 113623 ms
Total hot run time: 41905 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4064	4042	4036	4036
q2	329	230	220	220
q3	2994	2993	2995	2993
q4	1892	1889	1865	1865
q5	5292	5262	5250	5250
q6	221	133	134	133
q7	2065	1675	1682	1675
q8	3248	3293	3289	3289
q9	8349	8327	8313	8313
q10	3753	3866	3810	3810
q11	547	457	443	443
q12	720	567	557	557
q13	17013	2962	2956	2956
q14	289	256	253	253
q15	517	485	481	481
q16	433	402	394	394
q17	1718	1697	1694	1694
q18	7790	7329	7182	7182
q19	1675	1687	1675	1675
q20	1981	1772	1760	1760
q21	5445	5237	5189	5189
q22	511	477	473	473
Total cold run time: 70846 ms
Total hot run time: 54641 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169238 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bc3da05d1601eb79b9f899867eeca81e7238910, data reload: false

query1	909	375	366	366
query2	6466	1716	1634	1634
query3	6679	222	232	222
query4	20241	17413	17577	17413
query5	4305	517	529	517
query6	283	175	164	164
query7	4605	302	300	300
query8	259	194	196	194
query9	8530	2374	2353	2353
query10	465	289	269	269
query11	10522	10021	10174	10021
query12	147	87	89	87
query13	1635	376	377	376
query14	9574	7975	7228	7228
query15	227	165	170	165
query16	7117	471	494	471
query17	954	574	557	557
query18	1906	298	282	282
query19	213	148	152	148
query20	104	89	87	87
query21	203	102	100	100
query22	4242	3928	3881	3881
query23	33757	32802	32834	32802
query24	10349	3124	3031	3031
query25	706	404	413	404
query26	1789	154	153	153
query27	2994	284	286	284
query28	6979	1977	1962	1962
query29	1366	445	433	433
query30	285	153	152	152
query31	937	775	759	759
query32	104	57	59	57
query33	711	323	333	323
query34	899	488	487	487
query35	844	746	732	732
query36	1003	875	888	875
query37	302	82	81	81
query38	2889	2756	2767	2756
query39	870	805	814	805
query40	297	129	118	118
query41	48	47	50	47
query42	120	102	103	102
query43	452	411	425	411
query44	1195	731	727	727
query45	209	178	180	178
query46	1102	825	792	792
query47	1798	1705	1690	1690
query48	367	302	294	294
query49	1192	456	454	454
query50	905	444	445	444
query51	6855	6652	6641	6641
query52	101	96	93	93
query53	259	187	185	185
query54	646	463	460	460
query55	80	84	81	81
query56	304	276	263	263
query57	1167	1055	1030	1030
query58	309	282	275	275
query59	2694	2412	2593	2412
query60	311	280	316	280
query61	119	113	114	113
query62	1061	670	649	649
query63	220	189	193	189
query64	5823	1925	1879	1879
query65	3271	3112	3092	3092
query66	1429	337	332	332
query67	15317	14763	14652	14652
query68	6888	589	593	589
query69	742	371	304	304
query70	1096	1047	1124	1047
query71	574	282	286	282
query72	8020	2680	2502	2502
query73	981	329	337	329
query74	6006	5659	5676	5659
query75	4498	2755	2706	2706
query76	4939	1408	1425	1408
query77	764	328	322	322
query78	9526	8982	8830	8830
query79	3716	542	545	542
query80	1115	528	498	498
query81	576	229	230	229
query82	778	138	133	133
query83	345	182	177	177
query84	275	80	88	80
query85	1545	322	305	305
query86	443	336	300	300
query87	3283	3116	3109	3109
query88	3776	2433	2422	2422
query89	459	289	293	289
query90	2107	198	197	197
query91	129	105	101	101
query92	64	53	50	50
query93	4505	616	616	616
query94	971	282	293	282
query95	391	263	275	263
query96	623	283	280	280
query97	3235	3064	3108	3064
query98	210	203	195	195
query99	1631	1316	1314	1314
Total cold run time: 278232 ms
Total hot run time: 169238 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bc3da05d1601eb79b9f899867eeca81e7238910, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.07
query5	0.49	0.48	0.48
query6	1.12	0.72	0.71
query7	0.02	0.01	0.02
query8	0.06	0.05	0.04
query9	0.57	0.50	0.52
query10	0.57	0.57	0.56
query11	0.16	0.12	0.12
query12	0.14	0.13	0.13
query13	0.62	0.59	0.61
query14	0.78	0.80	0.80
query15	0.91	0.87	0.87
query16	0.37	0.36	0.35
query17	1.00	1.01	1.00
query18	0.22	0.21	0.21
query19	1.84	1.79	1.78
query20	0.01	0.01	0.01
query21	15.42	0.75	0.65
query22	4.02	7.51	1.14
query23	17.81	1.45	1.32
query24	2.29	0.22	0.23
query25	0.19	0.09	0.08
query26	0.33	0.22	0.22
query27	0.46	0.23	0.23
query28	13.15	1.00	0.97
query29	12.50	3.36	3.36
query30	0.26	0.06	0.06
query31	2.84	0.41	0.41
query32	3.23	0.50	0.48
query33	2.96	2.98	2.96
query34	15.49	4.26	4.24
query35	4.31	4.28	4.28
query36	0.68	0.48	0.47
query37	0.19	0.17	0.16
query38	0.17	0.15	0.14
query39	0.05	0.04	0.03
query40	0.16	0.12	0.12
query41	0.10	0.05	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.61 s
Total hot run time: 30.03 s

During ReplacePartitionClause.analyze(), the default value of isStrictRange is true.
However, ReplacePartitionClause instances constructed by internal code do not call analyze(),
so their isStrictRange value is incorrect (e.g., INSERT INTO ... OVERWRITE).

This PR handle the relevant properties when constructing ReplacePartitionClause.
@w41ter w41ter force-pushed the fix_is_strict_range branch from 6bc3da0 to 6cd0f80 Compare August 1, 2024 09:29
@w41ter
Copy link
Contributor Author

w41ter commented Aug 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42077 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6cd0f80ec0b82c62eda4242968d060cb3cc8ed41, data reload: false

------ Round 1 ----------------------------------
q1	18086	4207	4161	4161
q2	3016	203	206	203
q3	12465	1384	1401	1384
q4	11062	810	960	810
q5	7988	2937	2942	2937
q6	219	140	142	140
q7	1036	615	606	606
q8	9438	1932	1901	1901
q9	8428	6602	6621	6602
q10	8711	3838	3870	3838
q11	422	251	250	250
q12	414	235	236	235
q13	17755	2964	2966	2964
q14	284	245	250	245
q15	529	495	492	492
q16	492	409	393	393
q17	951	929	928	928
q18	7917	7390	7359	7359
q19	1381	1224	1216	1216
q20	576	321	342	321
q21	5375	4826	4806	4806
q22	352	286	287	286
Total cold run time: 116897 ms
Total hot run time: 42077 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4068	4050	4045	4045
q2	333	228	222	222
q3	3009	2999	3001	2999
q4	1899	1985	1880	1880
q5	5247	5264	5223	5223
q6	218	132	132	132
q7	2043	1691	1679	1679
q8	3180	3239	3258	3239
q9	8267	8264	8277	8264
q10	3752	3866	3885	3866
q11	543	451	456	451
q12	727	542	573	542
q13	12438	2971	2997	2971
q14	295	261	257	257
q15	515	484	476	476
q16	457	398	403	398
q17	1713	1705	1672	1672
q18	7784	7355	7210	7210
q19	1689	1679	1675	1675
q20	1967	1761	1761	1761
q21	5388	5228	5275	5228
q22	534	439	438	438
Total cold run time: 66066 ms
Total hot run time: 54628 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168545 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6cd0f80ec0b82c62eda4242968d060cb3cc8ed41, data reload: false

query1	915	371	373	371
query2	6469	1776	1773	1773
query3	6679	219	222	219
query4	20070	17553	17350	17350
query5	4311	521	528	521
query6	291	191	164	164
query7	4617	292	293	292
query8	272	198	197	197
query9	8528	2369	2346	2346
query10	448	275	267	267
query11	10530	10098	10084	10084
query12	137	90	89	89
query13	1624	385	375	375
query14	9309	6527	7426	6527
query15	211	168	162	162
query16	7104	468	484	468
query17	957	569	560	560
query18	1931	290	287	287
query19	197	150	149	149
query20	93	86	90	86
query21	208	101	100	100
query22	4155	4055	3917	3917
query23	33812	32934	33465	32934
query24	10321	3010	3061	3010
query25	723	407	410	407
query26	1817	155	153	153
query27	2872	280	281	280
query28	6952	1976	1956	1956
query29	1351	436	432	432
query30	290	152	151	151
query31	941	756	773	756
query32	103	57	57	57
query33	704	320	330	320
query34	918	484	487	484
query35	849	711	766	711
query36	1004	878	875	875
query37	312	81	79	79
query38	2899	2842	2791	2791
query39	872	814	824	814
query40	279	115	117	115
query41	51	48	51	48
query42	131	105	109	105
query43	512	434	426	426
query44	1177	719	739	719
query45	214	186	181	181
query46	1090	824	820	820
query47	1824	1705	1715	1705
query48	369	297	305	297
query49	1208	452	436	436
query50	905	436	443	436
query51	6836	6569	6702	6569
query52	103	88	91	88
query53	253	183	184	183
query54	643	462	456	456
query55	79	79	78	78
query56	277	266	272	266
query57	1123	1051	1068	1051
query58	303	263	276	263
query59	2752	2467	2549	2467
query60	313	281	289	281
query61	116	112	113	112
query62	1061	656	675	656
query63	225	189	184	184
query64	5935	1910	1894	1894
query65	3171	3080	3136	3080
query66	1427	336	326	326
query67	15674	14764	15004	14764
query68	8975	595	590	590
query69	761	380	316	316
query70	1135	1111	1015	1015
query71	534	284	273	273
query72	7845	2679	2470	2470
query73	1100	326	332	326
query74	6068	5687	5634	5634
query75	4809	2723	2726	2723
query76	6016	1297	1295	1295
query77	923	348	308	308
query78	9506	9036	8849	8849
query79	6741	536	533	533
query80	856	504	501	501
query81	539	231	238	231
query82	305	141	130	130
query83	312	168	172	168
query84	260	83	77	77
query85	999	307	327	307
query86	367	310	283	283
query87	3305	3147	3052	3052
query88	4016	2390	2417	2390
query89	510	281	293	281
query90	2117	195	195	195
query91	127	100	103	100
query92	65	50	54	50
query93	5515	602	613	602
query94	992	300	293	293
query95	385	263	271	263
query96	603	280	278	278
query97	3269	3066	3019	3019
query98	218	202	200	200
query99	1545	1310	1273	1273
Total cold run time: 284565 ms
Total hot run time: 168545 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6cd0f80ec0b82c62eda4242968d060cb3cc8ed41, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.22	0.04	0.06
query4	1.67	0.09	0.09
query5	0.48	0.48	0.50
query6	1.14	0.72	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.51
query10	0.58	0.56	0.58
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.77	0.79	0.79
query15	0.90	0.86	0.87
query16	0.34	0.36	0.35
query17	0.98	1.04	0.96
query18	0.22	0.21	0.21
query19	1.83	1.78	1.76
query20	0.03	0.01	0.01
query21	15.40	0.77	0.68
query22	3.96	7.64	1.28
query23	18.07	1.32	1.30
query24	2.23	0.23	0.22
query25	0.19	0.08	0.09
query26	0.32	0.21	0.22
query27	0.46	0.23	0.22
query28	13.19	1.00	0.98
query29	12.52	3.32	3.32
query30	0.25	0.06	0.06
query31	2.87	0.41	0.41
query32	3.24	0.49	0.48
query33	2.94	2.97	2.92
query34	15.44	4.28	4.27
query35	4.31	4.28	4.35
query36	0.68	0.48	0.50
query37	0.18	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.12
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.59 s
Total hot run time: 30.05 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 312c7e1 into apache:master Aug 4, 2024
@w41ter w41ter deleted the fix_is_strict_range branch August 5, 2024 01:34
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Aug 5, 2024
…ge (apache#38688)

During ReplacePartitionClause.analyze(), the default value of
isStrictRange is true. However, ReplacePartitionClause instances
constructed by internal code do not call analyze(), so their
isStrictRange value is incorrect (e.g., INSERT INTO ... OVERWRITE).

This PR handle the relevant properties when constructing
ReplacePartitionClause.
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Aug 5, 2024
…ge (apache#38688)

During ReplacePartitionClause.analyze(), the default value of
isStrictRange is true. However, ReplacePartitionClause instances
constructed by internal code do not call analyze(), so their
isStrictRange value is incorrect (e.g., INSERT INTO ... OVERWRITE).

This PR handle the relevant properties when constructing
ReplacePartitionClause.
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
…ge (#38688)

During ReplacePartitionClause.analyze(), the default value of
isStrictRange is true. However, ReplacePartitionClause instances
constructed by internal code do not call analyze(), so their
isStrictRange value is incorrect (e.g., INSERT INTO ... OVERWRITE).

This PR handle the relevant properties when constructing
ReplacePartitionClause.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
…ge (#38688)

During ReplacePartitionClause.analyze(), the default value of
isStrictRange is true. However, ReplacePartitionClause instances
constructed by internal code do not call analyze(), so their
isStrictRange value is incorrect (e.g., INSERT INTO ... OVERWRITE).

This PR handle the relevant properties when constructing
ReplacePartitionClause.
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants