Skip to content

[fix](dynamic partition) drop partition exclude history_partition_num #37539 #37571

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 2 commits into from
Jul 10, 2024
Merged

[fix](dynamic partition) drop partition exclude history_partition_num #37539 #37571

xiaokang merged 2 commits into from
Jul 10, 2024

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Jul 10, 2024

cherry pick from #37539

@yujun777 @dataroaring
[fix](dynamic partition) drop partition exclude history_partition_num (…
713e70e 
…apache#37539)

FIX:   

When dropping dynamic partition, PR apache#35778 will use math.max(start,
-history_partition_num) as the first partition, but it may delete users'
partitions if they specify both start and history_partition_num
inappropriately. For safety reason, revert this behavious changed, only
use start as the first partition when dropping partitions.

For those who had specified a very small start value, drop partitions
will catch an exception , and stop dropping this table's partition and
then record this error in dynamic info. Users can use command `SHOW
DYNAMIC PARTITION TABLES FROM DBXXX` to know this error. From this
error, it will give user hint to modify start if they really specify a
error start.

---------

Co-authored-by: Yongqiang YANG <98214048+dataroaring@users.noreply.github.com>
@yujun777
Copy link
Contributor Author

yujun777 commented Jul 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777
Copy link
Contributor Author

yujun777 commented Jul 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 49675 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e69bb98646fcb45f045604ce8f2622302a06902a, data reload: false

------ Round 1 ----------------------------------
q1	17721	4415	4347	4347
q2	2073	156	149	149
q3	10261	1915	1897	1897
q4	10356	1214	1309	1214
q5	8466	3906	3905	3905
q6	235	147	127	127
q7	2055	1600	1568	1568
q8	9289	2719	2698	2698
q9	10536	10339	10115	10115
q10	8612	3483	3468	3468
q11	424	250	252	250
q12	474	305	312	305
q13	18338	4043	4102	4043
q14	367	313	326	313
q15	508	472	472	472
q16	684	580	581	580
q17	1126	952	983	952
q18	7299	6819	6910	6819
q19	1789	1631	1618	1618
q20	549	329	311	311
q21	4451	4090	4135	4090
q22	529	434	439	434
Total cold run time: 116142 ms
Total hot run time: 49675 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4374	4343	4295	4295
q2	326	229	230	229
q3	4204	4161	4176	4161
q4	2749	2756	2729	2729
q5	7194	7089	7090	7089
q6	237	120	120	120
q7	3232	2848	2812	2812
q8	4349	4532	4441	4441
q9	16919	16765	16850	16765
q10	4226	4271	4290	4271
q11	738	679	719	679
q12	1025	863	861	861
q13	6900	3778	3796	3778
q14	448	428	420	420
q15	522	464	456	456
q16	732	677	689	677
q17	3820	3907	3916	3907
q18	8963	8787	8753	8753
q19	1699	1742	1653	1653
q20	2385	2154	2121	2121
q21	8464	8402	8331	8331
q22	1075	1001	997	997
Total cold run time: 84581 ms
Total hot run time: 79545 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 203521 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e69bb98646fcb45f045604ce8f2622302a06902a, data reload: false

query1	920	422	389	389
query2	6526	2729	2890	2729
query3	6922	209	199	199
query4	20095	18066	18084	18066
query5	19746	6577	6541	6541
query6	297	223	233	223
query7	4162	309	319	309
query8	415	409	401	401
query9	3083	2668	2595	2595
query10	419	313	284	284
query11	11283	10658	10679	10658
query12	121	74	78	74
query13	5616	692	709	692
query14	17817	13285	13741	13285
query15	368	251	241	241
query16	6473	284	263	263
query17	1703	1472	886	886
query18	2289	413	404	404
query19	211	149	155	149
query20	83	79	81	79
query21	194	94	105	94
query22	5287	5371	5145	5145
query23	32506	32142	31959	31959
query24	6961	6574	6554	6554
query25	540	438	428	428
query26	527	166	164	164
query27	1836	298	307	298
query28	6086	2343	2309	2309
query29	2886	2804	2642	2642
query30	248	172	172	172
query31	907	788	760	760
query32	69	65	63	63
query33	399	261	245	245
query34	849	490	488	488
query35	1136	934	840	840
query36	1540	1457	1066	1066
query37	94	59	59	59
query38	3072	2916	2907	2907
query39	1368	1344	1327	1327
query40	209	99	96	96
query41	46	45	43	43
query42	84	86	88	86
query43	943	679	694	679
query44	1149	725	737	725
query45	243	239	234	234
query46	1235	957	987	957
query47	2134	1702	1730	1702
query48	1017	719	713	713
query49	608	378	377	377
query50	859	615	619	615
query51	4797	4648	4658	4648
query52	93	86	81	81
query53	453	324	337	324
query54	2811	2481	2455	2455
query55	89	86	83	83
query56	229	228	203	203
query57	1146	1074	1103	1074
query58	218	204	212	204
query59	4695	4252	3691	3691
query60	219	216	205	205
query61	95	92	96	92
query62	791	450	472	450
query63	498	343	344	343
query64	2323	1535	1464	1464
query65	3697	3598	3594	3594
query66	786	387	389	387
query67	17845	15604	15102	15102
query68	8405	655	666	655
query69	567	348	349	348
query70	1548	1354	1383	1354
query71	404	317	325	317
query72	6582	3501	3485	3485
query73	734	325	323	323
query74	6330	5841	5819	5819
query75	4602	3598	3720	3598
query76	4801	1112	1170	1112
query77	664	258	260	258
query78	12574	11749	11978	11749
query79	8237	673	662	662
query80	1173	396	410	396
query81	498	239	243	239
query82	1718	102	103	102
query83	188	138	135	135
query84	252	72	70	70
query85	888	331	332	331
query86	327	295	293	293
query87	3289	3025	3071	3025
query88	5014	2321	2322	2321
query89	382	288	292	288
query90	1935	212	217	212
query91	171	148	151	148
query92	61	59	52	52
query93	5879	556	563	556
query94	743	217	205	205
query95	1129	1089	1073	1073
query96	636	338	331	331
query97	6475	6479	6429	6429
query98	192	174	186	174
query99	2984	847	922	847
Total cold run time: 313748 ms
Total hot run time: 203521 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e69bb98646fcb45f045604ce8f2622302a06902a, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.03	0.03
query3	0.25	0.05	0.05
query4	1.80	0.07	0.07
query5	0.54	0.53	0.52
query6	1.26	0.62	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.52	0.47	0.47
query10	0.54	0.53	0.55
query11	0.13	0.09	0.09
query12	0.12	0.09	0.08
query13	0.63	0.62	0.60
query14	0.80	0.79	0.80
query15	0.79	0.77	0.78
query16	0.38	0.37	0.39
query17	1.03	1.02	1.04
query18	0.24	0.24	0.23
query19	1.90	1.85	1.82
query20	0.01	0.01	0.01
query21	15.50	0.57	0.56
query22	2.41	2.22	1.46
query23	16.84	1.09	0.93
query24	6.05	0.89	1.41
query25	0.38	0.13	0.05
query26	0.61	0.16	0.17
query27	0.05	0.03	0.03
query28	6.83	0.78	0.75
query29	12.75	2.31	2.25
query30	0.62	0.54	0.53
query31	2.82	0.40	0.38
query32	3.34	0.49	0.51
query33	3.07	3.12	3.07
query34	15.26	4.80	4.80
query35	4.85	4.86	4.84
query36	1.12	1.01	1.02
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.99 s
Total hot run time: 30.56 s

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e69bb98646fcb45f045604ce8f2622302a06902a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang xiaokang merged commit 9ca846a into apache:branch-2.0 Jul 10, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@yujun777
[fix](dynamic partition) drop partition exclude history_partition_num a…
4352d79 
…pache#37539 (apache#37571)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yujun777 @doris-robot @xiaokang