Skip to content

[Enhancement](alter) support add partitions in batch #37114

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 7 commits into from
Aug 14, 2024
Merged

[Enhancement](alter) support add partitions in batch #37114

dataroaring merged 7 commits into from
Aug 14, 2024

Conversation

@felixwluo
Copy link
Contributor

@felixwluo felixwluo commented Jul 1, 2024

Proposed changes

Issue Number: close #32524

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@felixwluo
Copy link
Contributor Author

felixwluo commented Jul 1, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

TPC-H: Total hot run time: 39534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ed4c1cc6a34d9ab55afb530df32ba1d53fef292f, data reload: false

------ Round 1 ----------------------------------
q1	17610	4323	4285	4285
q2	2021	194	190	190
q3	10471	1158	1189	1158
q4	10205	783	802	783
q5	7473	2667	2611	2611
q6	236	139	135	135
q7	942	593	601	593
q8	9234	2056	2056	2056
q9	8851	6439	6439	6439
q10	8925	3713	3647	3647
q11	456	257	226	226
q12	444	227	219	219
q13	17769	2951	2944	2944
q14	257	224	224	224
q15	515	466	477	466
q16	499	369	364	364
q17	957	715	672	672
q18	7943	7465	7359	7359
q19	6455	1378	1407	1378
q20	647	320	328	320
q21	4858	3133	3199	3133
q22	386	332	333	332
Total cold run time: 117154 ms
Total hot run time: 39534 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4452	4251	4248	4248
q2	362	267	256	256
q3	2929	2735	2825	2735
q4	2014	1749	1711	1711
q5	5568	5456	5494	5456
q6	223	128	128	128
q7	2164	1790	1875	1790
q8	3245	3396	3372	3372
q9	8661	8637	8854	8637
q10	4112	3902	3707	3707
q11	571	486	498	486
q12	787	646	658	646
q13	16007	3186	3170	3170
q14	328	261	281	261
q15	527	478	493	478
q16	471	420	429	420
q17	1777	1523	1504	1504
q18	8080	7979	7769	7769
q19	2582	1570	1471	1471
q20	2067	1881	1882	1881
q21	5100	5003	4978	4978
q22	601	541	536	536
Total cold run time: 72628 ms
Total hot run time: 55640 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

TPC-DS: Total hot run time: 174581 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ed4c1cc6a34d9ab55afb530df32ba1d53fef292f, data reload: false

query1	912	374	369	369
query2	6447	2468	2312	2312
query3	6636	207	216	207
query4	19283	17409	17049	17049
query5	3643	478	481	478
query6	246	193	161	161
query7	4584	293	298	293
query8	304	281	292	281
query9	8929	2444	2397	2397
query10	585	312	300	300
query11	11145	9903	10046	9903
query12	122	85	88	85
query13	1648	379	354	354
query14	10100	7670	7783	7670
query15	290	191	193	191
query16	7948	281	277	277
query17	1901	583	534	534
query18	1920	277	271	271
query19	208	158	158	158
query20	89	82	83	82
query21	213	137	131	131
query22	4547	4173	4055	4055
query23	33647	33627	33649	33627
query24	10698	2956	2923	2923
query25	614	409	392	392
query26	717	162	155	155
query27	2249	327	320	320
query28	5965	2121	2127	2121
query29	909	680	652	652
query30	267	170	161	161
query31	990	771	759	759
query32	98	56	55	55
query33	669	315	294	294
query34	906	473	503	473
query35	778	639	638	638
query36	1137	983	966	966
query37	147	87	79	79
query38	2911	2813	2848	2813
query39	910	833	830	830
query40	204	130	129	129
query41	56	54	54	54
query42	109	104	106	104
query43	621	577	570	570
query44	1065	718	751	718
query45	189	167	163	163
query46	1083	733	713	713
query47	1868	1781	1743	1743
query48	369	302	308	302
query49	861	517	411	411
query50	773	376	379	376
query51	6908	6766	6665	6665
query52	106	88	86	86
query53	359	293	292	292
query54	918	454	434	434
query55	75	84	73	73
query56	284	287	263	263
query57	1105	1049	1051	1049
query58	243	233	234	233
query59	3458	3118	3114	3114
query60	292	271	277	271
query61	110	92	91	91
query62	616	462	429	429
query63	312	303	285	285
query64	8529	2239	1782	1782
query65	3152	3139	3113	3113
query66	750	327	331	327
query67	15707	15055	15059	15055
query68	8711	549	554	549
query69	778	441	330	330
query70	1374	1116	1109	1109
query71	507	280	272	272
query72	8761	5532	5411	5411
query73	2237	329	317	317
query74	5904	5529	5526	5526
query75	5258	2677	2685	2677
query76	5345	950	956	950
query77	790	304	295	295
query78	10380	9880	10276	9880
query79	7226	512	514	512
query80	917	463	472	463
query81	557	216	217	216
query82	259	104	106	104
query83	334	215	165	165
query84	265	86	83	83
query85	905	271	267	267
query86	358	303	305	303
query87	3274	3107	3080	3080
query88	4128	2375	2363	2363
query89	513	379	379	379
query90	2043	189	182	182
query91	129	102	101	101
query92	64	50	49	49
query93	4180	502	498	498
query94	1353	192	186	186
query95	413	354	317	317
query96	604	277	267	267
query97	3173	3041	3062	3041
query98	210	197	195	195
query99	1082	872	833	833
Total cold run time: 287466 ms
Total hot run time: 174581 ms

@doris-robot
Copy link

doris-robot commented Jul 1, 2024

ClickBench: Total hot run time: 30.96 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ed4c1cc6a34d9ab55afb530df32ba1d53fef292f, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.04
query4	1.66	0.07	0.08
query5	0.50	0.49	0.47
query6	1.14	0.73	0.74
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.56	0.49	0.50
query10	0.53	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.80	0.80	0.79
query15	0.84	0.81	0.81
query16	0.37	0.37	0.35
query17	1.03	1.04	0.96
query18	0.23	0.24	0.24
query19	1.80	1.72	1.83
query20	0.01	0.01	0.01
query21	15.44	0.79	0.66
query22	3.86	7.13	2.30
query23	18.26	1.34	1.29
query24	2.16	0.24	0.21
query25	0.15	0.08	0.09
query26	0.28	0.18	0.18
query27	0.09	0.07	0.07
query28	13.16	1.03	0.99
query29	12.58	3.29	3.26
query30	0.26	0.06	0.05
query31	2.85	0.41	0.38
query32	3.26	0.47	0.46
query33	2.88	2.91	2.90
query34	17.11	4.41	4.39
query35	4.51	4.52	4.52
query36	0.66	0.47	0.47
query37	0.19	0.16	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.05 s
Total hot run time: 30.96 s

cambyzju
cambyzju previously approved these changes Jul 2, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by anyone and no changes requested.

lide-reed
lide-reed previously approved these changes Jul 3, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@felixwluo felixwluo mentioned this pull request Jul 4, 2024
Closed
xy720
xy720 previously approved these changes Jul 4, 2024
View reviewed changes
@xy720
Copy link
Member

xy720 commented Jul 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 5, 2024

TPC-H: Total hot run time: 39627 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f2cabc8462c9673ad860c37e3dc8eaad7033f604, data reload: false

------ Round 1 ----------------------------------
q1	17629	4356	4236	4236
q2	2019	192	186	186
q3	10442	1238	1186	1186
q4	10189	834	722	722
q5	7573	2608	2542	2542
q6	221	139	137	137
q7	954	590	601	590
q8	9232	2027	2051	2027
q9	9051	6433	6469	6433
q10	8977	3719	3717	3717
q11	460	238	236	236
q12	508	235	241	235
q13	18616	3019	2976	2976
q14	277	226	235	226
q15	520	488	487	487
q16	516	375	371	371
q17	966	630	687	630
q18	8026	7531	7450	7450
q19	6342	1351	1516	1351
q20	667	329	325	325
q21	5055	3219	3913	3219
q22	403	345	355	345
Total cold run time: 118643 ms
Total hot run time: 39627 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4342	4237	4207	4207
q2	380	272	266	266
q3	2977	2919	2914	2914
q4	2000	1722	1727	1722
q5	5546	5471	5449	5449
q6	237	132	130	130
q7	2216	1860	1874	1860
q8	3267	3387	3426	3387
q9	8652	8751	8814	8751
q10	4118	3762	3778	3762
q11	624	502	499	499
q12	839	662	633	633
q13	16368	3164	3180	3164
q14	313	274	277	274
q15	549	493	481	481
q16	497	434	427	427
q17	1806	1534	1503	1503
q18	8116	8038	7847	7847
q19	1788	1586	1661	1586
q20	2137	1885	1878	1878
q21	5087	4845	4691	4691
q22	666	555	559	555
Total cold run time: 72525 ms
Total hot run time: 55986 ms

@doris-robot
Copy link

doris-robot commented Jul 5, 2024

TPC-DS: Total hot run time: 170831 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f2cabc8462c9673ad860c37e3dc8eaad7033f604, data reload: false

query1	933	367	362	362
query2	6458	2595	2456	2456
query3	6625	208	213	208
query4	27971	17591	17293	17293
query5	3692	491	473	473
query6	258	163	164	163
query7	4592	314	300	300
query8	324	325	301	301
query9	8573	2442	2436	2436
query10	571	325	283	283
query11	11702	10156	10115	10115
query12	118	86	88	86
query13	1658	376	372	372
query14	10100	6791	7710	6791
query15	224	184	192	184
query16	7749	313	314	313
query17	1803	554	535	535
query18	1863	289	284	284
query19	203	158	149	149
query20	92	84	84	84
query21	213	132	130	130
query22	4450	4095	4130	4095
query23	33806	33782	33620	33620
query24	11039	2861	2861	2861
query25	656	451	423	423
query26	1186	156	157	156
query27	2436	319	337	319
query28	7366	2156	2141	2141
query29	920	664	655	655
query30	249	162	153	153
query31	947	799	758	758
query32	99	55	62	55
query33	773	317	310	310
query34	886	505	512	505
query35	746	669	656	656
query36	1127	987	976	976
query37	158	82	84	82
query38	2994	2862	2820	2820
query39	902	830	796	796
query40	212	129	132	129
query41	62	53	58	53
query42	122	98	105	98
query43	588	586	566	566
query44	1210	718	724	718
query45	190	168	165	165
query46	1073	707	730	707
query47	1900	1784	1804	1784
query48	378	301	395	301
query49	847	405	424	405
query50	768	382	389	382
query51	6833	6839	6746	6746
query52	103	91	92	91
query53	360	288	292	288
query54	869	438	443	438
query55	76	72	68	68
query56	320	274	278	274
query57	1130	1061	1062	1061
query58	250	232	260	232
query59	3481	3511	3177	3177
query60	295	283	292	283
query61	94	89	95	89
query62	611	436	434	434
query63	324	281	291	281
query64	8796	2269	1724	1724
query65	3140	3147	3100	3100
query66	824	324	327	324
query67	15972	14989	15270	14989
query68	7491	568	542	542
query69	734	407	343	343
query70	1158	1054	1138	1054
query71	531	284	280	280
query72	8619	5373	2774	2774
query73	813	325	325	325
query74	5899	5471	5594	5471
query75	5081	2638	2705	2638
query76	4817	957	1048	957
query77	776	298	306	298
query78	9473	9096	8836	8836
query79	7586	518	516	516
query80	1035	467	478	467
query81	577	216	236	216
query82	769	110	109	109
query83	332	167	165	165
query84	277	86	90	86
query85	1362	314	297	297
query86	398	306	318	306
query87	3280	3060	3057	3057
query88	4624	2474	2443	2443
query89	485	378	381	378
query90	2010	185	189	185
query91	134	104	101	101
query92	63	50	48	48
query93	5904	508	508	508
query94	1268	209	206	206
query95	410	326	325	325
query96	607	267	274	267
query97	3188	3004	2987	2987
query98	218	211	194	194
query99	1163	877	846	846
Total cold run time: 298372 ms
Total hot run time: 170831 ms

@doris-robot
Copy link

doris-robot commented Jul 5, 2024

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f2cabc8462c9673ad860c37e3dc8eaad7033f604, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.51	0.47	0.49
query6	1.15	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.51	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.58	0.58
query14	0.76	0.78	0.78
query15	0.84	0.80	0.83
query16	0.35	0.37	0.36
query17	1.01	1.01	1.03
query18	0.22	0.26	0.23
query19	1.79	1.83	1.78
query20	0.02	0.01	0.01
query21	15.38	0.72	0.65
query22	3.83	7.47	2.05
query23	18.26	1.36	1.25
query24	2.07	0.24	0.22
query25	0.16	0.08	0.09
query26	0.30	0.21	0.20
query27	0.46	0.23	0.23
query28	13.51	1.02	1.00
query29	12.86	3.37	3.34
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.26	0.49	0.47
query33	2.89	2.86	2.93
query34	17.20	4.35	4.39
query35	4.40	4.38	4.48
query36	0.64	0.46	0.47
query37	0.18	0.15	0.15
query38	0.15	0.15	0.13
query39	0.05	0.03	0.03
query40	0.16	0.12	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.05
Total cold run time: 109.82 s
Total hot run time: 30.82 s

@felixwluo
Copy link
Contributor Author

felixwluo commented Jul 8, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-H: Total hot run time: 39678 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9c9434e4ff5f9c296f522ad95c7f755e80538e37, data reload: false

------ Round 1 ----------------------------------
q1	17614	4299	4315	4299
q2	2014	195	190	190
q3	10450	1201	1117	1117
q4	10192	748	751	748
q5	7476	2685	2607	2607
q6	222	137	134	134
q7	949	591	594	591
q8	9223	2050	2070	2050
q9	8902	6489	6483	6483
q10	9037	3719	3724	3719
q11	453	238	236	236
q12	506	227	223	223
q13	17819	2967	2986	2967
q14	277	230	230	230
q15	523	491	470	470
q16	515	379	373	373
q17	964	661	664	661
q18	7957	7565	7309	7309
q19	5853	1453	1451	1451
q20	658	326	340	326
q21	5024	3163	3824	3163
q22	384	331	331	331
Total cold run time: 117012 ms
Total hot run time: 39678 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4414	4254	4231	4231
q2	407	260	262	260
q3	2993	2845	2909	2845
q4	2041	1710	1723	1710
q5	5610	5514	5468	5468
q6	240	130	131	130
q7	2223	1886	1859	1859
q8	3222	3382	3387	3382
q9	8758	8651	8848	8651
q10	4066	3913	3736	3736
q11	582	498	487	487
q12	814	655	641	641
q13	16216	3165	3143	3143
q14	302	263	281	263
q15	528	496	487	487
q16	499	432	421	421
q17	1833	1542	1460	1460
q18	7978	7770	7872	7770
q19	2125	1623	1619	1619
q20	2244	1913	1861	1861
q21	10962	5062	4912	4912
q22	620	545	556	545
Total cold run time: 78677 ms
Total hot run time: 55881 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

TPC-DS: Total hot run time: 172591 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9c9434e4ff5f9c296f522ad95c7f755e80538e37, data reload: false

query1	933	364	357	357
query2	6453	2381	2344	2344
query3	6634	204	227	204
query4	28622	17289	17308	17289
query5	3711	495	480	480
query6	258	180	175	175
query7	4611	302	299	299
query8	316	286	291	286
query9	8525	2393	2355	2355
query10	564	305	299	299
query11	10566	9986	9811	9811
query12	117	93	88	88
query13	1675	374	376	374
query14	10246	7069	7777	7069
query15	243	181	187	181
query16	7666	315	294	294
query17	1801	533	524	524
query18	1892	273	275	273
query19	202	149	147	147
query20	88	80	79	79
query21	206	135	130	130
query22	4271	4152	4007	4007
query23	34070	33576	33553	33553
query24	11009	2912	2929	2912
query25	619	406	388	388
query26	1240	163	154	154
query27	2385	328	329	328
query28	6956	2154	2145	2145
query29	900	623	651	623
query30	246	149	154	149
query31	1006	758	763	758
query32	94	56	55	55
query33	766	318	291	291
query34	1015	500	497	497
query35	759	634	645	634
query36	1123	1014	1000	1000
query37	151	77	85	77
query38	2974	2853	2862	2853
query39	910	836	834	834
query40	209	128	135	128
query41	53	50	51	50
query42	111	98	106	98
query43	608	561	533	533
query44	1229	738	742	738
query45	191	162	161	161
query46	1081	735	714	714
query47	1858	1778	1777	1777
query48	380	308	311	308
query49	846	418	416	416
query50	770	384	392	384
query51	6837	6774	6831	6774
query52	112	92	98	92
query53	364	291	282	282
query54	871	448	451	448
query55	74	77	81	77
query56	287	271	282	271
query57	1131	1044	1055	1044
query58	270	257	250	250
query59	3513	3272	3190	3190
query60	309	270	283	270
query61	92	97	92	92
query62	624	446	462	446
query63	315	286	282	282
query64	9397	2178	1642	1642
query65	3154	3096	3088	3088
query66	720	329	325	325
query67	15228	14829	14748	14748
query68	4457	555	546	546
query69	650	357	345	345
query70	1205	1120	1148	1120
query71	378	281	279	279
query72	7379	5160	5486	5160
query73	749	327	329	327
query74	5881	5515	5414	5414
query75	3387	2702	2688	2688
query76	2459	977	886	886
query77	461	295	316	295
query78	9497	8886	8773	8773
query79	2770	518	513	513
query80	2077	469	468	468
query81	584	217	218	217
query82	920	104	108	104
query83	331	169	173	169
query84	281	86	91	86
query85	1786	310	294	294
query86	464	333	335	333
query87	3320	3110	3143	3110
query88	4377	2465	2443	2443
query89	480	393	380	380
query90	1785	192	195	192
query91	129	101	102	101
query92	62	48	48	48
query93	2993	527	531	527
query94	1127	228	220	220
query95	411	392	314	314
query96	601	277	270	270
query97	3194	2986	3031	2986
query98	225	204	198	198
query99	1299	845	841	841
Total cold run time: 282176 ms
Total hot run time: 172591 ms

@doris-robot
Copy link

doris-robot commented Jul 8, 2024

ClickBench: Total hot run time: 30.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9c9434e4ff5f9c296f522ad95c7f755e80538e37, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.10	0.08
query5	0.50	0.48	0.49
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.55	0.54	0.55
query11	0.15	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.59	0.58
query14	0.78	0.78	0.78
query15	0.85	0.81	0.82
query16	0.38	0.37	0.37
query17	0.94	1.03	1.04
query18	0.22	0.25	0.24
query19	1.89	1.76	1.78
query20	0.01	0.01	0.01
query21	15.40	0.76	0.65
query22	4.32	7.24	1.69
query23	18.31	1.33	1.22
query24	2.19	0.22	0.22
query25	0.16	0.08	0.09
query26	0.29	0.20	0.21
query27	0.44	0.23	0.23
query28	13.20	1.01	1.00
query29	12.61	3.31	3.30
query30	0.26	0.07	0.05
query31	2.88	0.37	0.38
query32	3.32	0.48	0.47
query33	2.87	2.90	2.90
query34	17.16	4.42	4.38
query35	4.41	4.43	4.42
query36	0.66	0.48	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.04	0.04
query40	0.16	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.98 s
Total hot run time: 30.52 s

yiguolei
yiguolei reviewed Jul 11, 2024
View reviewed changes
regression-test/suites/alter_p0/test_alter_add_multi_partition.groovy Outdated
"""
List<List<Object>> result1 = sql "show partitions from add_multi_partition;"
assertEquals(result1.size(), 1)
sql "ALTER TABLE add_multi_partition ADD PARTITIONS FROM ('2019-01-01') TO ('2025-01-01') INTERVAL 1 YEAR;"
Copy link
Contributor

@yiguolei yiguolei Jul 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the test is too simple.
Should test more complicated case.
For example, partition by day, then the partition interval is 5 day, to test if we could deal with Feb 29 or Aug 31.
And also test, if could support 3min

Copy link
Contributor Author

@felixwluo felixwluo Jul 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added 0229 partition. No problem

@felixwluo felixwluo dismissed stale reviews from xy720, cambyzju, and lide-reed via dbdf299 July 11, 2024 13:12
@felixwluo felixwluo force-pushed the add-alter-partition branch from 9c9434e to dbdf299 Compare July 11, 2024 13:12
@felixwluo
Copy link
Contributor Author

felixwluo commented Jul 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

TPC-DS: Total hot run time: 173221 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a765b634b823405d4761df5ac81bb1aa12aa77b1, data reload: false

query1	909	369	363	363
query2	6435	1852	1775	1775
query3	6663	208	220	208
query4	28439	17543	17118	17118
query5	3638	478	471	471
query6	282	192	169	169
query7	4586	288	299	288
query8	244	202	193	193
query9	8450	2443	2414	2414
query10	434	291	279	279
query11	10712	10063	10102	10063
query12	118	99	86	86
query13	1647	372	363	363
query14	10209	7873	7674	7674
query15	236	168	163	163
query16	7612	498	468	468
query17	1570	565	526	526
query18	1568	277	269	269
query19	198	149	149	149
query20	94	81	81	81
query21	204	130	132	130
query22	4430	4309	3869	3869
query23	34137	33895	33784	33784
query24	10736	2979	2828	2828
query25	603	403	401	401
query26	713	157	154	154
query27	2232	289	300	289
query28	5970	2098	2087	2087
query29	898	652	624	624
query30	251	152	151	151
query31	1010	733	786	733
query32	94	53	53	53
query33	645	331	326	326
query34	892	517	497	497
query35	895	773	776	773
query36	1152	962	979	962
query37	142	83	90	83
query38	3046	2894	2797	2797
query39	873	825	833	825
query40	198	129	117	117
query41	45	45	42	42
query42	123	101	100	100
query43	500	466	450	450
query44	1099	726	733	726
query45	195	163	165	163
query46	1093	729	739	729
query47	1855	1780	1780	1780
query48	372	295	298	295
query49	822	405	409	405
query50	779	389	392	389
query51	6833	6718	6714	6714
query52	108	91	94	91
query53	358	294	290	290
query54	925	454	455	454
query55	77	76	74	74
query56	288	262	271	262
query57	1119	1027	1071	1027
query58	268	243	262	243
query59	2810	2472	2613	2472
query60	303	264	303	264
query61	95	91	105	91
query62	793	631	635	631
query63	321	291	291	291
query64	9180	2217	1676	1676
query65	3184	3137	3164	3137
query66	741	332	324	324
query67	15632	14936	14760	14760
query68	6109	554	557	554
query69	650	465	389	389
query70	1241	1167	1080	1080
query71	461	280	327	280
query72	7157	5351	5763	5351
query73	810	326	338	326
query74	6159	5586	5654	5586
query75	3718	2696	2681	2681
query76	3868	982	946	946
query77	674	311	311	311
query78	9696	8931	9008	8931
query79	7436	531	528	528
query80	1445	478	490	478
query81	537	220	218	218
query82	820	141	134	134
query83	200	170	172	170
query84	278	87	89	87
query85	1267	311	304	304
query86	406	311	321	311
query87	3279	3068	3067	3067
query88	4320	2389	2403	2389
query89	516	396	393	393
query90	1841	216	199	199
query91	131	118	98	98
query92	61	50	53	50
query93	6003	525	513	513
query94	1096	294	294	294
query95	413	398	314	314
query96	619	269	277	269
query97	3210	3043	3045	3043
query98	212	206	190	190
query99	1603	1281	1254	1254
Total cold run time: 289319 ms
Total hot run time: 173221 ms

@doris-robot
Copy link

doris-robot commented Jul 23, 2024

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a765b634b823405d4761df5ac81bb1aa12aa77b1, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.09	0.08
query5	0.49	0.48	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.13
query13	0.60	0.59	0.59
query14	0.78	0.77	0.82
query15	0.85	0.81	0.81
query16	0.37	0.36	0.34
query17	1.00	0.97	0.99
query18	0.23	0.21	0.22
query19	1.84	1.68	1.83
query20	0.02	0.01	0.01
query21	15.42	0.78	0.66
query22	4.45	6.34	2.12
query23	18.27	1.47	1.29
query24	2.12	0.24	0.22
query25	0.15	0.09	0.10
query26	0.29	0.22	0.21
query27	0.46	0.24	0.23
query28	13.26	1.02	1.01
query29	12.62	3.30	3.27
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.28	0.48	0.47
query33	2.93	2.89	2.90
query34	17.30	4.30	4.37
query35	4.41	4.41	4.41
query36	0.65	0.46	0.47
query37	0.19	0.16	0.15
query38	0.15	0.15	0.16
query39	0.05	0.03	0.04
query40	0.16	0.12	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.25 s
Total hot run time: 30.76 s

yujun777
yujun777 approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

deardeng
deardeng approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring force-pushed the add-alter-partition branch from a765b63 to e247c1b Compare August 5, 2024 09:02
@dataroaring
Copy link
Contributor

dataroaring commented Aug 5, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 5, 2024
@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-H: Total hot run time: 42213 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e247c1b20dd5e77afa3dcdfabcbfb0d38772a06c, data reload: false

------ Round 1 ----------------------------------
q1	17813	4103	4042	4042
q2	2026	196	192	192
q3	11197	1373	1375	1373
q4	10157	835	942	835
q5	7650	2980	2969	2969
q6	219	138	141	138
q7	1055	623	631	623
q8	10698	1928	1899	1899
q9	8482	6611	6664	6611
q10	8791	3904	3870	3870
q11	423	251	246	246
q12	416	232	228	228
q13	17772	3007	3049	3007
q14	284	249	250	249
q15	516	493	498	493
q16	550	403	406	403
q17	991	921	920	920
q18	10113	7812	7600	7600
q19	2497	1175	1214	1175
q20	568	331	344	331
q21	5295	4735	4755	4735
q22	344	274	288	274
Total cold run time: 117857 ms
Total hot run time: 42213 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4077	4032	4037	4032
q2	335	224	219	219
q3	3133	3155	3138	3138
q4	1986	1962	1991	1962
q5	5459	5364	5461	5364
q6	225	132	132	132
q7	2177	1818	1796	1796
q8	3305	3364	3312	3312
q9	8560	8773	8565	8565
q10	3846	3950	4011	3950
q11	571	465	467	465
q12	780	618	600	600
q13	16595	3073	3163	3073
q14	310	279	266	266
q15	518	487	490	487
q16	456	395	402	395
q17	1736	1733	1727	1727
q18	8326	7787	7696	7696
q19	1765	1742	1722	1722
q20	2052	1828	1839	1828
q21	5705	5260	5398	5260
q22	521	476	457	457
Total cold run time: 72438 ms
Total hot run time: 56446 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

TPC-DS: Total hot run time: 167815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e247c1b20dd5e77afa3dcdfabcbfb0d38772a06c, data reload: false

query1	909	375	361	361
query2	6375	1662	1633	1633
query3	6663	215	221	215
query4	19688	17465	17245	17245
query5	3557	506	546	506
query6	265	174	177	174
query7	4599	289	284	284
query8	250	200	192	192
query9	8527	2375	2352	2352
query10	445	303	270	270
query11	10445	10031	9920	9920
query12	119	90	93	90
query13	1629	379	380	379
query14	9059	6844	5875	5875
query15	209	161	177	161
query16	6979	451	422	422
query17	949	614	536	536
query18	1904	278	278	278
query19	189	140	140	140
query20	92	84	87	84
query21	203	98	96	96
query22	4345	4162	3846	3846
query23	33878	33809	33498	33498
query24	9253	3117	3060	3060
query25	615	389	406	389
query26	717	148	147	147
query27	2310	280	279	279
query28	5832	2027	2009	2009
query29	908	425	452	425
query30	231	152	152	152
query31	924	768	783	768
query32	105	55	59	55
query33	635	310	320	310
query34	911	489	485	485
query35	868	788	774	774
query36	1087	880	892	880
query37	178	81	79	79
query38	2983	2885	2847	2847
query39	897	866	842	842
query40	231	110	108	108
query41	45	44	44	44
query42	119	104	100	100
query43	459	428	406	406
query44	1159	727	727	727
query45	206	176	177	176
query46	1102	795	785	785
query47	1819	1720	1757	1720
query48	362	290	289	289
query49	837	413	413	413
query50	890	430	418	418
query51	6785	6698	6646	6646
query52	101	88	91	88
query53	257	187	181	181
query54	635	440	454	440
query55	74	72	73	72
query56	287	255	257	255
query57	1111	1025	1061	1025
query58	275	267	264	264
query59	2537	2341	2464	2341
query60	303	288	289	288
query61	105	97	93	93
query62	865	667	650	650
query63	215	179	181	179
query64	4182	1893	1914	1893
query65	3138	3087	3079	3079
query66	841	325	323	323
query67	15380	14785	14747	14747
query68	5443	564	571	564
query69	708	373	298	298
query70	1145	1082	1078	1078
query71	460	274	275	274
query72	7590	2684	2468	2468
query73	914	321	321	321
query74	6104	5644	5619	5619
query75	3491	2737	2705	2705
query76	3147	1262	1290	1262
query77	561	302	302	302
query78	9467	8902	8864	8864
query79	1332	525	522	522
query80	803	563	494	494
query81	575	222	227	222
query82	295	127	134	127
query83	195	166	167	166
query84	266	79	82	79
query85	861	316	294	294
query86	376	283	296	283
query87	3269	3112	3060	3060
query88	3697	2386	2410	2386
query89	372	290	285	285
query90	1806	190	196	190
query91	129	101	100	100
query92	58	50	49	49
query93	1599	601	613	601
query94	799	288	288	288
query95	436	260	267	260
query96	601	285	276	276
query97	3182	3118	3092	3092
query98	236	207	200	200
query99	1656	1284	1278	1278
Total cold run time: 255625 ms
Total hot run time: 167815 ms

@doris-robot
Copy link

doris-robot commented Aug 5, 2024

ClickBench: Total hot run time: 29.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e247c1b20dd5e77afa3dcdfabcbfb0d38772a06c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.07
query5	0.51	0.50	0.49
query6	1.12	0.71	0.71
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.56	0.50	0.51
query10	0.56	0.56	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.61	0.60
query14	0.77	0.81	0.79
query15	0.89	0.87	0.86
query16	0.36	0.35	0.35
query17	1.00	1.01	0.97
query18	0.22	0.22	0.21
query19	1.82	1.74	1.70
query20	0.01	0.01	0.01
query21	15.40	0.76	0.66
query22	3.98	7.70	1.17
query23	18.05	1.31	1.28
query24	2.24	0.23	0.22
query25	0.18	0.08	0.08
query26	0.32	0.22	0.21
query27	0.46	0.23	0.23
query28	13.16	1.00	0.96
query29	12.56	3.27	3.27
query30	0.26	0.06	0.06
query31	2.86	0.40	0.40
query32	3.25	0.49	0.49
query33	2.92	2.92	2.96
query34	15.44	4.23	4.21
query35	4.28	4.29	4.34
query36	0.68	0.48	0.49
query37	0.19	0.17	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.04	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.63 s
Total hot run time: 29.76 s

dataroaring
dataroaring approved these changes Aug 14, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0dca9ef into apache:master Aug 14, 2024
dataroaring pushed a commit that referenced this pull request Aug 17, 2024
@felixwluo @dataroaring
[Enhancement](alter) support add partitions in batch (#37114)
0107e95 
## Proposed changes

Issue Number: close #32524
@felixwluo felixwluo mentioned this pull request Sep 2, 2024
Merged
6 tasks
morningman pushed a commit to apache/doris-website that referenced this pull request Sep 3, 2024
@felixwluo
[doc](alter) alter to add partitions in batches (#1077)
e559f9e 
# Versions 

- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0

# Languages

- [x] Chinese
- [x] English

Doris PR: apache/doris#37114
hechao-ustc pushed a commit to hechao-ustc/doris-website that referenced this pull request Sep 3, 2024
@felixwluo
[doc](alter) alter to add partitions in batches (apache#1077)
5d3bac4 
# Versions 

- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0

# Languages

- [x] Chinese
- [x] English

Doris PR: apache/doris#37114
Gabriel39 pushed a commit to Gabriel39/doris-website that referenced this pull request Sep 23, 2024
@felixwluo @Gabriel39
[doc](alter) alter to add partitions in batches (apache#1077)
1db54eb 
# Versions 

- [x] dev
- [x] 3.0
- [x] 2.1
- [x] 2.0

# Languages

- [x] Chinese
- [x] English

Doris PR: apache/doris#37114
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@cambyzju cambyzju cambyzju approved these changes

@dataroaring dataroaring dataroaring approved these changes

@lide-reed lide-reed lide-reed left review comments

@xy720 xy720 xy720 left review comments

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

@morningman morningman Awaiting requested review from morningman morningman is a code owner

+2 more reviewers

@deardeng deardeng deardeng approved these changes

@yujun777 yujun777 yujun777 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged doing reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Enhancement] support add partitions in batch

9 participants

@felixwluo @doris-robot @xy720 @dataroaring @lide-reed @deardeng @yujun777 @yiguolei @cambyzju