Skip to content

Conversation

@zclllyybb
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

before we forbid it in #33736, now it's time to let it go.

now we support:

mysql> create table auto_dynamic(
    ->             k0 datetime(6) NOT NULL
    ->         )
    ->         auto partition by range (date_trunc(k0, 'year'))
    ->         (
    ->         )
    ->         DISTRIBUTED BY HASH(`k0`) BUCKETS 2
    ->         properties(
    ->             "dynamic_partition.enable" = "true",
    ->             "dynamic_partition.prefix" = "p",
    ->             "dynamic_partition.start" = "-50",
    ->             "dynamic_partition.end" = "0",
    ->             "dynamic_partition.time_unit" = "year",
    ->             "replication_num" = "1"
    ->         );
Query OK, 0 rows affected (0.12 sec)

with this, the users could manage their data more flexible

we will refactor the docs of it, and the only recommended way it set end to zero.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38041 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 124660cda17d88504176259ce7e74e1d1d643675, data reload: false

------ Round 1 ----------------------------------
q1	18236	4453	4375	4375
q2	2615	210	214	210
q3	11036	1146	1002	1002
q4	10330	791	753	753
q5	7820	2849	2783	2783
q6	269	154	154	154
q7	1011	654	635	635
q8	9706	2070	2102	2070
q9	6980	6539	6546	6539
q10	7039	2218	2242	2218
q11	485	265	295	265
q12	429	259	263	259
q13	18778	2965	2999	2965
q14	294	268	263	263
q15	559	532	534	532
q16	518	423	409	409
q17	994	740	681	681
q18	7393	6850	6751	6751
q19	7405	1019	1146	1019
q20	810	358	360	358
q21	3768	3005	2771	2771
q22	1133	1029	1036	1029
Total cold run time: 117608 ms
Total hot run time: 38041 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4553	4316	4359	4316
q2	422	323	317	317
q3	2850	2648	2619	2619
q4	1888	1622	1613	1613
q5	5392	5394	5342	5342
q6	238	149	147	147
q7	2055	1704	1701	1701
q8	3205	3353	3363	3353
q9	8505	8417	8423	8417
q10	3421	3192	3155	3155
q11	600	539	532	532
q12	849	644	645	644
q13	16335	2985	3024	2985
q14	325	311	307	307
q15	573	527	526	526
q16	504	444	449	444
q17	1799	1504	1517	1504
q18	7917	7801	7346	7346
q19	4141	1591	1647	1591
q20	2078	1817	1786	1786
q21	13849	5191	5047	5047
q22	1101	1001	1027	1001
Total cold run time: 82600 ms
Total hot run time: 54693 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190853 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 124660cda17d88504176259ce7e74e1d1d643675, data reload: false

query1	987	399	389	389
query2	6746	2039	1925	1925
query3	6667	233	243	233
query4	33087	23343	23300	23300
query5	4378	729	708	708
query6	313	223	208	208
query7	4622	326	320	320
query8	491	451	430	430
query9	8620	2540	2483	2483
query10	481	333	312	312
query11	17754	15075	14990	14990
query12	185	125	123	123
query13	1681	439	441	439
query14	9797	7152	6840	6840
query15	264	190	189	189
query16	7920	497	497	497
query17	1599	603	597	597
query18	2120	340	336	336
query19	228	199	170	170
query20	143	138	141	138
query21	250	144	143	143
query22	4386	4350	4051	4051
query23	34178	33291	33229	33229
query24	11640	2886	2902	2886
query25	656	421	419	419
query26	1078	200	178	178
query27	2189	301	309	301
query28	5949	2141	2118	2118
query29	876	453	450	450
query30	322	186	184	184
query31	1066	845	838	838
query32	123	80	82	80
query33	822	343	338	338
query34	922	510	512	510
query35	898	749	786	749
query36	1143	947	982	947
query37	163	107	105	105
query38	3915	3886	3839	3839
query39	1557	1478	1464	1464
query40	238	160	155	155
query41	145	142	139	139
query42	143	116	119	116
query43	551	510	491	491
query44	1242	789	796	789
query45	229	195	200	195
query46	1120	781	750	750
query47	1933	1842	1844	1842
query48	419	345	342	342
query49	1250	595	591	591
query50	873	459	470	459
query51	6952	6789	6805	6789
query52	125	114	109	109
query53	303	229	234	229
query54	933	512	498	498
query55	92	90	90	90
query56	342	315	312	312
query57	1225	1122	1114	1114
query58	316	306	315	306
query59	3026	2788	2749	2749
query60	356	333	321	321
query61	153	151	150	150
query62	904	719	680	680
query63	263	227	227	227
query64	5408	2509	1959	1959
query65	3271	3170	3196	3170
query66	1168	687	696	687
query67	15614	14958	14905	14905
query68	8466	600	598	598
query69	717	433	344	344
query70	1253	1159	1185	1159
query71	569	323	321	321
query72	7593	2330	2104	2104
query73	1591	353	355	353
query74	9331	8904	8880	8880
query75	5138	2724	2719	2719
query76	5227	1048	1036	1036
query77	944	451	443	443
query78	9805	9635	9187	9187
query79	7795	565	558	558
query80	1091	620	622	620
query81	621	259	261	259
query82	329	159	161	159
query83	400	215	221	215
query84	290	98	98	98
query85	1015	373	368	368
query86	366	336	338	336
query87	4331	4192	4241	4192
query88	4353	2498	2491	2491
query89	500	336	326	326
query90	2015	240	237	237
query91	165	129	133	129
query92	90	80	79	79
query93	4932	565	555	555
query94	847	325	309	309
query95	401	298	302	298
query96	634	283	283	283
query97	3241	3078	3063	3063
query98	244	241	225	225
query99	1601	1289	1293	1289
Total cold run time: 314705 ms
Total hot run time: 190853 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 124660cda17d88504176259ce7e74e1d1d643675, data reload: false

query1	0.05	0.04	0.05
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.10
query5	0.50	0.50	0.50
query6	1.12	0.74	0.73
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.55	0.49	0.48
query10	0.54	0.55	0.56
query11	0.16	0.13	0.12
query12	0.16	0.13	0.14
query13	0.62	0.62	0.60
query14	0.76	0.77	0.78
query15	0.85	0.83	0.83
query16	0.37	0.37	0.37
query17	0.96	0.99	0.97
query18	0.21	0.21	0.21
query19	1.78	1.67	1.68
query20	0.01	0.01	0.02
query21	15.43	0.67	0.68
query22	4.55	6.10	2.37
query23	18.26	1.35	1.19
query24	2.05	0.23	0.23
query25	0.16	0.09	0.09
query26	0.27	0.19	0.19
query27	0.08	0.08	0.07
query28	13.31	1.02	1.00
query29	12.65	3.35	3.29
query30	0.44	0.25	0.24
query31	2.79	0.41	0.40
query32	3.25	0.49	0.49
query33	2.97	2.98	2.99
query34	16.83	4.31	4.44
query35	4.43	4.43	4.42
query36	0.67	0.50	0.50
query37	0.22	0.19	0.17
query38	0.18	0.16	0.16
query39	0.06	0.05	0.05
query40	0.18	0.15	0.15
query41	0.12	0.07	0.06
query42	0.08	0.07	0.07
query43	0.07	0.06	0.06
Total cold run time: 109.75 s
Total hot run time: 31.39 s

@zclllyybb
Copy link
Contributor Author

run buildall

@zclllyybb
Copy link
Contributor Author

wait #39695 merge firstly

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38134 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d1c885e995da2f1c97c8c5a45820ea35237eea47, data reload: false

------ Round 1 ----------------------------------
q1	18016	4463	4389	4389
q2	2918	189	183	183
q3	11263	1124	1040	1040
q4	10694	709	665	665
q5	8057	2877	2835	2835
q6	227	140	144	140
q7	972	615	597	597
q8	9322	2041	2047	2041
q9	7029	6527	6465	6465
q10	7014	2198	2231	2198
q11	470	256	256	256
q12	391	220	224	220
q13	18606	3090	3064	3064
q14	285	230	251	230
q15	526	502	489	489
q16	487	396	403	396
q17	974	650	721	650
q18	7339	6915	6991	6915
q19	1385	1125	1076	1076
q20	676	333	333	333
q21	4538	3068	2933	2933
q22	1149	1019	1046	1019
Total cold run time: 112338 ms
Total hot run time: 38134 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4329	4280	4236	4236
q2	386	282	280	280
q3	2899	2667	2633	2633
q4	1921	1656	1635	1635
q5	5369	5382	5372	5372
q6	216	133	135	133
q7	2121	1726	1694	1694
q8	3196	3352	3337	3337
q9	8460	8458	8444	8444
q10	3432	3177	3155	3155
q11	595	510	510	510
q12	798	610	604	604
q13	10989	3045	3061	3045
q14	314	272	298	272
q15	520	479	487	479
q16	500	443	438	438
q17	1793	1492	1485	1485
q18	7729	7410	7342	7342
q19	1647	1571	1522	1522
q20	2119	1807	1845	1807
q21	5437	5299	5365	5299
q22	1117	1054	1046	1046
Total cold run time: 65887 ms
Total hot run time: 54768 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186771 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d1c885e995da2f1c97c8c5a45820ea35237eea47, data reload: false

query1	901	370	366	366
query2	6454	1954	1919	1919
query3	6660	205	215	205
query4	28005	23275	23139	23139
query5	4124	485	480	480
query6	254	158	162	158
query7	4593	291	286	286
query8	259	211	216	211
query9	8627	2489	2441	2441
query10	450	275	267	267
query11	15809	15196	15019	15019
query12	159	104	101	101
query13	1641	371	378	371
query14	9966	7284	7238	7238
query15	270	167	175	167
query16	7987	425	460	425
query17	1570	568	542	542
query18	2118	280	287	280
query19	283	147	142	142
query20	122	109	105	105
query21	228	99	100	99
query22	4348	4093	4041	4041
query23	34117	33448	33635	33448
query24	11171	2877	2879	2877
query25	631	394	402	394
query26	1165	155	163	155
query27	2584	284	277	277
query28	7420	2041	2022	2022
query29	809	415	404	404
query30	305	154	148	148
query31	985	743	740	740
query32	96	57	65	57
query33	760	289	288	288
query34	973	475	488	475
query35	884	726	728	726
query36	1079	906	944	906
query37	154	86	81	81
query38	3956	3803	3852	3803
query39	1421	1400	1424	1400
query40	198	115	113	113
query41	51	50	48	48
query42	115	100	98	98
query43	523	465	468	465
query44	1199	749	762	749
query45	201	173	172	172
query46	1101	769	706	706
query47	1907	1781	1793	1781
query48	383	290	287	287
query49	1067	414	421	414
query50	802	414	405	405
query51	7300	7028	7110	7028
query52	97	89	89	89
query53	256	178	181	178
query54	917	456	467	456
query55	77	77	77	77
query56	272	254	263	254
query57	1184	1070	1105	1070
query58	235	218	234	218
query59	2920	2804	2777	2777
query60	297	261	262	261
query61	99	103	103	103
query62	847	673	641	641
query63	222	189	179	179
query64	5926	2239	1728	1728
query65	3238	3186	3162	3162
query66	1333	337	334	334
query67	15602	15125	15238	15125
query68	4542	568	549	549
query69	398	275	281	275
query70	1193	1137	1116	1116
query71	341	287	280	280
query72	6497	2364	1957	1957
query73	754	322	321	321
query74	9214	8785	8762	8762
query75	3399	2699	2738	2699
query76	2409	1009	999	999
query77	475	313	300	300
query78	10006	9073	9427	9073
query79	1035	530	527	527
query80	711	503	509	503
query81	491	235	233	233
query82	234	132	146	132
query83	173	157	156	156
query84	242	89	87	87
query85	930	368	279	279
query86	294	310	297	297
query87	4331	4307	4243	4243
query88	3064	2346	2536	2346
query89	381	299	282	282
query90	1731	200	194	194
query91	123	99	97	97
query92	63	54	54	54
query93	1031	532	533	532
query94	596	297	288	288
query95	349	262	257	257
query96	587	269	269	269
query97	3198	3088	3062	3062
query98	223	215	212	212
query99	1450	1282	1269	1269
Total cold run time: 282172 ms
Total hot run time: 186771 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d1c885e995da2f1c97c8c5a45820ea35237eea47, data reload: false

query1	0.05	0.05	0.04
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.68	0.08	0.07
query5	0.52	0.50	0.49
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.48
query10	0.55	0.55	0.54
query11	0.14	0.11	0.12
query12	0.14	0.11	0.13
query13	0.64	0.60	0.59
query14	0.79	0.81	0.77
query15	0.84	0.83	0.82
query16	0.37	0.38	0.39
query17	1.08	1.01	1.04
query18	0.24	0.21	0.20
query19	1.86	1.73	1.76
query20	0.01	0.01	0.00
query21	15.40	0.68	0.68
query22	4.97	6.23	2.38
query23	18.29	1.42	1.32
query24	2.12	0.22	0.22
query25	0.15	0.09	0.08
query26	0.26	0.17	0.18
query27	0.08	0.08	0.08
query28	13.24	1.03	0.99
query29	12.58	3.35	3.34
query30	0.24	0.05	0.05
query31	2.89	0.40	0.38
query32	3.27	0.48	0.47
query33	3.00	3.05	3.00
query34	17.18	4.45	4.45
query35	4.53	4.51	4.57
query36	0.65	0.48	0.47
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.04	0.04
query42	0.05	0.06	0.05
query43	0.05	0.05	0.05
Total cold run time: 110.55 s
Total hot run time: 31.41 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 22, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhangstar333 zhangstar333 merged commit 8a8a3ad into apache:master Aug 27, 2024
@zclllyybb zclllyybb deleted the enable_both branch September 11, 2024 02:17
zclllyybb added a commit to zclllyybb/doris that referenced this pull request Sep 11, 2024
…e same time (apache#39580)

before we forbid it in apache#33736, now
it's time to let it go.

now we support:
```sql
mysql> create table auto_dynamic(
    ->             k0 datetime(6) NOT NULL
    ->         )
    ->         auto partition by range (date_trunc(k0, 'year'))
    ->         (
    ->         )
    ->         DISTRIBUTED BY HASH(`k0`) BUCKETS 2
    ->         properties(
    ->             "dynamic_partition.enable" = "true",
    ->             "dynamic_partition.prefix" = "p",
    ->             "dynamic_partition.start" = "-50",
    ->             "dynamic_partition.end" = "0",
    ->             "dynamic_partition.time_unit" = "year",
    ->             "replication_num" = "1"
    ->         );
Query OK, 0 rows affected (0.12 sec)
```

with this, the users could manage their data more flexible

we will refactor the docs of it, and the only recommended way it set end
to zero.
yiguolei pushed a commit that referenced this pull request Sep 11, 2024
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
…e same time (#39580)

## Proposed changes
before we forbid it in #33736, now
it's time to let it go.

now we support:
```sql
mysql> create table auto_dynamic(
    ->             k0 datetime(6) NOT NULL
    ->         )
    ->         auto partition by range (date_trunc(k0, 'year'))
    ->         (
    ->         )
    ->         DISTRIBUTED BY HASH(`k0`) BUCKETS 2
    ->         properties(
    ->             "dynamic_partition.enable" = "true",
    ->             "dynamic_partition.prefix" = "p",
    ->             "dynamic_partition.start" = "-50",
    ->             "dynamic_partition.end" = "0",
    ->             "dynamic_partition.time_unit" = "year",
    ->             "replication_num" = "1"
    ->         );
Query OK, 0 rows affected (0.12 sec)
```

with this, the users could manage their data more flexible

we will refactor the docs of it, and the only recommended way it set end
to zero.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants