[Enhancement] (doris-future) Support auto partition name function #34258

wyxxxcat · 2024-04-28T12:34:06Z

Proposed changes

Issue Number: #34130

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-28T12:34:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_string.h

...main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/AutoPartitionName.java

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_string.h

wyxxxcat · 2024-05-01T15:38:58Z

run buildall

be/src/vec/functions/function_string.h

regression-test/suites/nereids_function_p0/scalar_function/A.groovy

be/src/vec/functions/function_string.h

doris-robot · 2024-05-01T16:31:04Z

TeamCity be ut coverage result:
Function Coverage: 35.68% (8959/25108)
Line Coverage: 27.27% (73887/270966)
Region Coverage: 26.45% (38173/144295)
Branch Coverage: 23.21% (19445/83782)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0a1e4b52d276a73d138ddd0592f4e4cee812de5f_0a1e4b52d276a73d138ddd0592f4e4cee812de5f/report/index.html

be/src/vec/functions/function_string.h

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_string.h

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_string.h

github-actions

clang-tidy made some suggestions

be/src/vec/functions/function_string.h

wyxxxcat · 2024-08-02T08:28:53Z

run buildall

doris-robot · 2024-08-02T08:49:05Z

TPC-H: Total hot run time: 41815 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf84cde879415e423d5c751c63f63392c320ec31, data reload: false

------ Round 1 ----------------------------------
q1	18027	4243	4158	4158
q2	2412	209	204	204
q3	11658	1347	1387	1347
q4	10848	834	937	834
q5	8214	2996	2993	2993
q6	221	136	136	136
q7	1061	617	602	602
q8	9432	1844	1928	1844
q9	8424	6556	6611	6556
q10	8676	3843	3821	3821
q11	420	245	249	245
q12	412	221	226	221
q13	17761	2937	2948	2937
q14	276	245	239	239
q15	523	498	498	498
q16	482	417	388	388
q17	955	921	935	921
q18	7970	7371	7302	7302
q19	1379	1211	1210	1210
q20	582	329	341	329
q21	5238	4741	4784	4741
q22	350	297	289	289
Total cold run time: 115321 ms
Total hot run time: 41815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4048	3997	4012	3997
q2	323	225	219	219
q3	3019	3008	2990	2990
q4	1892	1877	1848	1848
q5	5281	5221	5252	5221
q6	219	127	129	127
q7	2045	1652	1685	1652
q8	3192	3238	3248	3238
q9	8321	8251	8254	8251
q10	3717	3858	3849	3849
q11	539	437	443	437
q12	709	550	556	550
q13	13421	2953	2946	2946
q14	308	260	262	260
q15	518	485	477	477
q16	440	390	392	390
q17	1712	1693	1688	1688
q18	7652	7333	7186	7186
q19	1674	1658	1668	1658
q20	1966	1735	1758	1735
q21	5420	5197	5152	5152
q22	498	442	453	442
Total cold run time: 66914 ms
Total hot run time: 54313 ms

doris-robot · 2024-08-02T08:59:41Z

TPC-DS: Total hot run time: 167228 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf84cde879415e423d5c751c63f63392c320ec31, data reload: false

query1	925	377	371	371
query2	6501	1644	1663	1644
query3	6687	210	228	210
query4	20431	17335	17192	17192
query5	4341	527	517	517
query6	300	175	170	170
query7	4601	304	298	298
query8	260	193	204	193
query9	8548	2347	2345	2345
query10	435	292	268	268
query11	10472	10009	10150	10009
query12	133	92	87	87
query13	1630	392	381	381
query14	9378	6011	7351	6011
query15	209	167	161	161
query16	6796	429	444	429
query17	959	577	555	555
query18	1265	287	290	287
query19	196	151	148	148
query20	99	90	92	90
query21	207	107	112	107
query22	4143	3992	4012	3992
query23	33846	33153	32911	32911
query24	10444	3072	3036	3036
query25	655	372	387	372
query26	1723	148	149	148
query27	2647	273	278	273
query28	6637	1960	1953	1953
query29	1073	418	406	406
query30	292	153	151	151
query31	929	747	759	747
query32	102	55	54	54
query33	687	310	318	310
query34	908	475	480	475
query35	855	717	730	717
query36	1015	884	884	884
query37	242	78	77	77
query38	2888	2770	2763	2763
query39	864	808	803	803
query40	282	112	112	112
query41	47	46	44	44
query42	130	99	101	99
query43	465	406	407	406
query44	1195	740	730	730
query45	202	180	177	177
query46	1091	801	766	766
query47	1784	1716	1718	1716
query48	372	299	300	299
query49	1192	434	407	407
query50	899	442	435	435
query51	6713	6711	6611	6611
query52	107	91	98	91
query53	252	182	175	175
query54	642	477	454	454
query55	74	76	76	76
query56	274	260	249	249
query57	1128	1017	1012	1012
query58	295	271	265	265
query59	2526	2354	2314	2314
query60	290	271	274	271
query61	100	97	93	93
query62	928	646	661	646
query63	218	180	187	180
query64	5872	1932	1859	1859
query65	3157	3087	3083	3083
query66	1332	330	332	330
query67	15288	14756	14717	14717
query68	4364	551	579	551
query69	450	303	292	292
query70	1079	1017	1030	1017
query71	412	277	284	277
query72	7177	2653	2477	2477
query73	786	332	327	327
query74	5985	5709	5656	5656
query75	3383	2702	2686	2686
query76	2524	1199	1265	1199
query77	453	359	313	313
query78	9469	8964	8856	8856
query79	1463	529	546	529
query80	1196	501	502	501
query81	559	225	224	224
query82	1060	127	129	127
query83	238	170	174	170
query84	267	80	78	78
query85	1276	317	300	300
query86	394	297	286	286
query87	3314	3113	3111	3111
query88	3018	2521	2526	2521
query89	376	294	286	286
query90	1821	193	192	192
query91	125	98	100	98
query92	62	50	50	50
query93	1422	605	615	605
query94	881	278	295	278
query95	369	260	262	260
query96	609	284	283	283
query97	3214	3074	3088	3074
query98	211	201	204	201
query99	1636	1294	1241	1241
Total cold run time: 260771 ms
Total hot run time: 167228 ms

doris-robot · 2024-08-02T09:05:10Z

ClickBench: Total hot run time: 30.15 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf84cde879415e423d5c751c63f63392c320ec31, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.49	0.48	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.57	0.50	0.51
query10	0.56	0.57	0.55
query11	0.16	0.12	0.12
query12	0.15	0.11	0.12
query13	0.62	0.61	0.60
query14	0.77	0.79	0.80
query15	0.93	0.86	0.87
query16	0.35	0.35	0.35
query17	1.02	0.98	1.00
query18	0.21	0.20	0.21
query19	1.87	1.79	1.73
query20	0.01	0.02	0.01
query21	15.40	0.75	0.66
query22	4.13	7.14	1.45
query23	18.02	1.44	1.36
query24	2.28	0.22	0.22
query25	0.19	0.08	0.09
query26	0.31	0.21	0.22
query27	0.45	0.23	0.24
query28	13.16	1.01	0.97
query29	12.64	3.28	3.29
query30	0.26	0.06	0.05
query31	2.86	0.41	0.41
query32	3.24	0.49	0.48
query33	2.89	2.97	2.90
query34	15.45	4.23	4.21
query35	4.27	4.32	4.30
query36	0.68	0.50	0.47
query37	0.19	0.17	0.16
query38	0.17	0.14	0.16
query39	0.04	0.04	0.04
query40	0.15	0.12	0.14
query41	0.10	0.06	0.06
query42	0.05	0.05	0.06
query43	0.05	0.05	0.04
Total cold run time: 107.9 s
Total hot run time: 30.15 s

zclllyybb

the last things:

be/src/vec/functions/function_string.h

...main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/AutoPartitionName.java

be/src/vec/functions/function_string.h

wyxxxcat · 2024-08-04T15:56:49Z

run buildall

doris-robot · 2024-08-04T16:36:51Z

TPC-H: Total hot run time: 41970 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 17b99e13439c555bdb74822bbd8fb2b27b405285, data reload: false

------ Round 1 ----------------------------------
q1	18685	4129	4151	4129
q2	2016	198	205	198
q3	10435	1317	1332	1317
q4	10285	831	940	831
q5	8598	2888	2981	2888
q6	220	141	139	139
q7	1054	635	623	623
q8	9432	1915	1939	1915
q9	9067	6649	6637	6637
q10	9060	4087	4050	4050
q11	1512	273	269	269
q12	576	240	242	240
q13	18317	3021	3021	3021
q14	268	247	249	247
q15	524	489	486	486
q16	536	396	413	396
q17	975	909	936	909
q18	9018	7352	7208	7208
q19	2761	1146	1203	1146
q20	538	319	351	319
q21	5264	4729	4753	4729
q22	345	284	273	273
Total cold run time: 119486 ms
Total hot run time: 41970 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4123	4047	4079	4047
q2	341	242	234	234
q3	3119	3193	3140	3140
q4	2022	2045	2001	2001
q5	5532	5376	5481	5376
q6	224	134	144	134
q7	2190	1772	1796	1772
q8	3316	3353	3361	3353
q9	8653	8799	8634	8634
q10	3893	3984	4078	3984
q11	573	463	458	458
q12	745	628	598	598
q13	16566	3138	3176	3138
q14	295	278	270	270
q15	520	487	488	487
q16	465	417	428	417
q17	1730	1706	1739	1706
q18	8357	7755	7667	7667
q19	1711	1711	1717	1711
q20	2141	1823	1858	1823
q21	5851	5330	5451	5330
q22	528	502	465	465
Total cold run time: 72895 ms
Total hot run time: 56745 ms

doris-robot · 2024-08-04T16:47:30Z

TPC-DS: Total hot run time: 169633 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 17b99e13439c555bdb74822bbd8fb2b27b405285, data reload: false

query1	916	372	364	364
query2	6410	1664	1603	1603
query3	6677	209	225	209
query4	20056	17463	17381	17381
query5	3674	521	508	508
query6	298	181	162	162
query7	4618	311	293	293
query8	261	196	198	196
query9	8534	2357	2358	2357
query10	464	286	269	269
query11	10648	10076	9956	9956
query12	121	97	89	89
query13	1637	397	378	378
query14	9864	6878	6972	6878
query15	208	170	182	170
query16	6996	447	435	435
query17	956	583	601	583
query18	1932	293	307	293
query19	195	152	151	151
query20	95	86	92	86
query21	212	104	97	97
query22	4147	4054	4144	4054
query23	33709	33518	33519	33518
query24	9268	3123	3128	3123
query25	687	413	442	413
query26	1335	151	156	151
query27	2937	283	280	280
query28	7422	2002	1973	1973
query29	1017	448	442	442
query30	241	163	154	154
query31	960	763	798	763
query32	105	57	77	57
query33	670	321	334	321
query34	911	522	515	515
query35	846	742	767	742
query36	1065	916	890	890
query37	167	84	81	81
query38	3040	2858	2837	2837
query39	918	873	842	842
query40	268	111	110	110
query41	46	43	44	43
query42	116	105	99	99
query43	480	412	428	412
query44	1207	764	739	739
query45	212	178	177	177
query46	1083	804	799	799
query47	1850	1697	1699	1697
query48	368	298	306	298
query49	863	417	426	417
query50	896	438	446	438
query51	6755	6672	6540	6540
query52	103	93	91	91
query53	258	185	190	185
query54	643	457	468	457
query55	80	75	76	75
query56	278	264	254	254
query57	1121	1044	1040	1040
query58	261	272	267	267
query59	2501	2404	2315	2315
query60	329	279	276	276
query61	96	103	94	94
query62	899	683	664	664
query63	217	194	184	184
query64	4994	1915	1885	1885
query65	3165	3094	3092	3092
query66	999	329	334	329
query67	15286	14851	14697	14697
query68	4350	595	580	580
query69	603	439	303	303
query70	1086	1117	1068	1068
query71	380	286	287	286
query72	7140	2649	2530	2530
query73	775	335	332	332
query74	6054	5677	5618	5618
query75	3355	2710	2742	2710
query76	2163	1236	1294	1236
query77	519	306	303	303
query78	9417	8904	8965	8904
query79	2683	529	541	529
query80	967	517	514	514
query81	552	224	228	224
query82	1021	131	133	131
query83	260	172	173	172
query84	255	80	84	80
query85	1169	326	301	301
query86	442	305	291	291
query87	3265	3102	3123	3102
query88	3956	2515	2510	2510
query89	393	286	289	286
query90	1667	198	202	198
query91	199	104	101	101
query92	62	54	50	50
query93	1946	619	614	614
query94	798	309	298	298
query95	384	274	274	274
query96	605	284	286	284
query97	3206	3020	3094	3020
query98	228	203	202	202
query99	1647	1296	1294	1294
Total cold run time: 261138 ms
Total hot run time: 169633 ms

doris-robot · 2024-08-04T16:52:59Z

ClickBench: Total hot run time: 30.68 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 17b99e13439c555bdb74822bbd8fb2b27b405285, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.06
query5	0.49	0.49	0.49
query6	1.14	0.71	0.71
query7	0.02	0.01	0.02
query8	0.05	0.03	0.04
query9	0.57	0.53	0.51
query10	0.57	0.57	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.61	0.60
query14	0.77	0.80	0.82
query15	0.90	0.87	0.86
query16	0.35	0.35	0.35
query17	1.00	1.01	1.07
query18	0.22	0.21	0.21
query19	1.94	1.75	1.79
query20	0.02	0.01	0.01
query21	15.39	0.76	0.66
query22	4.15	6.53	1.85
query23	18.01	1.34	1.35
query24	2.27	0.22	0.22
query25	0.19	0.08	0.07
query26	0.32	0.22	0.21
query27	0.46	0.23	0.22
query28	13.20	0.99	0.97
query29	12.54	3.31	3.27
query30	0.25	0.06	0.05
query31	2.88	0.41	0.40
query32	3.24	0.49	0.49
query33	2.94	2.99	2.96
query34	15.42	4.27	4.28
query35	4.32	4.31	4.31
query36	0.67	0.47	0.48
query37	0.19	0.17	0.17
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.15	0.13	0.13
query41	0.10	0.05	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.98 s
Total hot run time: 30.68 s

zclllyybb

LGTM

github-actions · 2024-08-05T02:27:28Z

PR approved by anyone and no changes requested.

HappenLee

LGTM

github-actions · 2024-08-05T02:37:49Z

PR approved by at least one committer and no changes requested.

) cherry-pick #34258 to branch-2.1

apache/doris#34258

…4258) ## Proposed changes Issue Number: #34130

apache/doris#34258

wyxxxcat force-pushed the getPartitionName branch 3 times, most recently from fb611dd to 854bf3f Compare April 28, 2024 12:38

github-actions bot reviewed Apr 28, 2024

View reviewed changes

be/src/vec/functions/function_string.h Show resolved Hide resolved

be/src/vec/functions/function_string.h Show resolved Hide resolved

zclllyybb reviewed Apr 28, 2024

View reviewed changes

be/src/vec/functions/function_string.h Outdated Show resolved Hide resolved

zclllyybb reviewed Apr 28, 2024

View reviewed changes

be/src/vec/functions/function_string.h Outdated Show resolved Hide resolved

zclllyybb reviewed Apr 28, 2024

View reviewed changes

...main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/AutoPartitionName.java Outdated Show resolved Hide resolved

wyxxxcat force-pushed the getPartitionName branch from 6bda52e to 3aba850 Compare May 1, 2024 15:17

wyxxxcat marked this pull request as ready for review May 1, 2024 15:19

github-actions bot reviewed May 1, 2024

View reviewed changes

be/src/vec/functions/function_string.h Show resolved Hide resolved

wyxxxcat force-pushed the getPartitionName branch 3 times, most recently from 3aee35d to 0a1e4b5 Compare May 1, 2024 15:29

zclllyybb reviewed May 1, 2024

View reviewed changes

zclllyybb requested changes May 1, 2024

View reviewed changes

zclllyybb reviewed May 5, 2024

View reviewed changes

be/src/vec/functions/function_string.h Outdated Show resolved Hide resolved

wyxxxcat force-pushed the getPartitionName branch from 0a1e4b5 to cbab7e9 Compare May 5, 2024 14:04

github-actions bot reviewed May 5, 2024

View reviewed changes

be/src/vec/functions/function_string.h Show resolved Hide resolved

wyxxxcat force-pushed the getPartitionName branch 3 times, most recently from e6bf600 to e7d0e94 Compare May 5, 2024 14:30

github-actions bot reviewed May 5, 2024

View reviewed changes

be/src/vec/functions/function_string.h Show resolved Hide resolved

wyxxxcat force-pushed the getPartitionName branch from e7d0e94 to c594735 Compare May 6, 2024 15:07

github-actions bot reviewed May 6, 2024

View reviewed changes

be/src/vec/functions/function_string.h Show resolved Hide resolved

wyxxxcat force-pushed the getPartitionName branch 3 times, most recently from db7e98c to 091fcd6 Compare May 8, 2024 15:57

wyxxxcat force-pushed the getPartitionName branch from db056b5 to cf84cde Compare August 2, 2024 08:27

wyxxxcat mentioned this pull request Aug 4, 2024

[docs](function) support auto_partition_name function apache/doris-website#945

Merged

zclllyybb requested changes Aug 4, 2024

View reviewed changes

fix

17b99e1

wyxxxcat force-pushed the getPartitionName branch from cf84cde to 17b99e1 Compare August 4, 2024 15:56

Mryange approved these changes Aug 5, 2024

View reviewed changes

zclllyybb approved these changes Aug 5, 2024

View reviewed changes

github-actions bot added the reviewed label Aug 5, 2024

HappenLee approved these changes Aug 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2024

zhangstar333 merged commit d848afb into apache:master Aug 5, 2024

wyxxxcat mentioned this pull request Aug 5, 2024

[branch-2.1] (doris-future) Support auto partition name function #38853

Merged

yiguolei pushed a commit that referenced this pull request Aug 5, 2024

[branch-2.1] (doris-future) Support auto partition name function (#38853

65154f8

) cherry-pick #34258 to branch-2.1

wm1581066 added the dev/2.1.6-merged label Aug 5, 2024

luzhijing pushed a commit to apache/doris-website that referenced this pull request Aug 5, 2024

[docs](function) support auto_partition_name function (#945)

64dc084

apache/doris#34258

zclllyybb mentioned this pull request Aug 6, 2024

[Enhancement] (doris-future) Support auto partition name function #34130

Closed

3 tasks

dataroaring pushed a commit that referenced this pull request Aug 11, 2024

[Enhancement] (doris-future) Support auto partition name function (#3…

53d42c6

…4258) ## Proposed changes Issue Number: #34130

dataroaring added the dev/3.0.2-merged label Aug 12, 2024

dataroaring pushed a commit that referenced this pull request Aug 16, 2024

[Enhancement] (doris-future) Support auto partition name function (#3…

64f4179

…4258) ## Proposed changes Issue Number: #34130

Gabriel39 pushed a commit to Gabriel39/doris-website that referenced this pull request Sep 23, 2024

[docs](function) support auto_partition_name function (apache#945)

038d674

apache/doris#34258

[Enhancement] (doris-future) Support auto partition name function #34258

[Enhancement] (doris-future) Support auto partition name function #34258

Uh oh!

Conversation

wyxxxcat commented Apr 28, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Apr 28, 2024

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wyxxxcat commented May 1, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

doris-robot commented May 1, 2024

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wyxxxcat commented Aug 2, 2024

Uh oh!

doris-robot commented Aug 2, 2024

Uh oh!

doris-robot commented Aug 2, 2024

Uh oh!

doris-robot commented Aug 2, 2024

Uh oh!

zclllyybb left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wyxxxcat commented Aug 4, 2024

Uh oh!

doris-robot commented Aug 4, 2024

Uh oh!

doris-robot commented Aug 4, 2024

Uh oh!

doris-robot commented Aug 4, 2024

Uh oh!

zclllyybb left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 5, 2024

Uh oh!

HappenLee left a comment

Choose a reason for hiding this comment

Uh oh!