[improvement](hudi)Obtain partition information through HMS's API for 2.0

[wuwenchi]

Proposed changes

mirror #30962, add useHiveSyncPartition for #31008

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[doris-robot]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

[wuwenchi]

run buildall

[doris-robot]

TPC-H: Total hot run time: 50024 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85836d9fe98361450af872736a638e2bb4e83917, data reload: false

------ Round 1 ----------------------------------
q1	17907	4467	4367	4367
q2	2026	150	144	144
q3	10344	1914	1919	1914
q4	10113	1244	1309	1244
q5	8423	3931	3958	3931
q6	233	121	123	121
q7	2018	1588	1618	1588
q8	9322	2735	2714	2714
q9	10956	10439	10558	10439
q10	8612	3512	3543	3512
q11	419	234	247	234
q12	464	297	311	297
q13	18375	3970	4062	3970
q14	357	333	319	319
q15	522	451	455	451
q16	705	596	582	582
q17	1127	946	957	946
q18	7177	6842	6970	6842
q19	1695	1601	1578	1578
q20	551	310	312	310
q21	4458	4136	4128	4128
q22	509	399	393	393
Total cold run time: 116313 ms
Total hot run time: 50024 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4304	4284	4298	4284
q2	322	221	206	206
q3	4134	4101	4123	4101
q4	2734	2731	2746	2731
q5	7243	7164	7191	7164
q6	232	117	117	117
q7	3249	2829	2816	2816
q8	4366	4425	4472	4425
q9	17074	16953	16952	16952
q10	4206	4240	4231	4231
q11	775	672	682	672
q12	1019	855	856	855
q13	7247	3770	3755	3755
q14	453	433	418	418
q15	500	451	447	447
q16	754	704	688	688
q17	3779	3813	3867	3813
q18	8726	8796	8757	8757
q19	1696	1711	1651	1651
q20	2375	2169	2115	2115
q21	8412	8397	8497	8397
q22	1053	964	921	921
Total cold run time: 84653 ms
Total hot run time: 79516 ms

[doris-robot]

TPC-DS: Total hot run time: 235469 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85836d9fe98361450af872736a638e2bb4e83917, data reload: false

query1	928	392	383	383
query2	6521	2197	2146	2146
query3	6912	204	208	204
query4	20731	17816	17839	17816
query5	19726	6463	6447	6447
query6	285	216	232	216
query7	4150	294	301	294
query8	266	258	249	249
query9	3098	2679	2638	2638
query10	416	289	301	289
query11	11378	10928	10976	10928
query12	121	78	69	69
query13	5585	620	624	620
query14	17707	12917	13266	12917
query15	385	235	226	226
query16	6420	255	261	255
query17	1728	1461	848	848
query18	2330	401	397	397
query19	203	144	142	142
query20	75	79	70	70
query21	189	99	95	95
query22	5295	5094	5058	5058
query23	32349	31781	31995	31781
query24	6893	6416	6457	6416
query25	539	429	400	400
query26	521	158	153	153
query27	1829	288	288	288
query28	6117	2209	2173	2173
query29	2847	2795	2861	2795
query30	234	155	160	155
query31	891	702	732	702
query32	63	59	61	59
query33	381	240	248	240
query34	830	457	475	457
query35	1116	944	892	892
query36	1482	1418	1379	1379
query37	87	62	56	56
query38	3101	2908	2897	2897
query39	1361	1313	1315	1313
query40	194	89	93	89
query41	33	31	34	31
query42	81	77	85	77
query43	681	623	600	600
query44	1108	715	722	715
query45	241	225	226	225
query46	1235	973	961	961
query47	1910	1613	2063	1613
query48	991	670	670	670
query49	614	374	349	349
query50	853	613	599	599
query51	5512	5469	5519	5469
query52	88	83	78	78
query53	441	310	312	310
query54	2658	2437	2478	2437
query55	77	82	75	75
query56	216	203	180	180
query57	1278	1095	1114	1095
query58	212	196	184	184
query59	3569	3322	3250	3250
query60	194	196	194	194
query61	80	82	79	79
query62	795	476	484	476
query63	473	342	325	325
query64	2451	1483	1347	1347
query65	3652	3516	3532	3516
query66	763	373	359	359
query67	15565	15353	15746	15353
query68	8388	665	644	644
query69	550	343	340	340
query70	1922	1768	1661	1661
query71	402	301	297	297
query72	6487	3444	3385	3385
query73	728	326	315	315
query74	6301	5837	5846	5837
query75	4514	3666	3727	3666
query76	4699	1165	1202	1165
query77	608	246	240	240
query78	32307	50942	45143	45143
query79	15959	638	655	638
query80	5410	376	378	376
query81	559	226	228	226
query82	898	94	90	90
query83	421	132	128	128
query84	262	66	68	66
query85	2665	283	282	282
query86	477	388	378	378
query87	3260	2986	3016	2986
query88	6908	2343	2326	2326
query89	458	276	318	276
query90	2501	187	199	187
query91	150	129	117	117
query92	63	50	48	48
query93	6415	555	587	555
query94	1842	200	209	200
query95	1076	1058	1051	1051
query96	640	328	322	322
query97	6451	6350	6334	6334
query98	188	172	160	160
query99	3941	911	934	911
Total cold run time: 348505 ms
Total hot run time: 235469 ms

[doris-robot]

ClickBench: Total hot run time: 30.11 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85836d9fe98361450af872736a638e2bb4e83917, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.04	0.04
query4	1.83	0.07	0.06
query5	0.53	0.53	0.52
query6	1.33	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.52	0.50	0.48
query10	0.52	0.54	0.52
query11	0.12	0.09	0.08
query12	0.12	0.10	0.09
query13	0.61	0.60	0.61
query14	0.78	0.78	0.78
query15	0.77	0.75	0.75
query16	0.36	0.37	0.36
query17	1.01	0.99	1.00
query18	0.22	0.25	0.24
query19	1.92	1.86	1.82
query20	0.01	0.01	0.01
query21	15.49	0.55	0.53
query22	1.98	2.31	1.00
query23	17.05	1.06	0.90
query24	7.12	2.33	1.16
query25	1.56	0.12	0.11
query26	0.39	0.14	0.14
query27	0.11	0.11	0.11
query28	5.53	0.72	0.72
query29	12.63	2.37	2.12
query30	0.60	0.56	0.52
query31	2.82	0.36	0.38
query32	3.44	0.49	0.49
query33	3.07	3.05	3.04
query34	15.25	4.82	4.79
query35	4.82	4.85	4.83
query36	1.06	1.02	1.02
query37	0.05	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.32 s
Total hot run time: 30.11 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 85836d9fe98361450af872736a638e2bb4e83917 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

[morningman]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[morningman]

run buildall

[doris-robot]

TPC-H: Total hot run time: 50187 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 566084b3477de30852f38b0420511806ca412cd3, data reload: false

------ Round 1 ----------------------------------
q1	17586	4406	4488	4406
q2	2071	159	159	159
q3	10485	1950	1923	1923
q4	10105	1265	1310	1265
q5	8487	3975	3984	3975
q6	236	123	123	123
q7	2062	1601	1588	1588
q8	9307	2727	2739	2727
q9	10969	10475	10470	10470
q10	8669	3550	3550	3550
q11	418	235	247	235
q12	468	303	311	303
q13	18368	3963	4068	3963
q14	357	326	326	326
q15	501	462	449	449
q16	709	576	599	576
q17	1140	949	964	949
q18	7353	6892	6929	6892
q19	1693	1552	1488	1488
q20	533	319	313	313
q21	4469	4149	4115	4115
q22	502	392	403	392
Total cold run time: 116488 ms
Total hot run time: 50187 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4287	4335	4287
q2	315	226	224	224
q3	4157	4158	4124	4124
q4	2755	2752	2737	2737
q5	7333	7101	7241	7101
q6	233	119	117	117
q7	3209	2860	2853	2853
q8	4425	4500	4513	4500
q9	17028	16971	17035	16971
q10	4274	4253	4280	4253
q11	762	693	684	684
q12	1011	847	857	847
q13	6489	3718	3774	3718
q14	451	427	424	424
q15	502	460	454	454
q16	777	714	701	701
q17	3862	3818	3908	3818
q18	8818	8820	8863	8820
q19	1712	1711	1688	1688
q20	2371	2184	2129	2129
q21	8532	8581	8563	8563
q22	1039	987	917	917
Total cold run time: 84420 ms
Total hot run time: 79930 ms

[doris-robot]

TPC-DS: Total hot run time: 237135 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 566084b3477de30852f38b0420511806ca412cd3, data reload: false

query1	923	390	381	381
query2	6529	2167	2263	2167
query3	6924	203	202	202
query4	21104	18008	18121	18008
query5	19729	6563	6447	6447
query6	290	211	231	211
query7	4152	305	307	305
query8	244	227	278	227
query9	3155	2716	2656	2656
query10	434	291	304	291
query11	11361	10657	10658	10657
query12	129	78	74	74
query13	5568	648	630	630
query14	18177	13312	13378	13312
query15	359	235	242	235
query16	6402	264	260	260
query17	1723	1446	875	875
query18	2309	416	415	415
query19	203	146	143	143
query20	78	78	77	77
query21	186	101	90	90
query22	5221	5031	4733	4733
query23	32757	31888	32134	31888
query24	6960	6547	6523	6523
query25	525	431	444	431
query26	525	169	162	162
query27	1830	302	298	298
query28	6076	2230	2179	2179
query29	2970	2713	2674	2674
query30	247	157	161	157
query31	892	725	721	721
query32	66	59	55	55
query33	391	241	243	241
query34	846	472	475	472
query35	1121	919	924	919
query36	1741	1399	1463	1399
query37	90	63	58	58
query38	3111	2968	2878	2878
query39	1355	1347	1323	1323
query40	213	95	99	95
query41	42	34	36	34
query42	87	90	80	80
query43	712	643	585	585
query44	1147	716	723	716
query45	242	226	231	226
query46	1235	980	980	980
query47	1824	1791	2093	1791
query48	990	701	695	695
query49	626	375	372	372
query50	860	611	631	611
query51	5557	5510	5468	5468
query52	110	79	88	79
query53	437	318	323	318
query54	2693	2505	2483	2483
query55	83	80	80	80
query56	221	201	197	197
query57	1169	1171	1048	1048
query58	206	207	194	194
query59	3596	3518	3302	3302
query60	206	183	206	183
query61	82	86	80	80
query62	798	475	513	475
query63	483	342	336	336
query64	2453	1521	1481	1481
query65	3675	3598	3582	3582
query66	760	380	367	367
query67	16015	15524	15700	15524
query68	8220	657	688	657
query69	581	336	330	330
query70	1772	1551	1993	1551
query71	389	312	321	312
query72	6460	3370	3377	3370
query73	746	323	318	318
query74	6304	5831	5845	5831
query75	4552	3807	3774	3774
query76	4742	1133	1198	1133
query77	581	253	249	249
query78	31083	52038	45682	45682
query79	15267	645	648	645
query80	4981	378	385	378
query81	564	231	230	230
query82	756	91	100	91
query83	413	140	126	126
query84	261	69	68	68
query85	2642	281	276	276
query86	486	363	355	355
query87	3293	2996	3002	2996
query88	6403	2364	2349	2349
query89	402	285	293	285
query90	2480	212	210	210
query91	162	124	120	120
query92	65	49	51	49
query93	5033	576	583	576
query94	1844	208	209	208
query95	1107	1079	1054	1054
query96	646	340	328	328
query97	6606	6383	6427	6383
query98	191	178	178	178
query99	3848	1012	867	867
Total cold run time: 346105 ms
Total hot run time: 237135 ms

[doris-robot]

ClickBench: Total hot run time: 30.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 566084b3477de30852f38b0420511806ca412cd3, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.03
query3	0.25	0.04	0.04
query4	1.81	0.07	0.06
query5	0.53	0.52	0.52
query6	1.24	0.62	0.60
query7	0.01	0.00	0.01
query8	0.03	0.02	0.03
query9	0.52	0.48	0.48
query10	0.54	0.53	0.52
query11	0.12	0.09	0.08
query12	0.11	0.09	0.09
query13	0.62	0.62	0.61
query14	0.78	0.78	0.79
query15	0.78	0.75	0.76
query16	0.37	0.36	0.37
query17	0.99	1.02	1.02
query18	0.22	0.28	0.24
query19	1.93	1.85	1.81
query20	0.01	0.01	0.01
query21	15.49	0.56	0.55
query22	2.31	2.67	1.34
query23	17.37	0.95	0.94
query24	8.01	1.08	1.22
query25	1.52	0.11	0.11
query26	0.38	0.15	0.13
query27	0.11	0.11	0.11
query28	6.14	0.72	0.73
query29	12.64	2.16	2.27
query30	0.57	0.54	0.54
query31	2.81	0.37	0.40
query32	3.38	0.49	0.50
query33	3.07	3.04	3.02
query34	15.25	4.82	4.79
query35	4.85	4.83	4.85
query36	1.05	1.01	1.02
query37	0.06	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.01	0.02
query40	0.15	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.01
Total cold run time: 106.27 s
Total hot run time: 30.48 s

[doris-robot]

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 566084b3477de30852f38b0420511806ca412cd3 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s