[improvement](hudi)Obtain partition information through HMS's API #30962

wuwenchi · 2024-02-07T10:12:55Z

Proposed changes

When a Hudi table is synchronized to HMS, the partition information is also synchronized, so even if the metastore is not enabled in the Hudi table (for example, if the Metastore is false for a Hudi table created with Flink), you can still obtain the partition information through the HMS API.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-02-07T10:13:00Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

wuwenchi · 2024-02-07T10:15:52Z

run buildall

wuwenchi · 2024-02-07T10:16:11Z

run buildall

wuwenchi · 2024-02-08T06:57:19Z

run buildall

wuwenchi · 2024-02-08T06:58:00Z

run buildall

wuwenchi · 2024-02-08T07:04:53Z

run buildall

doris-robot · 2024-02-08T07:28:44Z

TPC-H: Total hot run time: 41260 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 270aee7e98b0da8f15c1e654ac970e682d5707f6, data reload: false

------ Round 1 ----------------------------------
q1	17649	5001	4862	4862
q2	2040	136	133	133
q3	10594	1002	1024	1002
q4	4653	972	968	968
q5	7693	3220	3262	3220
q6	197	136	131	131
q7	1277	769	756	756
q8	9238	2047	2057	2047
q9	7573	6656	6663	6656
q10	8300	2634	2631	2631
q11	403	212	201	201
q12	793	324	328	324
q13	17987	3672	3685	3672
q14	284	266	252	252
q15	589	506	554	506
q16	486	398	414	398
q17	911	842	777	777
q18	7461	6738	6626	6626
q19	1553	1486	1485	1485
q20	631	342	321	321
q21	7312	3968	3975	3968
q22	875	353	324	324
Total cold run time: 108499 ms
Total hot run time: 41260 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4818	4845	4816	4816
q2	291	185	179	179
q3	3596	3591	3557	3557
q4	2504	2511	2487	2487
q5	5759	5691	5724	5691
q6	211	127	122	122
q7	2247	1668	1664	1664
q8	3018	3092	3092	3092
q9	8690	8691	8685	8685
q10	6743	4213	4205	4205
q11	516	375	390	375
q12	756	547	532	532
q13	4664	3412	3405	3405
q14	265	237	246	237
q15	608	500	491	491
q16	486	446	437	437
q17	1659	1603	1600	1600
q18	8249	7593	7631	7593
q19	1632	1627	1623	1623
q20	2133	1822	1816	1816
q21	6538	6124	6180	6124
q22	551	497	492	492
Total cold run time: 65934 ms
Total hot run time: 59223 ms

doris-robot · 2024-02-08T07:40:21Z

TPC-DS: Total hot run time: 180763 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 270aee7e98b0da8f15c1e654ac970e682d5707f6, data reload: false

query1	923	357	355	355
query2	6502	1671	1803	1671
query3	6707	203	197	197
query4	24459	22512	22667	22512
query5	4211	454	383	383
query6	258	173	166	166
query7	5388	317	308	308
query8	702	193	249	193
query9	9377	2839	2833	2833
query10	431	228	220	220
query11	16279	15732	15797	15732
query12	145	88	94	88
query13	1733	425	439	425
query14	14199	8088	7767	7767
query15	216	199	194	194
query16	7838	276	249	249
query17	2438	1067	567	567
query18	3197	271	259	259
query19	196	146	148	146
query20	87	82	84	82
query21	189	119	122	119
query22	4943	4793	4777	4777
query23	32528	31628	31550	31550
query24	10248	3359	3451	3359
query25	608	372	357	357
query26	1404	151	163	151
query27	2970	305	308	305
query28	6758	1836	1816	1816
query29	1021	614	613	613
query30	270	142	141	141
query31	950	751	762	751
query32	102	67	56	56
query33	727	225	226	225
query34	1076	489	487	487
query35	950	837	828	828
query36	973	886	899	886
query37	130	61	58	58
query38	3479	3322	3303	3303
query39	1376	1310	1325	1310
query40	197	109	102	102
query41	37	35	34	34
query42	108	95	102	95
query43	471	432	433	432
query44	1038	686	701	686
query45	200	189	174	174
query46	1056	766	730	730
query47	1672	1587	1584	1584
query48	410	341	344	341
query49	1097	298	299	298
query50	768	390	369	369
query51	5324	5205	5193	5193
query52	104	90	85	85
query53	392	299	300	299
query54	311	216	228	216
query55	85	81	76	76
query56	218	198	195	195
query57	1077	961	943	943
query58	209	204	193	193
query59	2331	2183	2182	2182
query60	252	217	215	215
query61	91	87	83	83
query62	591	381	379	379
query63	325	272	284	272
query64	6291	3078	3156	3078
query65	3272	3262	3241	3241
query66	1298	326	318	318
query67	14610	14481	14362	14362
query68	5225	524	532	524
query69	511	367	352	352
query70	1211	1194	1265	1194
query71	423	253	254	253
query72	6289	2785	2588	2588
query73	690	308	307	307
query74	7092	6611	6673	6611
query75	3253	2559	2549	2549
query76	3396	1133	1192	1133
query77	390	232	230	230
query78	9488	8839	8706	8706
query79	970	489	504	489
query80	514	364	339	339
query81	448	201	204	201
query82	161	86	85	85
query83	138	124	118	118
query84	222	77	76	76
query85	1012	336	333	333
query86	307	307	311	307
query87	3521	3410	3388	3388
query88	2652	2283	2254	2254
query89	427	361	363	361
query90	2019	168	162	162
query91	155	125	131	125
query92	54	46	47	46
query93	960	473	464	464
query94	1287	180	174	174
query95	8793	8356	369	369
query96	574	270	270	270
query97	4398	4259	4338	4259
query98	221	200	194	194
query99	1077	703	688	688
Total cold run time: 287701 ms
Total hot run time: 180763 ms

doris-robot · 2024-02-08T07:45:45Z

ClickBench: Total hot run time: 30.53 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 270aee7e98b0da8f15c1e654ac970e682d5707f6, data reload: false

query1	0.03	0.03	0.02
query2	0.05	0.03	0.03
query3	0.22	0.08	0.08
query4	1.63	0.09	0.08
query5	0.48	0.48	0.47
query6	1.41	0.59	0.61
query7	0.02	0.01	0.01
query8	0.05	0.03	0.03
query9	0.53	0.47	0.45
query10	0.50	0.49	0.48
query11	0.14	0.09	0.09
query12	0.12	0.11	0.10
query13	0.59	0.59	0.59
query14	0.76	0.79	0.77
query15	0.83	0.80	0.79
query16	0.35	0.33	0.34
query17	0.90	0.89	0.91
query18	0.20	0.17	0.14
query19	1.77	1.67	1.69
query20	0.02	0.01	0.01
query21	15.42	0.63	0.59
query22	3.54	3.37	1.92
query23	17.61	1.10	0.98
query24	2.00	0.54	0.37
query25	0.63	0.06	0.06
query26	0.16	0.14	0.14
query27	0.05	0.05	0.05
query28	12.01	0.84	0.83
query29	12.50	3.38	3.34
query30	0.54	0.50	0.46
query31	2.78	0.35	0.37
query32	3.34	0.47	0.48
query33	3.13	3.14	3.15
query34	15.37	4.50	4.49
query35	4.50	4.49	4.51
query36	1.07	0.94	0.94
query37	0.07	0.05	0.05
query38	0.03	0.04	0.03
query39	0.01	0.01	0.02
query40	0.19	0.15	0.15
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.67 s
Total hot run time: 30.53 s

doris-robot · 2024-02-08T07:48:51Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 270aee7e98b0da8f15c1e654ac970e682d5707f6 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

morningman

LGTM

github-actions · 2024-02-09T07:24:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-02-09T07:24:03Z

PR approved by anyone and no changes requested.

#31008 bp #30962 Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>

morningman · 2024-02-18T01:57:04Z

run buildall

doris-robot · 2024-02-18T02:35:26Z

TPC-H: Total hot run time: 41683 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c3fbfb0711136c78edff3ba7dc85a0f117134a1a, data reload: false

------ Round 1 ----------------------------------
q1	17717	4920	4838	4838
q2	2032	144	132	132
q3	10588	1032	1000	1000
q4	4657	956	976	956
q5	7713	3219	3252	3219
q6	198	138	131	131
q7	1258	794	777	777
q8	9355	2074	2065	2065
q9	7574	6726	6684	6684
q10	8348	2668	2665	2665
q11	415	226	208	208
q12	768	334	357	334
q13	20762	3706	3688	3688
q14	294	264	262	262
q15	622	516	535	516
q16	467	412	408	408
q17	917	851	833	833
q18	7481	6660	6661	6660
q19	1552	1487	1490	1487
q20	637	348	308	308
q21	7448	4219	4134	4134
q22	1015	393	378	378
Total cold run time: 111818 ms
Total hot run time: 41683 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5460	5450	5167	5167
q2	292	186	186	186
q3	3593	3644	3676	3644
q4	2567	2566	2557	2557
q5	5754	5802	5758	5758
q6	210	126	131	126
q7	2236	1662	1657	1657
q8	3008	3083	3078	3078
q9	8673	8659	8703	8659
q10	6765	4207	4207	4207
q11	526	381	378	378
q12	770	535	552	535
q13	4336	3403	3395	3395
q14	261	238	230	230
q15	626	514	521	514
q16	479	436	431	431
q17	1671	1596	1582	1582
q18	8429	7619	7654	7619
q19	1618	1632	1623	1623
q20	2123	1840	1843	1840
q21	6528	6147	6111	6111
q22	579	519	494	494
Total cold run time: 66504 ms
Total hot run time: 59791 ms

doris-robot · 2024-02-18T02:47:04Z

TPC-DS: Total hot run time: 189254 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c3fbfb0711136c78edff3ba7dc85a0f117134a1a, data reload: false

query1	920	349	355	349
query2	6532	1700	1721	1700
query3	6701	207	197	197
query4	24407	22478	22483	22478
query5	4309	463	453	453
query6	255	166	162	162
query7	4606	306	290	290
query8	247	197	198	197
query9	8417	2850	2827	2827
query10	420	217	225	217
query11	16216	15614	15602	15602
query12	144	82	85	82
query13	1716	422	417	417
query14	9091	7706	7729	7706
query15	217	181	195	181
query16	7352	262	252	252
query17	1414	565	533	533
query18	1964	259	275	259
query19	199	143	148	143
query20	84	84	89	84
query21	190	127	126	126
query22	5004	4713	4754	4713
query23	32462	31513	31511	31511
query24	12756	3508	3402	3402
query25	622	370	358	358
query26	1823	153	159	153
query27	3059	312	313	312
query28	6659	1883	1878	1878
query29	1142	681	621	621
query30	275	142	151	142
query31	951	795	786	786
query32	103	60	54	54
query33	722	243	238	238
query34	1082	504	507	504
query35	932	837	835	835
query36	1002	944	924	924
query37	136	60	62	60
query38	3454	3310	3296	3296
query39	1374	1323	1315	1315
query40	281	105	107	105
query41	37	35	35	35
query42	108	98	102	98
query43	464	436	447	436
query44	1083	690	718	690
query45	201	182	182	182
query46	1047	809	749	749
query47	1692	1582	1587	1582
query48	428	348	347	347
query49	1167	317	315	315
query50	779	390	393	390
query51	5328	5175	5181	5175
query52	109	91	94	91
query53	396	297	303	297
query54	291	227	229	227
query55	83	79	76	76
query56	218	201	221	201
query57	1027	948	991	948
query58	227	201	202	201
query59	2246	2207	2095	2095
query60	255	225	215	215
query61	84	85	84	84
query62	592	377	359	359
query63	319	282	290	282
query64	6467	3133	3137	3133
query65	3292	3262	3253	3253
query66	1164	332	320	320
query67	14774	14396	14344	14344
query68	5106	575	570	570
query69	496	348	353	348
query70	1265	1245	1182	1182
query71	364	251	252	251
query72	6369	2821	2585	2585
query73	693	313	312	312
query74	7097	6632	6655	6632
query75	3146	2583	2583	2583
query76	3060	1154	1199	1154
query77	340	237	233	233
query78	9420	8837	8778	8778
query79	1005	506	493	493
query80	624	349	345	345
query81	450	207	202	202
query82	333	83	85	83
query83	142	123	119	119
query84	229	80	81	80
query85	1317	345	343	343
query86	326	307	305	305
query87	3614	3393	3386	3386
query88	2715	2311	2297	2297
query89	446	355	343	343
query90	2000	163	168	163
query91	157	129	132	129
query92	55	51	51	51
query93	1177	483	488	483
query94	1344	182	183	182
query95	8778	8601	8561	8561
query96	575	274	273	273
query97	4395	4292	4287	4287
query98	218	203	194	194
query99	1063	720	726	720
Total cold run time: 281437 ms
Total hot run time: 189254 ms

doris-robot · 2024-02-18T02:52:29Z

ClickBench: Total hot run time: 31.41 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c3fbfb0711136c78edff3ba7dc85a0f117134a1a, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.65	0.08	0.08
query5	0.48	0.48	0.49
query6	1.35	0.62	0.61
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.53	0.45	0.45
query10	0.49	0.49	0.51
query11	0.13	0.10	0.10
query12	0.12	0.09	0.09
query13	0.59	0.60	0.59
query14	0.76	0.77	0.78
query15	0.83	0.82	0.79
query16	0.33	0.33	0.34
query17	0.96	0.90	0.92
query18	0.19	0.17	0.19
query19	1.79	1.68	1.75
query20	0.01	0.01	0.01
query21	15.40	0.67	0.61
query22	3.36	3.52	2.67
query23	17.65	1.11	0.92
query24	2.05	0.58	0.39
query25	0.60	0.07	0.05
query26	0.17	0.14	0.14
query27	0.06	0.06	0.06
query28	11.91	0.82	0.84
query29	12.57	3.36	3.44
query30	0.54	0.49	0.48
query31	2.78	0.37	0.37
query32	3.32	0.48	0.47
query33	3.16	3.13	3.18
query34	15.35	4.54	4.51
query35	4.54	4.52	4.50
query36	1.06	0.97	0.98
query37	0.07	0.05	0.06
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.18	0.15	0.14
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.55 s
Total hot run time: 31.41 s

doris-robot · 2024-02-18T02:55:36Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c3fbfb0711136c78edff3ba7dc85a0f117134a1a with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.9 seconds inserted 10000000 Rows, about 719K ops/s

wuwenchi · 2024-02-19T03:39:03Z

run buildall

morningman · 2024-02-19T09:31:55Z

run buildall

wuwenchi · 2024-02-20T10:18:46Z

run buildall

wuwenchi · 2024-02-20T10:46:36Z

run buildall

doris-robot · 2024-02-20T11:18:13Z

TPC-H: Total hot run time: 41344 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c396702f6def4385ff68fd91b9ed38588162660c, data reload: false

------ Round 1 ----------------------------------
q1	17670	4963	4858	4858
q2	2039	139	131	131
q3	10579	1031	994	994
q4	4663	959	975	959
q5	7682	3224	3231	3224
q6	191	131	132	131
q7	1235	794	764	764
q8	9256	2070	2058	2058
q9	7599	6680	6671	6671
q10	8309	2613	2626	2613
q11	414	219	208	208
q12	722	336	328	328
q13	17982	3690	3659	3659
q14	292	255	258	255
q15	573	530	522	522
q16	502	400	408	400
q17	915	867	850	850
q18	7312	6619	6771	6619
q19	1543	1498	1490	1490
q20	642	325	341	325
q21	6965	4003	3936	3936
q22	854	357	349	349
Total cold run time: 107939 ms
Total hot run time: 41344 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5279	4818	4805	4805
q2	289	181	178	178
q3	3592	3581	3558	3558
q4	2514	2517	2522	2517
q5	5773	5727	5713	5713
q6	210	122	126	122
q7	2248	1669	1660	1660
q8	2990	3108	3111	3108
q9	8714	8722	8642	8642
q10	6795	4238	4237	4237
q11	511	391	367	367
q12	770	557	546	546
q13	4320	3410	3374	3374
q14	270	241	247	241
q15	628	504	503	503
q16	471	445	452	445
q17	1664	1604	1606	1604
q18	8333	7596	7605	7596
q19	1629	1634	1638	1634
q20	2080	1815	1823	1815
q21	6501	6129	6107	6107
q22	605	500	512	500
Total cold run time: 66186 ms
Total hot run time: 59272 ms

doris-robot · 2024-02-20T11:29:24Z

TPC-DS: Total hot run time: 176797 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c396702f6def4385ff68fd91b9ed38588162660c, data reload: false

query1	918	351	351	351
query2	6517	1687	1733	1687
query3	6695	205	200	200
query4	23170	21006	21229	21006
query5	4280	366	461	366
query6	261	165	160	160
query7	4613	292	299	292
query8	232	194	196	194
query9	8426	2789	2772	2772
query10	422	222	222	222
query11	14956	14661	14336	14336
query12	146	81	82	81
query13	1677	422	406	406
query14	9558	7435	7529	7435
query15	213	188	194	188
query16	7551	249	251	249
query17	1403	564	526	526
query18	1968	272	254	254
query19	190	137	141	137
query20	86	83	79	79
query21	185	116	120	116
query22	4983	4723	4655	4655
query23	32512	31462	31660	31462
query24	12719	3377	3357	3357
query25	642	366	356	356
query26	1893	148	156	148
query27	3065	317	318	317
query28	6625	1852	1846	1846
query29	1156	611	605	605
query30	276	137	145	137
query31	937	749	769	749
query32	94	56	56	56
query33	719	234	224	224
query34	1089	479	489	479
query35	948	811	821	811
query36	991	878	864	864
query37	180	60	61	60
query38	3283	3172	3181	3172
query39	1381	1320	1305	1305
query40	279	105	104	104
query41	36	38	34	34
query42	99	100	98	98
query43	471	449	451	449
query44	1063	693	717	693
query45	198	182	176	176
query46	1053	767	735	735
query47	1653	1547	1618	1547
query48	414	346	350	346
query49	1193	303	299	299
query50	760	395	368	368
query51	5255	5182	5147	5147
query52	102	94	87	87
query53	392	292	287	287
query54	297	214	226	214
query55	78	82	80	80
query56	216	195	197	195
query57	1043	964	925	925
query58	223	192	198	192
query59	2243	2200	2117	2117
query60	244	209	208	208
query61	81	79	87	79
query62	605	375	379	375
query63	329	272	278	272
query64	6329	3047	3113	3047
query65	3287	3246	3240	3240
query66	1344	334	320	320
query67	14466	14404	14454	14404
query68	5134	549	561	549
query69	500	351	345	345
query70	1283	1178	1208	1178
query71	443	245	267	245
query72	6272	2767	2612	2612
query73	703	309	309	309
query74	6940	6498	6625	6498
query75	3239	2553	2530	2530
query76	3272	1137	1211	1137
query77	362	238	233	233
query78	9534	8869	8807	8807
query79	958	491	511	491
query80	519	356	336	336
query81	436	203	200	200
query82	240	82	82	82
query83	139	121	116	116
query84	227	77	75	75
query85	1026	349	333	333
query86	294	299	310	299
query87	3460	3276	3316	3276
query88	2681	2264	2264	2264
query89	421	349	348	348
query90	1969	160	162	160
query91	149	121	132	121
query92	54	50	48	48
query93	1009	530	493	493
query94	1252	177	176	176
query95	464	8443	362	362
query96	573	265	260	260
query97	4451	4288	4321	4288
query98	222	201	191	191
query99	1046	737	770	737
Total cold run time: 269985 ms
Total hot run time: 176797 ms

doris-robot · 2024-02-20T11:34:53Z

ClickBench: Total hot run time: 31.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c396702f6def4385ff68fd91b9ed38588162660c, data reload: false

query1	0.03	0.02	0.02
query2	0.08	0.03	0.03
query3	0.23	0.07	0.08
query4	1.63	0.09	0.08
query5	0.48	0.47	0.48
query6	1.37	0.61	0.62
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.52	0.45	0.45
query10	0.48	0.47	0.48
query11	0.13	0.09	0.09
query12	0.12	0.10	0.10
query13	0.59	0.59	0.58
query14	0.76	0.78	0.78
query15	0.81	0.81	0.80
query16	0.33	0.36	0.33
query17	0.90	0.91	0.92
query18	0.17	0.17	0.17
query19	1.78	1.67	1.67
query20	0.02	0.01	0.01
query21	15.40	0.67	0.59
query22	3.45	3.76	2.63
query23	17.25	0.95	0.98
query24	2.25	0.33	0.58
query25	0.55	0.07	0.06
query26	0.16	0.13	0.14
query27	0.04	0.06	0.04
query28	11.78	0.85	0.80
query29	12.60	3.32	3.33
query30	0.60	0.52	0.45
query31	2.78	0.36	0.38
query32	3.32	0.47	0.47
query33	3.08	3.16	3.12
query34	15.39	4.46	4.47
query35	4.50	4.50	4.49
query36	1.09	0.93	0.97
query37	0.07	0.05	0.05
query38	0.04	0.04	0.03
query39	0.02	0.02	0.01
query40	0.18	0.14	0.15
query41	0.08	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.16 s
Total hot run time: 31.08 s

doris-robot · 2024-02-20T11:38:00Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c396702f6def4385ff68fd91b9ed38588162660c with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

morningman

LGTM

…0962) When a Hudi table is synchronized to HMS, the partition information is also synchronized, so even if the metastore is not enabled in the Hudi table (for example, if the Metastore is false for a Hudi table created with Flink), you can still obtain the partition information through the HMS API.

…che#30962 apache#31008 bp apache#30962 Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>

wuwenchi closed this Feb 7, 2024

wuwenchi reopened this Feb 7, 2024

morningman approved these changes Feb 9, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 9, 2024

github-actions bot added the reviewed label Feb 9, 2024

morningman mentioned this pull request Feb 9, 2024

[improvement](hudi)Obtain partition information through HMS's API #30962 #31008

Merged

morningman added a commit that referenced this pull request Feb 9, 2024

[improvement](hudi)Obtain partition information through HMS's API #30962

fc45da6

#31008 bp #30962 Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>

wm1581066 added the dev/2.0.5-merged label Feb 15, 2024

morningman previously approved these changes Feb 18, 2024

View reviewed changes

wuwenchi dismissed morningman’s stale review via aa2bd6b February 19, 2024 03:38

wuwenchi mentioned this pull request Feb 19, 2024

[improvement](hudi)Obtain partition information through HMS's API for 2.0 #31087

Merged

AshinGau previously approved these changes Feb 20, 2024

View reviewed changes

wuwenchi added 4 commits February 20, 2024 17:58

opt

ded1f8f

add comment

fafe6c1

add useHiveSyncPartition

8297bf6

doc

a714c38

wuwenchi dismissed AshinGau’s stale review via a714c38 February 20, 2024 10:18

merge

c396702

AshinGau approved these changes Feb 21, 2024

View reviewed changes

morningman approved these changes Feb 22, 2024

View reviewed changes

morningman merged commit c837695 into apache:master Feb 22, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[improvement](hudi)Obtain partition information through HMS's API apa…

08244fd

…che#30962 apache#31008 bp apache#30962 Co-authored-by: wuwenchi <wuwenchihdu@hotmail.com>

[improvement](hudi)Obtain partition information through HMS's API #30962

[improvement](hudi)Obtain partition information through HMS's API #30962

Uh oh!

Conversation

wuwenchi commented Feb 7, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Feb 7, 2024

Uh oh!

wuwenchi commented Feb 7, 2024

Uh oh!

wuwenchi commented Feb 7, 2024

Uh oh!

wuwenchi commented Feb 8, 2024

Uh oh!

wuwenchi commented Feb 8, 2024

Uh oh!

wuwenchi commented Feb 8, 2024

Uh oh!

doris-robot commented Feb 8, 2024

Uh oh!

doris-robot commented Feb 8, 2024

Uh oh!

doris-robot commented Feb 8, 2024

Uh oh!

doris-robot commented Feb 8, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Feb 9, 2024

Uh oh!

github-actions bot commented Feb 9, 2024

Uh oh!

morningman commented Feb 18, 2024

Uh oh!

doris-robot commented Feb 18, 2024

Uh oh!

doris-robot commented Feb 18, 2024

Uh oh!

doris-robot commented Feb 18, 2024

Uh oh!

doris-robot commented Feb 18, 2024

Uh oh!

wuwenchi commented Feb 19, 2024

Uh oh!

morningman commented Feb 19, 2024

Uh oh!

wuwenchi commented Feb 20, 2024

Uh oh!

wuwenchi commented Feb 20, 2024

Uh oh!

doris-robot commented Feb 20, 2024

Uh oh!

doris-robot commented Feb 20, 2024

Uh oh!

doris-robot commented Feb 20, 2024

Uh oh!

doris-robot commented Feb 20, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants