Skip to content

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Feb 10, 2025

What problem does this PR solve?

Issue Number: CIR-14357

Problem Summary:

Azure operated in China uses chinacloudapi endpoint.
We should accept user specified endpoint for Azure S3 load.

A new FE and BE config force_azure_blob_global_endpoint is added.
This parameter controls whether to force the use of the Azure global endpoint.
The default is false, meaning the system will use the user-specified endpoint.
If set to true, the system will force the use of {account}.blob.core.windows.net.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaijchen
Copy link
Member Author

run buildall

1 similar comment
@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.89% (17549/26634)
Region Coverage: 65.39% (8656/13237)
Branch Coverage: 55.25% (4664/8442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8621502fafaf571316a01706e1e6eb6ef01a8ce6_8621502fafaf571316a01706e1e6eb6ef01a8ce6_cloud/report/index.html

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.90% (17553/26634)
Region Coverage: 65.43% (8661/13237)
Branch Coverage: 55.29% (4668/8442)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5a7e3ae690f39588af274d638a58b7452d7da0b1_5a7e3ae690f39588af274d638a58b7452d7da0b1_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 31171 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5a7e3ae690f39588af274d638a58b7452d7da0b1, data reload: false

------ Round 1 ----------------------------------
q1	17623	5199	5047	5047
q2	2048	295	163	163
q3	10423	1297	682	682
q4	10231	1000	523	523
q5	7518	2399	2342	2342
q6	185	164	131	131
q7	907	771	605	605
q8	9296	1319	1023	1023
q9	4824	4540	4611	4540
q10	6794	2298	1891	1891
q11	499	270	245	245
q12	336	350	208	208
q13	17758	3677	3063	3063
q14	219	219	205	205
q15	506	465	464	464
q16	624	613	568	568
q17	549	870	325	325
q18	6561	6188	6276	6188
q19	1081	942	525	525
q20	291	323	200	200
q21	2875	2149	1926	1926
q22	363	312	307	307
Total cold run time: 101511 ms
Total hot run time: 31171 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5049	5494	5043	5043
q2	232	333	229	229
q3	2133	2669	2264	2264
q4	1415	1838	1351	1351
q5	4234	4113	4135	4113
q6	200	164	126	126
q7	1854	1788	1635	1635
q8	2543	2545	2616	2545
q9	7273	7150	7087	7087
q10	2941	3207	2775	2775
q11	580	500	480	480
q12	653	779	588	588
q13	3441	3923	3229	3229
q14	281	303	260	260
q15	497	478	467	467
q16	672	668	620	620
q17	1134	1568	1337	1337
q18	7575	7361	7244	7244
q19	763	804	858	804
q20	1933	2025	1865	1865
q21	5387	4881	4852	4852
q22	660	597	530	530
Total cold run time: 51450 ms
Total hot run time: 49444 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183362 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5a7e3ae690f39588af274d638a58b7452d7da0b1, data reload: false

query1	977	373	385	373
query2	6536	1886	1834	1834
query3	6791	214	211	211
query4	26543	23548	23395	23395
query5	4398	668	494	494
query6	296	195	179	179
query7	4606	505	303	303
query8	306	240	241	240
query9	8648	2493	2487	2487
query10	470	305	253	253
query11	15739	15137	14917	14917
query12	175	113	108	108
query13	1657	522	395	395
query14	10044	6144	6120	6120
query15	203	204	174	174
query16	7645	618	423	423
query17	1192	711	559	559
query18	1995	401	303	303
query19	194	203	161	161
query20	119	115	112	112
query21	214	125	111	111
query22	4140	4477	4581	4477
query23	34429	33247	32628	32628
query24	7898	2331	2378	2331
query25	521	445	369	369
query26	1239	262	153	153
query27	2573	493	314	314
query28	4292	2366	2351	2351
query29	695	531	413	413
query30	228	181	151	151
query31	930	854	787	787
query32	70	62	60	60
query33	567	347	303	303
query34	758	827	503	503
query35	771	827	752	752
query36	959	989	904	904
query37	120	98	75	75
query38	4057	4399	4285	4285
query39	1461	1409	1440	1409
query40	203	111	99	99
query41	53	52	52	52
query42	122	101	103	101
query43	503	522	478	478
query44	1260	798	772	772
query45	179	170	157	157
query46	858	1046	649	649
query47	1744	1771	1735	1735
query48	372	399	288	288
query49	776	509	412	412
query50	679	743	419	419
query51	4188	4125	4113	4113
query52	103	101	91	91
query53	214	250	180	180
query54	466	467	403	403
query55	80	81	79	79
query56	251	258	240	240
query57	1115	1125	1065	1065
query58	244	221	245	221
query59	2603	2606	2596	2596
query60	284	269	243	243
query61	113	111	113	111
query62	833	770	670	670
query63	223	198	182	182
query64	4244	981	683	683
query65	3217	3152	3132	3132
query66	1066	400	294	294
query67	15663	15840	15518	15518
query68	7934	751	486	486
query69	447	304	270	270
query70	1234	1142	1096	1096
query71	439	304	265	265
query72	5544	3541	3757	3541
query73	738	723	346	346
query74	9167	9212	8932	8932
query75	3624	3185	2677	2677
query76	3415	1161	731	731
query77	775	370	281	281
query78	9942	10040	9310	9310
query79	2323	888	590	590
query80	602	529	439	439
query81	506	274	230	230
query82	589	148	119	119
query83	170	172	159	159
query84	237	82	70	70
query85	769	353	286	286
query86	389	310	288	288
query87	4424	4505	4387	4387
query88	3719	2238	2184	2184
query89	396	316	283	283
query90	1918	188	191	188
query91	125	131	110	110
query92	68	61	57	57
query93	1796	1008	570	570
query94	662	403	294	294
query95	358	259	259	259
query96	483	554	263	263
query97	2796	2871	2756	2756
query98	251	206	201	201
query99	1652	1403	1294	1294
Total cold run time: 273604 ms
Total hot run time: 183362 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.99 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5a7e3ae690f39588af274d638a58b7452d7da0b1, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.11	0.11
query5	0.40	0.42	0.40
query6	1.19	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.52	0.52
query10	0.58	0.59	0.57
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.62
query14	2.71	2.71	2.74
query15	0.92	0.86	0.86
query16	0.37	0.38	0.38
query17	1.01	0.99	1.06
query18	0.21	0.19	0.19
query19	1.92	1.76	2.01
query20	0.02	0.01	0.01
query21	15.35	0.90	0.53
query22	0.75	1.15	0.66
query23	14.98	1.43	0.61
query24	6.65	2.55	0.36
query25	0.34	0.14	0.19
query26	0.74	0.17	0.14
query27	0.06	0.05	0.05
query28	9.20	0.93	0.43
query29	12.56	3.92	3.27
query30	0.25	0.09	0.06
query31	2.82	0.58	0.38
query32	3.22	0.53	0.46
query33	3.01	3.01	3.07
query34	15.80	5.14	4.51
query35	4.52	4.45	4.52
query36	0.68	0.50	0.47
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.04	0.02	0.02
query40	0.15	0.13	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 104.27 s
Total hot run time: 29.99 s

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.91% (17561/26645)
Region Coverage: 65.41% (8659/13239)
Branch Coverage: 55.27% (4667/8444)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6f640e67030d7bd4597fe6bab6997f1eecbbc926_6f640e67030d7bd4597fe6bab6997f1eecbbc926_cloud/report/index.html

@kaijchen kaijchen force-pushed the CIR-14357 branch 2 times, most recently from 9db7fd2 to 2d4e97b Compare February 11, 2025 08:21
@kaijchen
Copy link
Member Author

run buildall

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.38% (1061/1288)
Line Coverage: 65.85% (17545/26645)
Region Coverage: 65.35% (8652/13239)
Branch Coverage: 55.20% (4661/8444)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c386906e3e3bd00a5566a2862d4d2aa85066cee9_c386906e3e3bd00a5566a2862d4d2aa85066cee9_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 31237 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c386906e3e3bd00a5566a2862d4d2aa85066cee9, data reload: false

------ Round 1 ----------------------------------
q1	17589	5388	5049	5049
q2	2042	309	175	175
q3	10394	1303	708	708
q4	10219	992	530	530
q5	7518	2427	2334	2334
q6	194	168	137	137
q7	896	737	611	611
q8	9292	1347	1051	1051
q9	4860	4503	4815	4503
q10	6819	2329	1886	1886
q11	479	275	258	258
q12	343	345	213	213
q13	17776	3668	3060	3060
q14	227	228	209	209
q15	507	471	440	440
q16	627	620	588	588
q17	574	880	332	332
q18	6575	6279	6189	6189
q19	1210	947	553	553
q20	313	321	186	186
q21	2767	2197	1913	1913
q22	356	318	312	312
Total cold run time: 101577 ms
Total hot run time: 31237 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5119	5131	5126	5126
q2	235	329	229	229
q3	2186	2695	2327	2327
q4	1419	1814	1350	1350
q5	4234	4148	4192	4148
q6	204	162	122	122
q7	1885	1815	1664	1664
q8	2613	2658	2543	2543
q9	7301	7115	7115	7115
q10	2990	3227	2747	2747
q11	583	538	490	490
q12	717	823	647	647
q13	3519	3935	3250	3250
q14	278	328	294	294
q15	513	467	459	459
q16	632	680	645	645
q17	1127	1603	1350	1350
q18	7510	7358	7267	7267
q19	794	821	861	821
q20	1946	2026	1868	1868
q21	5287	4926	4876	4876
q22	603	582	583	582
Total cold run time: 51695 ms
Total hot run time: 49920 ms

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.81% (17584/26719)
Region Coverage: 65.35% (8667/13262)
Branch Coverage: 55.23% (4670/8456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43f86be3aa094674f57a839bd3b13e7394061a17_43f86be3aa094674f57a839bd3b13e7394061a17_cloud/report/index.html

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.97% (11307/26312)
Line Coverage: 32.89% (94919/288574)
Region Coverage: 32.05% (48669/151839)
Branch Coverage: 27.91% (24546/87956)
Coverage Report: http://coverage.selectdb-in.cc/coverage/43f86be3aa094674f57a839bd3b13e7394061a17_43f86be3aa094674f57a839bd3b13e7394061a17/report/index.html

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.78% (17577/26719)
Region Coverage: 65.32% (8663/13262)
Branch Coverage: 55.22% (4669/8456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eaad8d48f657ad676fc8e087d296520d407c2fb4_eaad8d48f657ad676fc8e087d296520d407c2fb4_cloud/report/index.html

@kaijchen
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.25% (1061/1290)
Line Coverage: 65.78% (17577/26719)
Region Coverage: 65.32% (8663/13262)
Branch Coverage: 55.19% (4667/8456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8d83444e64a17aa2759e7d9b96588cd46c26b124_8d83444e64a17aa2759e7d9b96588cd46c26b124_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 31799 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8d83444e64a17aa2759e7d9b96588cd46c26b124, data reload: false

------ Round 1 ----------------------------------
q1	17592	5303	5100	5100
q2	2045	311	166	166
q3	10429	1331	728	728
q4	10291	1003	571	571
q5	8457	2451	2334	2334
q6	212	167	134	134
q7	924	776	615	615
q8	9304	1274	1145	1145
q9	5277	4685	4801	4685
q10	6870	2311	1900	1900
q11	473	279	264	264
q12	355	360	219	219
q13	17766	3722	3092	3092
q14	244	225	215	215
q15	524	477	463	463
q16	614	619	575	575
q17	580	878	341	341
q18	6702	6145	6171	6145
q19	1338	962	577	577
q20	319	329	197	197
q21	2958	2297	2025	2025
q22	372	334	308	308
Total cold run time: 103646 ms
Total hot run time: 31799 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5366	5148	5127	5127
q2	240	330	235	235
q3	2188	2700	2309	2309
q4	1452	1841	1421	1421
q5	4259	4158	4161	4158
q6	204	162	124	124
q7	1883	1850	1806	1806
q8	2598	2575	2565	2565
q9	7171	7046	7102	7046
q10	3007	3205	2786	2786
q11	572	522	490	490
q12	689	777	622	622
q13	3520	3941	3284	3284
q14	291	293	273	273
q15	501	472	467	467
q16	631	683	644	644
q17	1161	1652	1275	1275
q18	7572	7397	7361	7361
q19	817	784	934	784
q20	2015	2023	1880	1880
q21	5432	5055	4919	4919
q22	649	610	552	552
Total cold run time: 52218 ms
Total hot run time: 50128 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183137 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8d83444e64a17aa2759e7d9b96588cd46c26b124, data reload: false

query1	969	380	370	370
query2	6513	1902	1908	1902
query3	6804	210	209	209
query4	26167	23356	23379	23356
query5	4343	663	490	490
query6	327	198	191	191
query7	4605	506	314	314
query8	291	240	235	235
query9	8617	2574	2574	2574
query10	453	344	250	250
query11	15353	15120	14985	14985
query12	158	112	105	105
query13	1658	513	413	413
query14	8698	6679	6125	6125
query15	210	197	173	173
query16	7360	637	445	445
query17	1207	713	554	554
query18	1947	409	310	310
query19	196	193	157	157
query20	127	113	115	113
query21	208	119	104	104
query22	4314	4166	4075	4075
query23	34016	33076	32981	32981
query24	7724	2348	2375	2348
query25	539	447	388	388
query26	1238	296	150	150
query27	2554	450	331	331
query28	4362	2418	2395	2395
query29	740	534	420	420
query30	232	180	156	156
query31	930	861	789	789
query32	78	61	65	61
query33	568	359	310	310
query34	797	839	503	503
query35	787	806	739	739
query36	986	981	878	878
query37	124	94	75	75
query38	4234	4202	4005	4005
query39	1616	1374	1418	1374
query40	208	114	103	103
query41	53	52	50	50
query42	122	104	104	104
query43	508	525	485	485
query44	1276	803	776	776
query45	180	171	159	159
query46	858	1024	637	637
query47	1754	1776	1717	1717
query48	370	407	293	293
query49	776	511	426	426
query50	698	741	410	410
query51	4157	4141	4155	4141
query52	105	107	98	98
query53	223	254	189	189
query54	497	482	407	407
query55	84	77	78	77
query56	267	272	247	247
query57	1118	1131	1063	1063
query58	256	239	247	239
query59	2662	2804	2628	2628
query60	276	286	252	252
query61	121	117	118	117
query62	810	720	661	661
query63	225	198	188	188
query64	4368	989	657	657
query65	3235	3149	3109	3109
query66	1149	403	313	313
query67	15916	15541	15346	15346
query68	7323	763	502	502
query69	490	297	259	259
query70	1239	1135	1111	1111
query71	404	300	270	270
query72	5875	3502	3673	3502
query73	732	714	353	353
query74	9236	9128	8883	8883
query75	3153	3159	2706	2706
query76	3175	1157	750	750
query77	471	374	292	292
query78	9913	10050	9255	9255
query79	1995	846	579	579
query80	599	520	454	454
query81	494	280	237	237
query82	198	136	94	94
query83	176	187	157	157
query84	232	87	70	70
query85	752	345	359	345
query86	375	284	290	284
query87	4515	4630	4351	4351
query88	3872	2215	2197	2197
query89	410	317	284	284
query90	1977	199	192	192
query91	132	137	107	107
query92	136	60	52	52
query93	1659	1014	577	577
query94	672	417	303	303
query95	358	264	258	258
query96	486	560	272	272
query97	2732	2836	2767	2767
query98	228	202	206	202
query99	1340	1412	1227	1227
Total cold run time: 269671 ms
Total hot run time: 183137 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8d83444e64a17aa2759e7d9b96588cd46c26b124, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.40	0.41	0.39
query6	1.17	0.67	0.65
query7	0.02	0.01	0.02
query8	0.04	0.02	0.03
query9	0.59	0.53	0.52
query10	0.57	0.57	0.56
query11	0.15	0.10	0.10
query12	0.14	0.10	0.11
query13	0.62	0.60	0.60
query14	2.68	2.73	2.85
query15	0.91	0.84	0.85
query16	0.38	0.37	0.38
query17	1.04	1.06	0.99
query18	0.21	0.20	0.20
query19	1.89	1.75	1.98
query20	0.02	0.01	0.01
query21	15.35	0.89	0.56
query22	0.76	1.16	0.93
query23	14.74	1.39	0.61
query24	8.72	3.99	0.54
query25	0.34	0.09	0.08
query26	0.98	0.19	0.13
query27	0.05	0.05	0.04
query28	6.27	0.80	0.44
query29	12.58	3.88	3.23
query30	0.26	0.09	0.06
query31	2.82	0.59	0.38
query32	3.24	0.54	0.47
query33	2.98	3.02	3.01
query34	15.65	5.09	4.52
query35	4.53	4.53	4.52
query36	0.68	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.25 s
Total hot run time: 30.36 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 42.98% (11309/26312)
Line Coverage: 32.90% (94951/288574)
Region Coverage: 32.06% (48687/151839)
Branch Coverage: 27.92% (24555/87956)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8d83444e64a17aa2759e7d9b96588cd46c26b124_8d83444e64a17aa2759e7d9b96588cd46c26b124/report/index.html

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2025

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit db78a11 into apache:master Mar 4, 2025
24 of 26 checks passed
@dataroaring dataroaring added the usercase Important user case type label label Mar 4, 2025
github-actions bot pushed a commit that referenced this pull request Mar 4, 2025
@gavinchou gavinchou added the p0_c label Mar 4, 2025
dataroaring pushed a commit that referenced this pull request Mar 10, 2025
…#48642)

Cherry-picked from #47703

Co-authored-by: Kaijie Chen <chenkaijie@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged p0_c reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants