[enhance](meta-service) add real request ip for be rpc #53114

koarz · 2025-07-11T08:44:10Z

What problem does this PR solve?

Currently, when sending an RPC request to MS, it may go through LB. At this point, MS begins RPC output with LB's IP, which makes it difficult for us to know where the RPC request actually came from, making troubleshooting difficult. Now, add a request IP field to all requests and set it when sending RPC from BE. This way, even if it goes through LB, we can still obtain the original BE IP data.

Thearas · 2025-07-11T08:44:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

koarz · 2025-07-11T09:36:21Z

run buildall

hello-stephen · 2025-07-11T10:00:38Z

Cloud UT Coverage Report

Increment line coverage 100.00% (24/24) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.77% (1270/1592)
Line Coverage	65.40% (21297/32565)
Region Coverage	66.85% (10774/16116)
Branch Coverage	56.50% (5674/10042)

hello-stephen · 2025-07-11T10:50:29Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-11T11:06:18Z

TPC-H: Total hot run time: 33139 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18736821952dba1655ea3a39bcec32d24ce651b1, data reload: false

------ Round 1 ----------------------------------
q1	17610	5179	5022	5022
q2	1944	299	195	195
q3	10383	1324	706	706
q4	10238	1023	510	510
q5	7712	2495	2282	2282
q6	183	163	129	129
q7	904	755	624	624
q8	9325	1286	1081	1081
q9	7539	5151	5140	5140
q10	6893	2355	1964	1964
q11	500	288	271	271
q12	344	349	224	224
q13	17754	3682	3024	3024
q14	219	232	211	211
q15	530	471	484	471
q16	431	420	375	375
q17	605	900	394	394
q18	7562	7204	7097	7097
q19	1608	943	579	579
q20	330	354	228	228
q21	3681	3191	2314	2314
q22	356	315	298	298
Total cold run time: 106651 ms
Total hot run time: 33139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5123	5073	5086	5073
q2	236	319	220	220
q3	2178	2698	2283	2283
q4	1351	1774	1340	1340
q5	4237	4445	4602	4445
q6	218	168	126	126
q7	2011	1971	1762	1762
q8	2622	2631	2614	2614
q9	7265	7537	7155	7155
q10	3030	3350	2874	2874
q11	584	504	500	500
q12	682	894	750	750
q13	3515	4006	3313	3313
q14	269	312	283	283
q15	537	488	478	478
q16	443	494	423	423
q17	1280	1575	1395	1395
q18	7977	7684	7608	7608
q19	840	812	879	812
q20	2027	2097	1911	1911
q21	5154	4522	4528	4522
q22	645	628	565	565
Total cold run time: 52224 ms
Total hot run time: 50452 ms

cloud/src/meta-service/meta_service_helper.h

doris-robot · 2025-07-11T11:17:59Z

TPC-DS: Total hot run time: 187412 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18736821952dba1655ea3a39bcec32d24ce651b1, data reload: false

query1	1006	398	384	384
query2	6518	1684	1663	1663
query3	6742	211	216	211
query4	26349	23246	23535	23246
query5	4351	576	438	438
query6	301	208	209	208
query7	4622	491	287	287
query8	255	226	213	213
query9	8592	2645	2658	2645
query10	493	334	285	285
query11	16078	14947	14787	14787
query12	163	108	100	100
query13	1656	532	390	390
query14	8733	6001	6029	6001
query15	219	184	164	164
query16	7137	426	250	250
query17	1351	707	558	558
query18	1971	398	291	291
query19	185	189	164	164
query20	118	116	113	113
query21	215	122	105	105
query22	4109	4157	4565	4157
query23	34831	33972	33626	33626
query24	8515	2383	2366	2366
query25	560	481	414	414
query26	1244	264	144	144
query27	2759	519	340	340
query28	4270	2141	2122	2122
query29	786	560	446	446
query30	278	220	196	196
query31	898	833	750	750
query32	72	66	60	60
query33	544	349	283	283
query34	786	830	510	510
query35	582	633	551	551
query36	939	1007	891	891
query37	110	98	74	74
query38	4222	4064	4120	4064
query39	1476	1413	1427	1413
query40	205	115	104	104
query41	54	53	48	48
query42	123	108	105	105
query43	502	506	483	483
query44	1314	834	818	818
query45	175	181	162	162
query46	827	1011	632	632
query47	1735	1800	1728	1728
query48	382	420	304	304
query49	741	499	399	399
query50	634	674	415	415
query51	5526	5514	5501	5501
query52	106	105	96	96
query53	216	246	183	183
query54	569	566	498	498
query55	82	77	80	77
query56	302	314	276	276
query57	1171	1181	1115	1115
query58	263	248	245	245
query59	2607	2613	2503	2503
query60	317	304	291	291
query61	125	121	120	120
query62	802	714	646	646
query63	220	179	179	179
query64	4450	1237	818	818
query65	4294	4195	4185	4185
query66	1131	419	310	310
query67	15766	15381	15222	15222
query68	8929	896	525	525
query69	506	312	268	268
query70	1247	1104	1082	1082
query71	466	303	290	290
query72	5594	4786	4798	4786
query73	718	605	344	344
query74	8996	8946	8702	8702
query75	4285	3175	2680	2680
query76	3752	1142	728	728
query77	789	369	300	300
query78	10849	11092	10293	10293
query79	1742	903	579	579
query80	697	517	440	440
query81	460	253	220	220
query82	424	130	96	96
query83	290	253	237	237
query84	291	103	82	82
query85	779	364	326	326
query86	342	310	290	290
query87	4424	4423	4366	4366
query88	2864	2302	2267	2267
query89	389	314	281	281
query90	1922	214	210	210
query91	141	139	111	111
query92	77	57	58	57
query93	1102	967	604	604
query94	719	315	196	196
query95	376	308	289	289
query96	489	566	280	280
query97	2745	2752	2635	2635
query98	237	207	207	207
query99	1439	1429	1268	1268
Total cold run time: 276755 ms
Total hot run time: 187412 ms

gavinchou · 2025-07-11T11:19:10Z

be/src/cloud/cloud_meta_mgr.cpp

    static_assert(std::is_base_of_v<::google::protobuf::Message, Request>);
    static_assert(std::is_base_of_v<::google::protobuf::Message, Response>);

+    const_cast<Request&>(req).set_request_ip(BackendOptions::get_be_endpoint());


what about BackendOptions::get_local_backend()

get_local_backend return a TBackend, get_be_endpoint returned ‘local_host+heartbeat’ is sufficient to determine which be it is.

doris-robot · 2025-07-11T11:23:18Z

ClickBench: Total hot run time: 29.64 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18736821952dba1655ea3a39bcec32d24ce651b1, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.25	0.07	0.07
query4	1.62	0.11	0.11
query5	0.45	0.42	0.42
query6	1.15	0.65	0.64
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.62	0.51	0.51
query10	0.58	0.56	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.64	0.61	0.61
query14	0.80	0.83	0.82
query15	0.91	0.90	0.88
query16	0.39	0.39	0.39
query17	1.07	1.06	1.05
query18	0.23	0.22	0.21
query19	1.94	1.93	1.84
query20	0.02	0.01	0.01
query21	15.38	0.89	0.53
query22	0.77	1.26	0.67
query23	14.93	1.42	0.58
query24	7.42	1.26	0.85
query25	0.55	0.10	0.26
query26	0.63	0.16	0.14
query27	0.06	0.05	0.06
query28	9.89	0.91	0.45
query29	12.53	3.94	3.30
query30	0.26	0.09	0.06
query31	2.84	0.62	0.39
query32	3.24	0.57	0.47
query33	3.07	3.12	3.20
query34	16.05	5.42	4.79
query35	4.80	4.86	4.83
query36	0.69	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.05	0.04	0.04
Total cold run time: 104.79 s
Total hot run time: 29.64 s

doris-robot · 2025-07-11T12:33:00Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.46% (15614/27176)
Line Coverage	46.17% (139333/301757)
Region Coverage	35.53% (104241/293349)
Branch Coverage	38.04% (45932/120743)

koarz · 2025-07-14T02:34:55Z

run buildall

doris-robot · 2025-07-14T02:57:22Z

Cloud UT Coverage Report

Increment line coverage 100.00% (30/30) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	79.77% (1270/1592)
Line Coverage	65.40% (21300/32568)
Region Coverage	66.87% (10777/16116)
Branch Coverage	56.52% (5676/10042)

hello-stephen · 2025-07-14T03:45:37Z

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-07-14T04:20:40Z

TPC-H: Total hot run time: 33385 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 72f4542d9f216c8ddeee03c99e36c86c9153d8bf, data reload: false

------ Round 1 ----------------------------------
q1	17643	5286	5086	5086
q2	1972	295	178	178
q3	10505	1282	722	722
q4	10250	1025	505	505
q5	8085	2540	2378	2378
q6	184	164	131	131
q7	908	753	605	605
q8	9322	1298	1068	1068
q9	6889	5131	5110	5110
q10	6894	2372	1992	1992
q11	476	292	277	277
q12	348	352	214	214
q13	17774	3694	3092	3092
q14	229	228	219	219
q15	545	481	476	476
q16	419	441	375	375
q17	584	881	367	367
q18	7882	7158	7167	7158
q19	1787	960	575	575
q20	328	350	223	223
q21	3681	3140	2341	2341
q22	360	317	293	293
Total cold run time: 107065 ms
Total hot run time: 33385 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5185	5096	5094	5094
q2	240	341	223	223
q3	2192	2722	2265	2265
q4	1350	1752	1311	1311
q5	4260	4468	4486	4468
q6	218	171	128	128
q7	2167	2003	1874	1874
q8	2690	2558	2530	2530
q9	7363	7270	7199	7199
q10	3156	3294	2922	2922
q11	578	528	514	514
q12	676	800	628	628
q13	3650	4045	3442	3442
q14	310	305	281	281
q15	516	483	480	480
q16	438	481	438	438
q17	1180	1530	1463	1463
q18	7900	7787	7126	7126
q19	762	731	840	731
q20	1898	1997	1811	1811
q21	4821	4419	4261	4261
q22	617	577	534	534
Total cold run time: 52167 ms
Total hot run time: 49723 ms

doris-robot · 2025-07-14T04:32:22Z

TPC-DS: Total hot run time: 187288 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 72f4542d9f216c8ddeee03c99e36c86c9153d8bf, data reload: false

query1	1016	391	382	382
query2	6506	1711	1644	1644
query3	6740	221	217	217
query4	26158	23238	23338	23238
query5	4374	587	444	444
query6	311	228	211	211
query7	4629	503	299	299
query8	256	228	221	221
query9	8630	2618	2618	2618
query10	473	343	276	276
query11	15901	15001	14785	14785
query12	165	111	103	103
query13	1659	525	413	413
query14	9515	6117	6019	6019
query15	212	195	177	177
query16	7662	439	270	270
query17	1331	732	603	603
query18	2023	404	315	315
query19	203	198	169	169
query20	130	115	113	113
query21	214	128	109	109
query22	4056	4039	4012	4012
query23	34329	33280	33161	33161
query24	8400	2422	2405	2405
query25	535	460	390	390
query26	1229	267	152	152
query27	2719	506	354	354
query28	4322	2129	2099	2099
query29	732	561	462	462
query30	286	224	190	190
query31	943	824	778	778
query32	68	63	62	62
query33	570	332	294	294
query34	796	850	528	528
query35	616	627	565	565
query36	961	1007	898	898
query37	116	99	75	75
query38	4134	4218	4176	4176
query39	1519	1422	1405	1405
query40	204	117	109	109
query41	57	59	52	52
query42	128	111	114	111
query43	519	506	496	496
query44	1359	825	830	825
query45	177	190	168	168
query46	866	1016	637	637
query47	1740	1763	1710	1710
query48	386	415	297	297
query49	745	479	390	390
query50	654	685	424	424
query51	5505	5484	5560	5484
query52	107	105	96	96
query53	228	259	186	186
query54	585	579	500	500
query55	83	79	83	79
query56	326	301	330	301
query57	1161	1168	1152	1152
query58	269	261	253	253
query59	2530	2684	2558	2558
query60	317	319	294	294
query61	128	122	122	122
query62	784	744	645	645
query63	230	186	192	186
query64	4397	1192	838	838
query65	4353	4200	4160	4160
query66	1062	447	345	345
query67	15594	15687	15293	15293
query68	8502	896	526	526
query69	504	319	272	272
query70	1273	1134	1108	1108
query71	470	331	315	315
query72	5540	4689	4780	4689
query73	727	629	354	354
query74	9413	9075	8634	8634
query75	3803	3191	2725	2725
query76	3577	1217	741	741
query77	770	418	305	305
query78	10892	10939	10276	10276
query79	2402	848	574	574
query80	661	530	441	441
query81	463	260	221	221
query82	487	195	96	96
query83	282	253	237	237
query84	300	109	89	89
query85	793	375	310	310
query86	388	297	297	297
query87	4405	4434	4373	4373
query88	3622	2326	2274	2274
query89	382	321	281	281
query90	1913	219	209	209
query91	224	144	114	114
query92	81	62	58	58
query93	1652	979	591	591
query94	681	314	202	202
query95	382	294	278	278
query96	490	555	278	278
query97	2731	2801	2635	2635
query98	230	211	200	200
query99	1490	1411	1242	1242
Total cold run time: 278463 ms
Total hot run time: 187288 ms

doris-robot · 2025-07-14T04:37:44Z

ClickBench: Total hot run time: 30.37 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 72f4542d9f216c8ddeee03c99e36c86c9153d8bf, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.06	0.05
query3	0.30	0.06	0.06
query4	1.60	0.09	0.08
query5	0.43	0.42	0.42
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.06	0.06	0.05
query9	0.65	0.55	0.52
query10	0.59	0.57	0.58
query11	0.26	0.13	0.13
query12	0.24	0.14	0.16
query13	0.65	0.63	0.64
query14	0.81	0.82	0.82
query15	1.00	0.91	0.91
query16	0.38	0.39	0.38
query17	1.10	1.07	1.10
query18	0.25	0.23	0.23
query19	1.96	1.85	1.90
query20	0.01	0.02	0.01
query21	15.35	1.00	0.68
query22	0.93	1.02	0.80
query23	14.70	1.60	0.78
query24	5.42	0.59	0.29
query25	0.17	0.09	0.09
query26	0.56	0.22	0.18
query27	0.09	0.09	0.08
query28	11.06	1.24	0.60
query29	12.55	4.10	3.44
query30	0.29	0.09	0.06
query31	2.83	0.64	0.44
query32	3.23	0.61	0.51
query33	3.20	3.12	3.12
query34	16.58	5.46	4.70
query35	4.77	4.79	4.82
query36	0.67	0.52	0.50
query37	0.20	0.18	0.18
query38	0.16	0.18	0.16
query39	0.06	0.05	0.04
query40	0.20	0.17	0.16
query41	0.11	0.05	0.06
query42	0.06	0.05	0.06
query43	0.06	0.05	0.04
Total cold run time: 104.87 s
Total hot run time: 30.37 s

hello-stephen · 2025-07-14T05:29:38Z

BE UT Coverage Report

Increment line coverage 0.00% (0/1) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	57.45% (15615/27182)
Line Coverage	46.16% (139376/301968)
Region Coverage	35.49% (104177/293538)
Branch Coverage	38.03% (45940/120798)

koarz · 2025-07-14T07:39:03Z

run vault_p0

github-actions · 2025-07-14T09:21:49Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-07-14T09:21:51Z

PR approved by anyone and no changes requested.

…3114 (#53319) pick #53114

. (#53320) pick #53114

koarz added 2 commits July 11, 2025 14:42

add be ip for ms begin rpc log

cff2fa8

be ip

928cd83

koarz requested review from dataroaring, gavinchou and w41ter as code owners July 11, 2025 08:44

fix

1873682

gavinchou reviewed Jul 11, 2025

View reviewed changes

cloud/src/meta-service/meta_service_helper.h Outdated Show resolved Hide resolved

gavinchou reviewed Jul 11, 2025

View reviewed changes

meaningful log str

72f4542

gavinchou approved these changes Jul 14, 2025

View reviewed changes

gavinchou added dev/3.0.x dev/3.1.x cloud labels Jul 14, 2025

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 14, 2025

github-actions bot added the reviewed label Jul 14, 2025

Yukang-Lian approved these changes Jul 14, 2025

View reviewed changes

dataroaring merged commit 41119c7 into apache:master Jul 14, 2025
27 of 30 checks passed

github-actions bot added dev/3.0.x-conflict dev/3.1.x-conflict labels Jul 14, 2025

koarz deleted the ms_log_be_ip branch July 15, 2025 10:41

koarz added a commit to koarz/doris that referenced this pull request Jul 15, 2025

[enhance](meta-service) add real request ip for be rpc apache#53114

ea988a6

koarz added a commit to koarz/doris that referenced this pull request Jul 15, 2025

[enhance](meta-service) add real request ip for be rpc apache#53114.

4c5488c

This was referenced Jul 15, 2025

branch-3.1 [enhance](meta-service) add real request ip for be rpc #53114 #53319

Merged

branch-3.0 [enhance](meta-service) add real request ip for be rpc #53114. #53320

Merged

morrySnow pushed a commit that referenced this pull request Jul 16, 2025

branch-3.1: [enhance](meta-service) add real request ip for be rpc #5…

83e701e

…3114 (#53319) pick #53114

morrySnow added dev/3.1.0-merged and removed dev/3.1.x dev/3.1.x-conflict labels Jul 16, 2025

dataroaring pushed a commit that referenced this pull request Jul 17, 2025

branch-3.0 [enhance](meta-service) add real request ip for be rpc #53114

2e0cd89

. (#53320) pick #53114

dataroaring added dev/3.0.7-merged and removed dev/3.0.x dev/3.0.x-conflict labels Jul 17, 2025

gavinchou mentioned this pull request Aug 15, 2025

Release Note 3.0.7 #54868

Open

[enhance](meta-service) add real request ip for be rpc #53114

[enhance](meta-service) add real request ip for be rpc #53114

Uh oh!

Conversation

koarz commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Uh oh!

Thearas commented Jul 11, 2025

Uh oh!

koarz commented Jul 11, 2025

Uh oh!

hello-stephen commented Jul 11, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 11, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

gavinchou Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

koarz Jul 14, 2025

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Jul 11, 2025

Uh oh!

doris-robot commented Jul 11, 2025

BE UT Coverage Report

Uh oh!

koarz commented Jul 14, 2025

Uh oh!

doris-robot commented Jul 14, 2025

Cloud UT Coverage Report

Uh oh!

hello-stephen commented Jul 14, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Jul 14, 2025

Uh oh!

doris-robot commented Jul 14, 2025

Uh oh!

doris-robot commented Jul 14, 2025

Uh oh!

hello-stephen commented Jul 14, 2025

BE UT Coverage Report

Uh oh!

koarz commented Jul 14, 2025

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

github-actions bot commented Jul 14, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

koarz commented Jul 11, 2025 •

edited

Loading