[opt](multi-catalog) Optimize remote scan concurrency.

[kaka11chen]

What problem does this PR solve?

Problem Summary:

Release note

Cherry-pick #51415

Check List (For Author)

• Test

  • Regression test
  • Unit Test
  • Manual test (add detailed scripts or steps below)
  • No need to test or manual test. Explain why:
    • This is a refactor/code format and no logic has been changed.
    • Previous test can cover this change.
    • No code files have been changed.
    • Other reason

• Behavior changed:

  • No.
  • Yes.

• Does this need documentation?

  • No.
  • Yes.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[Thearas]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[kaka11chen]

run buildall

[doris-robot]

TPC-H: Total hot run time: 39660 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b52684a1b7fec5cc3ebe6e07dcaa0895f129ff6, data reload: false

------ Round 1 ----------------------------------
q1	17929	7152	6575	6575
q2	2074	170	162	162
q3	10955	1085	1200	1085
q4	10525	773	730	730
q5	7731	2890	2763	2763
q6	217	139	138	138
q7	971	648	606	606
q8	9355	1964	2030	1964
q9	6640	6430	6433	6430
q10	6996	2266	2267	2266
q11	468	261	267	261
q12	403	215	222	215
q13	17768	3053	2996	2996
q14	241	201	203	201
q15	499	471	476	471
q16	505	393	379	379
q17	996	554	611	554
q18	7388	6602	6514	6514
q19	1399	1131	1070	1070
q20	495	204	203	203
q21	3995	3086	3147	3086
q22	1124	1011	991	991
Total cold run time: 108674 ms
Total hot run time: 39660 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6640	6558	6606	6558
q2	327	239	232	232
q3	2947	2772	2963	2772
q4	2070	1859	1799	1799
q5	5751	5708	5783	5708
q6	209	127	127	127
q7	2193	1818	1811	1811
q8	3347	3552	3534	3534
q9	8978	8735	8926	8735
q10	3545	3527	3517	3517
q11	585	507	492	492
q12	813	592	599	592
q13	9965	3148	3172	3148
q14	304	270	265	265
q15	510	463	468	463
q16	491	428	454	428
q17	1847	1630	1592	1592
q18	8321	7973	7837	7837
q19	1684	1648	1551	1551
q20	2160	1816	1846	1816
q21	5092	5127	5084	5084
q22	1176	1057	1033	1033
Total cold run time: 68955 ms
Total hot run time: 59094 ms

[doris-robot]

TPC-DS: Total hot run time: 196528 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b52684a1b7fec5cc3ebe6e07dcaa0895f129ff6, data reload: false

query1	1302	925	889	889
query2	6339	2047	1935	1935
query3	10912	4516	4425	4425
query4	61893	29614	23159	23159
query5	5263	442	464	442
query6	407	189	186	186
query7	5523	321	312	312
query8	313	221	219	219
query9	8639	2575	2590	2575
query10	463	272	259	259
query11	17654	15205	15808	15205
query12	176	103	101	101
query13	1463	446	425	425
query14	11083	6865	7284	6865
query15	204	187	175	175
query16	7099	486	502	486
query17	1147	608	595	595
query18	1904	327	316	316
query19	209	166	174	166
query20	117	116	107	107
query21	207	101	101	101
query22	4732	4465	4634	4465
query23	34871	34444	34352	34352
query24	6232	2937	2990	2937
query25	538	406	427	406
query26	657	178	173	173
query27	1834	375	368	368
query28	4163	2177	2155	2155
query29	750	478	433	433
query30	244	155	154	154
query31	971	800	828	800
query32	64	53	57	53
query33	411	299	304	299
query34	929	532	518	518
query35	864	741	732	732
query36	1098	962	965	962
query37	116	73	68	68
query38	4096	3964	4005	3964
query39	1535	1463	1439	1439
query40	204	96	103	96
query41	48	47	48	47
query42	112	102	101	101
query43	533	487	499	487
query44	1245	844	843	843
query45	186	171	169	169
query46	1152	732	733	732
query47	2013	1915	1904	1904
query48	474	398	394	394
query49	720	406	397	397
query50	866	448	437	437
query51	7509	7224	7147	7147
query52	105	94	91	91
query53	273	191	196	191
query54	576	466	480	466
query55	78	75	77	75
query56	260	244	259	244
query57	1337	1224	1175	1175
query58	222	205	207	205
query59	3287	3113	3062	3062
query60	266	247	248	247
query61	108	107	121	107
query62	757	666	686	666
query63	215	186	188	186
query64	1347	666	646	646
query65	3243	3175	3221	3175
query66	692	294	295	294
query67	15857	15557	15375	15375
query68	4291	599	581	581
query69	430	263	265	263
query70	1145	1018	1064	1018
query71	357	259	252	252
query72	6392	4237	4056	4056
query73	747	346	369	346
query74	10493	9016	9060	9016
query75	3300	2663	2636	2636
query76	2057	978	1123	978
query77	497	269	283	269
query78	10610	9549	9550	9549
query79	2140	600	618	600
query80	1361	422	430	422
query81	512	222	222	222
query82	1300	88	87	87
query83	157	147	141	141
query84	287	79	80	79
query85	1010	306	289	289
query86	363	292	293	292
query87	4472	4284	4238	4238
query88	4054	2380	2374	2374
query89	413	291	294	291
query90	1987	187	181	181
query91	176	147	147	147
query92	60	50	50	50
query93	2839	567	561	561
query94	780	294	290	290
query95	358	264	258	258
query96	623	282	281	281
query97	3330	3110	3232	3110
query98	210	210	189	189
query99	1647	1293	1282	1282
Total cold run time: 318505 ms
Total hot run time: 196528 ms

[doris-robot]

ClickBench: Total hot run time: 29.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b52684a1b7fec5cc3ebe6e07dcaa0895f129ff6, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.04
query3	0.23	0.06	0.07
query4	1.62	0.11	0.11
query5	0.52	0.51	0.52
query6	1.16	0.73	0.73
query7	0.02	0.04	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.52
query10	0.55	0.56	0.55
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.61
query14	0.77	0.78	0.83
query15	0.84	0.84	0.83
query16	0.38	0.38	0.37
query17	1.07	1.06	1.01
query18	0.23	0.23	0.21
query19	1.89	1.83	1.74
query20	0.01	0.01	0.01
query21	15.45	0.61	0.56
query22	2.14	2.48	1.79
query23	17.02	1.01	0.85
query24	3.34	0.77	0.99
query25	0.14	0.08	0.10
query26	0.50	0.13	0.13
query27	0.05	0.07	0.06
query28	10.56	0.51	0.48
query29	12.59	3.27	3.24
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.23	0.48	0.45
query33	2.99	2.96	3.06
query34	16.96	4.55	4.61
query35	4.54	4.51	4.62
query36	0.68	0.48	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.72 s
Total hot run time: 29.95 s

[morningman]

run buildall

[doris-robot]

TPC-H: Total hot run time: 39444 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b19cb3019589435104ef5433c4eca4c893dc24a3, data reload: false

------ Round 1 ----------------------------------
q1	17730	7100	6570	6570
q2	2068	169	170	169
q3	10816	1070	1210	1070
q4	10536	778	682	682
q5	7724	2866	2834	2834
q6	209	132	131	131
q7	975	627	610	610
q8	9348	1924	2001	1924
q9	6564	6385	6374	6374
q10	6990	2266	2228	2228
q11	461	264	259	259
q12	404	214	209	209
q13	17806	2982	2986	2982
q14	223	200	207	200
q15	501	456	459	456
q16	447	379	376	376
q17	967	645	511	511
q18	7186	6712	6663	6663
q19	1399	1045	1039	1039
q20	463	209	203	203
q21	4002	3048	2987	2987
q22	1088	967	978	967
Total cold run time: 107907 ms
Total hot run time: 39444 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6584	6579	6524	6524
q2	330	232	233	232
q3	2921	2720	2841	2720
q4	2074	1801	1857	1801
q5	5750	5733	5705	5705
q6	210	128	136	128
q7	2213	1782	1823	1782
q8	3363	3512	3545	3512
q9	8954	8691	8913	8691
q10	3546	3488	3507	3488
q11	587	496	508	496
q12	793	588	592	588
q13	6643	3179	3174	3174
q14	310	280	282	280
q15	510	468	462	462
q16	508	429	442	429
q17	1821	1637	1605	1605
q18	8405	7713	7785	7713
q19	1693	1484	1546	1484
q20	2121	1917	1792	1792
q21	5026	5033	5122	5033
q22	1115	1050	1042	1042
Total cold run time: 65477 ms
Total hot run time: 58681 ms

[doris-robot]

TPC-DS: Total hot run time: 196240 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b19cb3019589435104ef5433c4eca4c893dc24a3, data reload: false

query1	1269	904	918	904
query2	6309	1889	1850	1850
query3	10846	4289	4206	4206
query4	61305	29611	23333	23333
query5	5201	478	461	461
query6	407	193	189	189
query7	5479	316	311	311
query8	319	231	228	228
query9	8448	2551	2550	2550
query10	462	262	259	259
query11	17989	15254	15570	15254
query12	177	107	115	107
query13	1433	434	447	434
query14	10685	7957	6949	6949
query15	210	189	177	177
query16	7112	509	479	479
query17	1232	625	580	580
query18	1876	333	325	325
query19	213	169	180	169
query20	126	112	112	112
query21	209	103	104	103
query22	4654	4372	4569	4372
query23	34848	34447	34136	34136
query24	6132	2965	2956	2956
query25	554	431	446	431
query26	664	168	170	168
query27	1749	354	353	353
query28	3761	2182	2105	2105
query29	675	433	425	425
query30	233	155	155	155
query31	990	805	822	805
query32	69	60	58	58
query33	432	306	295	295
query34	927	505	525	505
query35	855	701	730	701
query36	1104	984	970	970
query37	113	67	76	67
query38	4074	3997	3916	3916
query39	1532	1485	1473	1473
query40	210	98	98	98
query41	49	51	48	48
query42	113	103	101	101
query43	540	493	487	487
query44	1222	833	835	833
query45	182	171	169	169
query46	1171	761	723	723
query47	2011	1891	1917	1891
query48	479	397	386	386
query49	710	408	384	384
query50	860	434	435	434
query51	7441	7372	7253	7253
query52	98	89	89	89
query53	268	200	187	187
query54	581	451	462	451
query55	84	82	79	79
query56	260	255	258	255
query57	1298	1186	1178	1178
query58	231	214	217	214
query59	3223	3006	2939	2939
query60	298	265	261	261
query61	112	157	112	112
query62	818	715	694	694
query63	230	210	197	197
query64	1393	686	653	653
query65	3297	3204	3165	3165
query66	686	289	297	289
query67	16061	15626	15498	15498
query68	4276	586	580	580
query69	435	267	268	267
query70	1168	1153	1103	1103
query71	351	254	251	251
query72	6404	4256	3951	3951
query73	752	347	347	347
query74	10512	8923	9176	8923
query75	3388	2646	2614	2614
query76	1956	1185	1098	1098
query77	525	285	287	285
query78	10785	9641	9523	9523
query79	2141	598	626	598
query80	1318	433	423	423
query81	497	223	220	220
query82	610	86	86	86
query83	163	147	142	142
query84	288	82	79	79
query85	974	338	299	299
query86	382	264	295	264
query87	4425	4264	4307	4264
query88	3650	2380	2333	2333
query89	424	289	300	289
query90	2015	187	188	187
query91	182	152	147	147
query92	60	49	51	49
query93	2504	566	563	563
query94	786	274	284	274
query95	364	262	266	262
query96	632	280	280	280
query97	3277	3130	3164	3130
query98	214	204	198	198
query99	1546	1272	1286	1272
Total cold run time: 315603 ms
Total hot run time: 196240 ms

[doris-robot]

BE UT Coverage Report

Increment line coverage 72.73% (8/11) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	41.24% (10944/26539)
Line Coverage	32.01% (93676/292634)
Region Coverage	31.11% (48317/155294)
Branch Coverage	27.56% (24743/89778)

[doris-robot]

ClickBench: Total hot run time: 29.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b19cb3019589435104ef5433c4eca4c893dc24a3, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.64	0.11	0.11
query5	0.51	0.49	0.50
query6	1.13	0.73	0.74
query7	0.02	0.01	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.51
query10	0.55	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.59
query14	0.77	0.79	0.81
query15	0.84	0.82	0.81
query16	0.37	0.38	0.39
query17	0.95	0.98	1.05
query18	0.22	0.21	0.21
query19	1.93	1.79	1.78
query20	0.02	0.01	0.01
query21	15.39	0.60	0.59
query22	2.14	1.57	1.49
query23	17.01	1.17	0.73
query24	3.22	1.26	2.07
query25	0.17	0.09	0.05
query26	0.72	0.14	0.13
query27	0.05	0.04	0.04
query28	9.05	0.50	0.52
query29	12.58	3.20	3.11
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.23	0.48	0.45
query33	2.98	2.98	3.04
query34	16.94	4.51	4.47
query35	4.52	4.55	4.53
query36	0.63	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.07	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.03 s
Total hot run time: 29.85 s

[dataroaring]

LGTM