Skip to content

branch-3.0: [fix](meta) fix timeout for syncing image when FE scales out #51120 #51148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 24, 2025
Merged

branch-3.0: [fix](meta) fix timeout for syncing image when FE scales out #51120 #51148

dataroaring merged 1 commit into from
May 24, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 22, 2025

Cherry-picked from #51120

@yagagagaga
[fix](meta) fix timeout for syncing image when FE scales out (#51120)
0597a07 
Related PR: #26003

Problem Summary:

> If image file is very huge, it maybe occurs http Read timed out when
pulling image from FE Master

```java
java.io.IOException: java.net.SocketTimeoutException: Read timed out
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1664)
        at org.apache.doris.catalog.Env.getClusterIdAndRole(Env.java:1126)
        at org.apache.doris.catalog.Env.initialize(Env.java:884)
        at org.apache.doris.DorisFE.start(DorisFE.java:163)
        at org.apache.doris.DorisFE.main(DorisFE.java:83)
Caused by: java.net.SocketTimeoutException: Read timed out
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun. reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1952)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1947)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.iava:1946)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1516)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at org.apache.doris.master.MetaHelper.getResponse(MetaHelper.java:130)
        at org.apache.doris.master.MetaHelper.getRemoteFile(MetaHelper.java:98)
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1657)
        ... 4 more
```
@github-actions github-actions bot requested a review from dataroaring as a code owner May 22, 2025 06:15
@hello-stephen
Copy link
Contributor

hello-stephen commented May 22, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this May 22, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented May 22, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 22, 2025

TPC-H: Total hot run time: 39621 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0597a073181c6d3c2b199b17620b8e4adeb3ed95, data reload: false

------ Round 1 ----------------------------------
q1	17565	6646	6558	6558
q2	2071	167	159	159
q3	10611	1067	1200	1067
q4	10572	764	779	764
q5	7736	2856	2806	2806
q6	216	138	131	131
q7	948	625	593	593
q8	9350	1930	2008	1930
q9	6568	6373	6402	6373
q10	7000	2280	2294	2280
q11	465	280	255	255
q12	409	212	203	203
q13	17764	2960	2954	2954
q14	228	208	214	208
q15	506	457	458	457
q16	474	375	371	371
q17	966	595	518	518
q18	7195	6657	6590	6590
q19	1391	1098	977	977
q20	470	206	193	193
q21	3944	3410	3230	3230
q22	1106	1022	1004	1004
Total cold run time: 107555 ms
Total hot run time: 39621 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6531	6560	6547	6547
q2	333	241	236	236
q3	2972	2827	2863	2827
q4	2098	1869	1811	1811
q5	5877	5840	5816	5816
q6	207	129	130	129
q7	2253	1826	1780	1780
q8	3372	3496	3498	3496
q9	8857	8756	8847	8756
q10	3607	3523	3543	3523
q11	583	495	483	483
q12	797	586	584	584
q13	9877	3127	3189	3127
q14	310	284	267	267
q15	504	459	452	452
q16	478	438	444	438
q17	1831	1626	1592	1592
q18	8174	7790	7650	7650
q19	1686	1604	1530	1530
q20	2091	1908	1819	1819
q21	5229	5473	5268	5268
q22	1089	1021	1028	1021
Total cold run time: 68756 ms
Total hot run time: 59152 ms

@doris-robot
Copy link

doris-robot commented May 22, 2025

TPC-DS: Total hot run time: 196497 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0597a073181c6d3c2b199b17620b8e4adeb3ed95, data reload: false

query1	1266	894	890	890
query2	6435	1940	1888	1888
query3	10828	4266	4173	4173
query4	60988	29401	23376	23376
query5	5220	457	442	442
query6	403	179	177	177
query7	5517	316	312	312
query8	299	218	219	218
query9	8562	2592	2592	2592
query10	451	278	255	255
query11	17686	15242	16083	15242
query12	160	106	105	105
query13	1484	465	443	443
query14	10434	6755	6960	6755
query15	201	182	175	175
query16	7245	497	474	474
query17	1215	580	627	580
query18	1899	309	313	309
query19	219	154	148	148
query20	118	107	116	107
query21	212	101	103	101
query22	4632	4597	4556	4556
query23	34700	34296	34328	34296
query24	6099	2896	2877	2877
query25	492	379	399	379
query26	676	169	173	169
query27	1991	355	355	355
query28	4032	2155	2122	2122
query29	698	462	470	462
query30	255	157	158	157
query31	985	813	839	813
query32	68	61	58	58
query33	456	298	297	297
query34	902	517	506	506
query35	838	747	725	725
query36	1082	985	964	964
query37	110	73	71	71
query38	4099	3970	3935	3935
query39	1542	1627	1484	1484
query40	211	105	106	105
query41	53	53	49	49
query42	113	101	106	101
query43	531	496	508	496
query44	1180	812	822	812
query45	197	169	170	169
query46	1168	724	743	724
query47	2024	1921	1935	1921
query48	487	417	399	399
query49	763	420	421	420
query50	847	460	435	435
query51	7431	7240	7069	7069
query52	102	100	96	96
query53	267	195	194	194
query54	593	472	498	472
query55	83	80	87	80
query56	279	271	266	266
query57	1265	1137	1160	1137
query58	251	225	220	220
query59	3289	3072	3128	3072
query60	282	261	277	261
query61	110	108	108	108
query62	809	708	704	704
query63	216	187	190	187
query64	1373	641	647	641
query65	3274	3190	3182	3182
query66	718	285	295	285
query67	15980	15619	15578	15578
query68	4195	573	582	573
query69	436	262	263	262
query70	1164	1127	1115	1115
query71	339	258	262	258
query72	6311	4012	3949	3949
query73	743	345	372	345
query74	10547	9164	8988	8988
query75	3392	2642	2716	2642
query76	2016	1126	1054	1054
query77	473	298	281	281
query78	10563	9564	9541	9541
query79	1766	587	582	582
query80	1170	425	430	425
query81	501	227	222	222
query82	1245	89	88	88
query83	270	139	144	139
query84	279	84	72	72
query85	975	308	299	299
query86	361	303	276	276
query87	4407	4189	4183	4183
query88	3927	2429	2398	2398
query89	416	290	294	290
query90	1997	186	185	185
query91	182	148	148	148
query92	62	49	51	49
query93	1928	543	546	543
query94	777	278	301	278
query95	352	265	263	263
query96	611	279	281	279
query97	3320	3120	3128	3120
query98	228	210	200	200
query99	1628	1279	1284	1279
Total cold run time: 314976 ms
Total hot run time: 196497 ms

@doris-robot
Copy link

doris-robot commented May 22, 2025

ClickBench: Total hot run time: 30.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0597a073181c6d3c2b199b17620b8e4adeb3ed95, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.11	0.10
query5	0.51	0.52	0.51
query6	1.14	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.56	0.49	0.50
query10	0.55	0.54	0.54
query11	0.14	0.10	0.11
query12	0.14	0.10	0.11
query13	0.60	0.59	0.59
query14	0.78	0.78	0.82
query15	0.83	0.83	0.82
query16	0.39	0.39	0.38
query17	1.07	1.09	1.06
query18	0.24	0.22	0.22
query19	1.96	1.89	1.73
query20	0.02	0.01	0.01
query21	15.40	0.59	0.58
query22	2.83	2.48	2.85
query23	17.05	0.99	0.92
query24	3.03	1.66	1.18
query25	0.23	0.04	0.12
query26	0.56	0.15	0.14
query27	0.04	0.05	0.04
query28	9.74	0.52	0.49
query29	12.60	3.26	3.22
query30	0.24	0.06	0.05
query31	2.85	0.38	0.41
query32	3.23	0.46	0.46
query33	2.95	2.98	2.97
query34	17.12	4.44	4.47
query35	4.50	4.55	4.49
query36	0.68	0.47	0.47
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.48 s
Total hot run time: 30.98 s

dataroaring
dataroaring approved these changes May 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 78d124b into branch-3.0 May 24, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-51120-branch-3.0 branch May 24, 2025 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring @yagagagaga