Skip to content

Conversation

@yagagagaga
Copy link
Contributor

@yagagagaga yagagagaga commented May 21, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #26003

Problem Summary:

If image file is very huge, it maybe occurs http Read timed out when pulling image from FE Master

java.io.IOException: java.net.SocketTimeoutException: Read timed out
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1664)
        at org.apache.doris.catalog.Env.getClusterIdAndRole(Env.java:1126)
        at org.apache.doris.catalog.Env.initialize(Env.java:884)
        at org.apache.doris.DorisFE.start(DorisFE.java:163)
        at org.apache.doris.DorisFE.main(DorisFE.java:83)
Caused by: java.net.SocketTimeoutException: Read timed out
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun. reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1952)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1947)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.iava:1946)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1516)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at org.apache.doris.master.MetaHelper.getResponse(MetaHelper.java:130)
        at org.apache.doris.master.MetaHelper.getRemoteFile(MetaHelper.java:98)
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1657)
        ... 4 more

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yagagagaga
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33828 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ef907691b554a3ff176b49c8a0d713f81a9fe24, data reload: false

------ Round 1 ----------------------------------
q1	26081	5040	4996	4996
q2	2059	282	180	180
q3	10402	1222	686	686
q4	10222	1003	528	528
q5	7562	2441	2324	2324
q6	183	162	136	136
q7	926	754	614	614
q8	9329	1308	1113	1113
q9	6802	5113	5077	5077
q10	6827	2287	1897	1897
q11	472	293	275	275
q12	342	348	221	221
q13	17774	3754	3140	3140
q14	232	232	223	223
q15	527	479	486	479
q16	422	436	379	379
q17	581	843	383	383
q18	7659	7083	7062	7062
q19	1203	944	564	564
q20	351	333	217	217
q21	3853	3728	2327	2327
q22	1027	1007	1015	1007
Total cold run time: 114836 ms
Total hot run time: 33828 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5077	5062	5039	5039
q2	237	327	235	235
q3	2174	2662	2301	2301
q4	1348	1795	1403	1403
q5	4492	4446	4416	4416
q6	216	170	129	129
q7	1958	1876	1769	1769
q8	2577	2569	2515	2515
q9	7209	7071	7111	7071
q10	2993	3162	2753	2753
q11	561	491	484	484
q12	711	769	598	598
q13	3524	3923	3293	3293
q14	301	287	285	285
q15	522	467	492	467
q16	438	488	433	433
q17	1133	1545	1365	1365
q18	7711	7437	7518	7437
q19	770	812	771	771
q20	2011	2002	1877	1877
q21	4914	4324	4344	4324
q22	1041	984	981	981
Total cold run time: 51918 ms
Total hot run time: 49946 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 21, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 186926 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0ef907691b554a3ff176b49c8a0d713f81a9fe24, data reload: false

query1	1012	492	483	483
query2	6556	1839	1866	1839
query3	6753	219	219	219
query4	26438	23584	23639	23584
query5	4569	627	470	470
query6	309	204	197	197
query7	4622	487	287	287
query8	294	238	240	238
query9	8664	2639	2649	2639
query10	484	344	281	281
query11	15752	15057	15232	15057
query12	168	118	109	109
query13	1657	519	402	402
query14	9573	6196	6184	6184
query15	201	190	189	189
query16	7352	622	454	454
query17	1172	704	555	555
query18	1970	386	286	286
query19	186	184	160	160
query20	121	125	122	122
query21	222	126	116	116
query22	4077	4162	3964	3964
query23	34033	33019	33131	33019
query24	8398	2378	2362	2362
query25	534	453	388	388
query26	1272	265	160	160
query27	2751	498	331	331
query28	4326	2151	2127	2127
query29	760	558	432	432
query30	280	219	188	188
query31	935	869	744	744
query32	75	70	64	64
query33	570	391	326	326
query34	811	880	513	513
query35	780	816	734	734
query36	959	969	883	883
query37	116	102	81	81
query38	4097	4201	4124	4124
query39	1498	1414	1390	1390
query40	221	124	116	116
query41	64	61	60	60
query42	122	117	116	116
query43	514	511	480	480
query44	1318	824	826	824
query45	180	181	170	170
query46	847	1009	644	644
query47	1752	1787	1730	1730
query48	410	425	316	316
query49	807	546	457	457
query50	644	685	419	419
query51	4168	4101	4083	4083
query52	116	108	101	101
query53	237	257	193	193
query54	587	603	503	503
query55	82	87	84	84
query56	324	290	285	285
query57	1159	1111	1105	1105
query58	267	249	256	249
query59	2595	2673	2621	2621
query60	319	306	348	306
query61	122	124	121	121
query62	792	741	659	659
query63	231	182	184	182
query64	4309	1011	672	672
query65	4282	4243	4302	4243
query66	1128	412	312	312
query67	15961	15542	15322	15322
query68	7828	886	512	512
query69	467	302	274	274
query70	1174	1131	1105	1105
query71	445	316	300	300
query72	5535	4717	4775	4717
query73	660	615	355	355
query74	9314	9078	8911	8911
query75	3844	3200	2710	2710
query76	3484	1181	753	753
query77	798	364	296	296
query78	10168	10059	9466	9466
query79	2443	818	653	653
query80	587	523	469	469
query81	490	253	225	225
query82	482	125	100	100
query83	284	258	253	253
query84	294	120	93	93
query85	825	366	301	301
query86	392	311	298	298
query87	4399	4442	4439	4439
query88	3923	2321	2325	2321
query89	402	332	290	290
query90	1914	216	216	216
query91	145	146	151	146
query92	77	62	62	62
query93	1885	961	573	573
query94	686	414	291	291
query95	374	299	290	290
query96	506	573	288	288
query97	2808	2727	2675	2675
query98	238	207	215	207
query99	1432	1397	1263	1263
Total cold run time: 276208 ms
Total hot run time: 186926 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0ef907691b554a3ff176b49c8a0d713f81a9fe24, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.24	0.20	0.20
query4	1.60	0.18	0.20
query5	0.44	0.43	0.42
query6	1.15	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.51	0.51
query10	0.56	0.57	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.62	0.60
query14	0.79	0.80	0.82
query15	0.87	0.84	0.87
query16	0.36	0.39	0.41
query17	1.07	1.02	1.09
query18	0.23	0.23	0.21
query19	1.94	1.82	1.79
query20	0.01	0.01	0.02
query21	15.40	0.89	0.55
query22	0.76	1.24	0.67
query23	14.87	1.35	0.62
query24	6.90	1.55	0.70
query25	0.43	0.25	0.10
query26	0.61	0.16	0.15
query27	0.05	0.06	0.05
query28	9.37	0.91	0.45
query29	12.56	4.07	3.37
query30	0.25	0.09	0.07
query31	2.81	0.59	0.39
query32	3.25	0.55	0.47
query33	3.10	3.04	3.07
query34	15.75	5.11	4.48
query35	4.50	4.52	4.52
query36	0.69	0.51	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.78 s
Total hot run time: 29.01 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 8fa8c4a into apache:master May 22, 2025
28 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request May 22, 2025
Related PR: #26003

Problem Summary:

> If image file is very huge, it maybe occurs http Read timed out when
pulling image from FE Master

```java
java.io.IOException: java.net.SocketTimeoutException: Read timed out
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1664)
        at org.apache.doris.catalog.Env.getClusterIdAndRole(Env.java:1126)
        at org.apache.doris.catalog.Env.initialize(Env.java:884)
        at org.apache.doris.DorisFE.start(DorisFE.java:163)
        at org.apache.doris.DorisFE.main(DorisFE.java:83)
Caused by: java.net.SocketTimeoutException: Read timed out
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun. reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1952)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1947)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.iava:1946)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1516)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at org.apache.doris.master.MetaHelper.getResponse(MetaHelper.java:130)
        at org.apache.doris.master.MetaHelper.getRemoteFile(MetaHelper.java:98)
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1657)
        ... 4 more
```
github-actions bot pushed a commit that referenced this pull request May 22, 2025
Related PR: #26003

Problem Summary:

> If image file is very huge, it maybe occurs http Read timed out when
pulling image from FE Master

```java
java.io.IOException: java.net.SocketTimeoutException: Read timed out
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1664)
        at org.apache.doris.catalog.Env.getClusterIdAndRole(Env.java:1126)
        at org.apache.doris.catalog.Env.initialize(Env.java:884)
        at org.apache.doris.DorisFE.start(DorisFE.java:163)
        at org.apache.doris.DorisFE.main(DorisFE.java:83)
Caused by: java.net.SocketTimeoutException: Read timed out
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun. reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1952)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1947)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.iava:1946)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1516)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at org.apache.doris.master.MetaHelper.getResponse(MetaHelper.java:130)
        at org.apache.doris.master.MetaHelper.getRemoteFile(MetaHelper.java:98)
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1657)
        ... 4 more
```
yiguolei pushed a commit that referenced this pull request May 22, 2025
…out #51120 (#51149)

Cherry-picked from #51120

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
dataroaring pushed a commit that referenced this pull request May 24, 2025
…out #51120 (#51148)

Cherry-picked from #51120

Co-authored-by: yagagagaga <zhangminke@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…51120)

Related PR: apache#26003

Problem Summary:

> If image file is very huge, it maybe occurs http Read timed out when
pulling image from FE Master

```java
java.io.IOException: java.net.SocketTimeoutException: Read timed out
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1664)
        at org.apache.doris.catalog.Env.getClusterIdAndRole(Env.java:1126)
        at org.apache.doris.catalog.Env.initialize(Env.java:884)
        at org.apache.doris.DorisFE.start(DorisFE.java:163)
        at org.apache.doris.DorisFE.main(DorisFE.java:83)
Caused by: java.net.SocketTimeoutException: Read timed out
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun. reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1952)
        at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1947)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.iava:1946)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1516)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1500)
        at org.apache.doris.master.MetaHelper.getResponse(MetaHelper.java:130)
        at org.apache.doris.master.MetaHelper.getRemoteFile(MetaHelper.java:98)
        at org.apache.doris.catalog.Env.getNewImage(Env.java:1657)
        ... 4 more
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.6-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants