Skip to content

[fix](s3client) Avoild dead loop when storage not support ListObjectsV2 #50252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 1 commit into from
Apr 25, 2025
Merged

[fix](s3client) Avoild dead loop when storage not support ListObjectsV2 #50252

gavinchou merged 1 commit into from
Apr 25, 2025

Conversation

@swjtu-zhanglei
Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Apr 21, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Apr 21, 2025

run buildall

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Apr 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

TeamCity cloud ut coverage result:
Function Coverage: 82.93% (1093/1318)
Line Coverage: 65.65% (18267/27823)
Region Coverage: 65.04% (9046/13908)
Branch Coverage: 55.07% (4875/8852)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c600ff19df6a841aa26d664d2f65970d0dbb5777_c600ff19df6a841aa26d664d2f65970d0dbb5777_cloud/report/index.html

gavinchou
gavinchou previously approved these changes Apr 21, 2025
View reviewed changes
@gavinchou gavinchou added p0_b dev/3.0.x usercase Important user case type label labels Apr 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 21, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 21, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

TPC-H: Total hot run time: 34093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c600ff19df6a841aa26d664d2f65970d0dbb5777, data reload: false

------ Round 1 ----------------------------------
q1	26157	5103	5029	5029
q2	2070	277	189	189
q3	10391	1252	749	749
q4	10252	1005	528	528
q5	7542	2344	2407	2344
q6	184	165	130	130
q7	906	750	615	615
q8	9336	1288	1100	1100
q9	6786	5058	5033	5033
q10	6807	2283	1909	1909
q11	488	288	269	269
q12	341	358	223	223
q13	17758	3683	3090	3090
q14	227	233	219	219
q15	525	483	485	483
q16	453	451	396	396
q17	598	856	365	365
q18	7690	7241	7185	7185
q19	1204	962	557	557
q20	330	336	228	228
q21	4412	3408	2484	2484
q22	1075	1035	968	968
Total cold run time: 115532 ms
Total hot run time: 34093 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5090	5069	5135	5069
q2	239	329	230	230
q3	2194	2687	2318	2318
q4	1389	1830	1411	1411
q5	4544	4461	4399	4399
q6	215	161	124	124
q7	1948	1880	1713	1713
q8	2565	2559	2486	2486
q9	7125	7149	7159	7149
q10	2981	3188	2755	2755
q11	566	532	473	473
q12	650	774	590	590
q13	3477	4239	3313	3313
q14	271	295	273	273
q15	506	473	463	463
q16	481	509	471	471
q17	1175	1534	1409	1409
q18	7745	7538	7413	7413
q19	812	783	951	783
q20	1961	2023	1890	1890
q21	5192	4729	4611	4611
q22	1077	1015	1002	1002
Total cold run time: 52203 ms
Total hot run time: 50345 ms

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

TPC-DS: Total hot run time: 185735 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c600ff19df6a841aa26d664d2f65970d0dbb5777, data reload: false

query1	1025	464	510	464
query2	6564	1812	1799	1799
query3	6760	228	218	218
query4	26475	23745	23631	23631
query5	4436	617	468	468
query6	288	199	186	186
query7	4620	482	273	273
query8	301	234	232	232
query9	8645	2545	2559	2545
query10	482	302	263	263
query11	15774	15045	14802	14802
query12	171	109	103	103
query13	1643	513	397	397
query14	8791	6127	6103	6103
query15	200	189	172	172
query16	7134	618	478	478
query17	1207	741	601	601
query18	1992	413	308	308
query19	191	198	160	160
query20	122	117	119	117
query21	212	123	106	106
query22	4310	4216	4175	4175
query23	33796	32940	32885	32885
query24	8443	2379	2384	2379
query25	559	512	395	395
query26	1237	268	149	149
query27	2886	494	326	326
query28	4559	2103	2091	2091
query29	758	548	428	428
query30	277	214	188	188
query31	951	889	771	771
query32	71	64	64	64
query33	575	351	319	319
query34	775	880	503	503
query35	779	825	718	718
query36	962	988	909	909
query37	116	101	78	78
query38	4133	4093	4036	4036
query39	1455	1381	1369	1369
query40	211	121	105	105
query41	62	55	51	51
query42	119	107	106	106
query43	485	502	482	482
query44	1293	790	789	789
query45	177	174	169	169
query46	821	1019	609	609
query47	1753	1763	1708	1708
query48	384	403	294	294
query49	777	518	410	410
query50	642	668	397	397
query51	4108	4111	4065	4065
query52	111	107	99	99
query53	228	249	186	186
query54	574	566	491	491
query55	82	76	81	76
query56	298	301	285	285
query57	1138	1162	1070	1070
query58	264	262	256	256
query59	2538	2627	2500	2500
query60	325	325	297	297
query61	128	124	119	119
query62	769	712	686	686
query63	218	182	189	182
query64	4336	1009	663	663
query65	4319	4230	4237	4230
query66	1144	424	310	310
query67	15678	15407	15259	15259
query68	7891	911	515	515
query69	469	290	266	266
query70	1117	1099	1065	1065
query71	468	343	280	280
query72	5896	4799	4856	4799
query73	734	666	349	349
query74	8889	9020	8699	8699
query75	3761	3187	2716	2716
query76	3626	1187	760	760
query77	783	369	300	300
query78	9920	9975	9294	9294
query79	2826	813	566	566
query80	617	492	441	441
query81	498	268	213	213
query82	470	127	100	100
query83	280	256	242	242
query84	291	102	86	86
query85	825	445	310	310
query86	386	293	304	293
query87	4368	4512	4383	4383
query88	3528	2202	2214	2202
query89	388	317	280	280
query90	1888	212	214	212
query91	140	140	115	115
query92	75	72	58	58
query93	1970	952	576	576
query94	667	417	303	303
query95	387	293	282	282
query96	480	567	278	278
query97	3164	3216	3123	3123
query98	239	213	205	205
query99	1404	1387	1321	1321
Total cold run time: 275102 ms
Total hot run time: 185735 ms

@doris-robot
Copy link

doris-robot commented Apr 21, 2025

ClickBench: Total hot run time: 29.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c600ff19df6a841aa26d664d2f65970d0dbb5777, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.11	0.10
query3	0.24	0.20	0.20
query4	1.59	0.19	0.19
query5	0.59	0.57	0.58
query6	1.18	0.71	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.58	0.53	0.50
query10	0.57	0.56	0.57
query11	0.16	0.11	0.10
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	1.20	1.17	1.21
query15	0.85	0.86	0.85
query16	0.38	0.40	0.37
query17	1.04	1.02	1.02
query18	0.21	0.20	0.19
query19	1.91	1.79	1.79
query20	0.02	0.01	0.01
query21	15.41	0.90	0.54
query22	0.76	1.08	0.64
query23	15.03	1.38	0.63
query24	7.16	0.99	0.82
query25	0.47	0.23	0.15
query26	0.52	0.17	0.15
query27	0.06	0.06	0.05
query28	8.91	0.90	0.44
query29	12.53	3.99	3.33
query30	0.25	0.08	0.06
query31	2.84	0.58	0.38
query32	3.26	0.54	0.47
query33	2.97	3.16	3.03
query34	15.68	5.13	4.44
query35	4.52	4.56	4.47
query36	0.65	0.50	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.12
query41	0.09	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.02 s
Total hot run time: 29.5 s

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Apr 22, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 22, 2025
@doris-robot
Copy link

doris-robot commented Apr 22, 2025

TeamCity cloud ut coverage result:
Function Coverage: 82.93% (1093/1318)
Line Coverage: 65.64% (18264/27823)
Region Coverage: 65.03% (9044/13908)
Branch Coverage: 55.02% (4870/8852)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3675655598912846f629ed33d356881392c91f63_3675655598912846f629ed33d356881392c91f63_cloud/report/index.html

@doris-robot
Copy link

doris-robot commented Apr 22, 2025

TPC-H: Total hot run time: 33907 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3675655598912846f629ed33d356881392c91f63, data reload: false

------ Round 1 ----------------------------------
q1	26341	5129	5046	5046
q2	2055	273	187	187
q3	10393	1213	680	680
q4	10231	990	553	553
q5	7500	2398	2342	2342
q6	186	163	131	131
q7	914	753	602	602
q8	9322	1286	1036	1036
q9	6821	5122	5115	5115
q10	6838	2323	1880	1880
q11	479	293	279	279
q12	338	352	219	219
q13	17740	3641	3140	3140
q14	229	227	203	203
q15	528	487	478	478
q16	441	453	410	410
q17	575	854	361	361
q18	7389	7253	7087	7087
q19	886	937	531	531
q20	343	337	222	222
q21	3887	3341	2416	2416
q22	1080	1009	989	989
Total cold run time: 114516 ms
Total hot run time: 33907 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5056	5119	5077	5077
q2	238	333	227	227
q3	2114	2648	2309	2309
q4	1425	1823	1381	1381
q5	4205	4126	4140	4126
q6	203	161	121	121
q7	1863	1798	1651	1651
q8	2487	2468	2449	2449
q9	6831	6732	6793	6732
q10	2898	3043	2641	2641
q11	563	489	477	477
q12	642	710	592	592
q13	3291	3704	3147	3147
q14	262	286	258	258
q15	512	470	455	455
q16	458	515	457	457
q17	1115	1478	1347	1347
q18	7282	7066	7025	7025
q19	774	797	954	797
q20	1958	1928	1765	1765
q21	5181	4773	4722	4722
q22	1094	1040	1032	1032
Total cold run time: 50452 ms
Total hot run time: 48788 ms

@doris-robot
Copy link

doris-robot commented Apr 22, 2025

TPC-DS: Total hot run time: 185930 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3675655598912846f629ed33d356881392c91f63, data reload: false

query1	1006	472	463	463
query2	6579	1782	1756	1756
query3	6741	222	228	222
query4	26135	23410	23510	23410
query5	5630	628	433	433
query6	285	200	194	194
query7	4610	478	269	269
query8	284	244	230	230
query9	8638	2551	2566	2551
query10	499	324	262	262
query11	15425	15014	14936	14936
query12	164	113	109	109
query13	1663	507	400	400
query14	9956	6117	6076	6076
query15	212	184	162	162
query16	7604	644	502	502
query17	1420	728	556	556
query18	2005	388	301	301
query19	201	178	157	157
query20	158	114	118	114
query21	204	121	101	101
query22	4228	4314	4089	4089
query23	33871	33200	33119	33119
query24	8330	2378	2356	2356
query25	500	438	389	389
query26	1225	270	146	146
query27	2731	491	322	322
query28	4296	2097	2087	2087
query29	743	542	431	431
query30	274	213	183	183
query31	931	818	755	755
query32	70	65	62	62
query33	546	370	322	322
query34	797	866	512	512
query35	813	826	746	746
query36	965	987	884	884
query37	114	116	78	78
query38	4207	4151	4094	4094
query39	1482	1420	1387	1387
query40	223	130	111	111
query41	61	59	63	59
query42	121	107	109	107
query43	493	493	466	466
query44	1277	801	800	800
query45	181	181	170	170
query46	841	1024	618	618
query47	1811	1861	1759	1759
query48	372	405	303	303
query49	813	540	429	429
query50	638	685	410	410
query51	4090	4108	4053	4053
query52	110	106	140	106
query53	224	252	179	179
query54	565	561	499	499
query55	83	79	81	79
query56	302	302	308	302
query57	1131	1183	1119	1119
query58	266	249	253	249
query59	2601	2647	2600	2600
query60	319	317	303	303
query61	173	127	125	125
query62	803	736	672	672
query63	231	184	199	184
query64	4295	1012	687	687
query65	4349	4265	4242	4242
query66	1001	406	320	320
query67	15606	15657	15332	15332
query68	8313	902	514	514
query69	502	320	270	270
query70	1214	1072	1080	1072
query71	496	319	296	296
query72	5715	4773	4794	4773
query73	705	648	342	342
query74	8875	8788	9073	8788
query75	3970	3209	2704	2704
query76	3786	1190	757	757
query77	795	363	283	283
query78	10044	9948	9253	9253
query79	2468	812	563	563
query80	792	520	432	432
query81	468	262	219	219
query82	453	126	98	98
query83	289	244	234	234
query84	297	106	81	81
query85	759	357	315	315
query86	338	302	297	297
query87	4412	4455	4257	4257
query88	3580	2167	2170	2167
query89	380	320	278	278
query90	1939	207	200	200
query91	140	145	112	112
query92	82	62	61	61
query93	2059	936	575	575
query94	666	416	307	307
query95	361	287	274	274
query96	488	567	274	274
query97	3151	3246	3161	3161
query98	226	212	209	209
query99	1449	1418	1315	1315
Total cold run time: 277734 ms
Total hot run time: 185930 ms

@doris-robot
Copy link

doris-robot commented Apr 22, 2025

ClickBench: Total hot run time: 29.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3675655598912846f629ed33d356881392c91f63, data reload: false

query1	0.04	0.04	0.04
query2	0.12	0.10	0.10
query3	0.25	0.20	0.19
query4	1.60	0.19	0.19
query5	0.60	0.58	0.58
query6	1.17	0.73	0.72
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.58	0.53	0.52
query10	0.58	0.56	0.57
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.59	0.61
query14	1.16	1.16	1.18
query15	0.88	0.86	0.86
query16	0.38	0.38	0.40
query17	1.03	1.03	1.04
query18	0.21	0.19	0.20
query19	1.88	1.76	1.80
query20	0.01	0.02	0.01
query21	15.40	0.91	0.56
query22	0.76	1.29	0.70
query23	14.82	1.40	0.63
query24	6.68	1.52	0.54
query25	0.44	0.23	0.13
query26	0.62	0.17	0.14
query27	0.05	0.06	0.05
query28	9.70	0.92	0.45
query29	12.61	3.96	3.32
query30	0.25	0.11	0.06
query31	2.81	0.61	0.38
query32	3.25	0.55	0.47
query33	3.05	3.09	2.98
query34	16.02	5.17	4.55
query35	4.53	4.54	4.55
query36	0.66	0.50	0.50
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.61 s
Total hot run time: 29.46 s

@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Apr 22, 2025

run cloud_p0

1 similar comment
@swjtu-zhanglei
Copy link
Contributor Author

swjtu-zhanglei commented Apr 23, 2025

run cloud_p0

dataroaring
dataroaring approved these changes Apr 23, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 23, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 23, 2025

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou self-requested a review April 25, 2025 06:53
@gavinchou gavinchou merged commit dedef0b into apache:master Apr 25, 2025
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Apr 25, 2025
@swjtu-zhanglei
[fix](s3client) Avoild dead loop when storage not support `ListObject…
7f0b909 
…sV2` (#50252)
@github-actions github-actions bot mentioned this pull request Apr 25, 2025
Merged
dataroaring pushed a commit that referenced this pull request Apr 25, 2025
@github-actions @swjtu-zhanglei
branch-3.0: [fix](s3client) Avoild dead loop when storage not support…
79fcf4f 
… `ListObjectsV2` #50252 (#50413)

Cherry-picked from #50252

Co-authored-by: Lei Zhang <zhanglei@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@swjtu-zhanglei
[fix](s3client) Avoild dead loop when storage not support `ListObject…
0ee421f 
…sV2` (apache#50252)
@gavinchou gavinchou mentioned this pull request Jun 11, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.6-merged p0_b reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@swjtu-zhanglei @doris-robot @gavinchou @dataroaring