[fix](hive) Fix StackOverflowError in insert overwrite on S3-compatible storage by zy-kkk · Pull Request #58504 · apache/doris

zy-kkk · 2025-11-28T08:07:36Z

Fix StackOverflowError when executing insert overwrite on Hive tables stored in S3-compatible object storage (OBS, COS, OSS, etc.). The root cause is that
S3ObjStorage.listDirectories hardcoded "s3://" scheme when constructing directory paths, causing scheme mismatch (e.g., obs:// becomes s3://) which leads to
infinite recursion in HMSTransaction.doRecursiveDeleteFiles. This PR preserves the original scheme from input path。

hello-stephen · 2025-11-28T08:07:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

…le storage

morningman · 2025-11-28T08:45:48Z

run buildall

hello-stephen · 2025-11-28T10:05:03Z

FE UT Coverage Report

Increment line coverage 75.00% (3/4) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-11-28T10:36:46Z

TPC-H: Total hot run time: 34137 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a0f3705f6168c670488561735a883e2d409561b, data reload: false

------ Round 1 ----------------------------------
q1	17684	4971	4885	4885
q2	2048	330	234	234
q3	10178	1329	711	711
q4	10231	899	365	365
q5	7561	2493	2188	2188
q6	185	168	135	135
q7	954	790	648	648
q8	9363	1384	965	965
q9	7097	5344	5340	5340
q10	6905	2229	1788	1788
q11	501	318	283	283
q12	377	366	227	227
q13	17801	3664	3043	3043
q14	240	246	208	208
q15	598	518	514	514
q16	906	866	807	807
q17	587	756	514	514
q18	7752	7127	7173	7127
q19	1088	946	566	566
q20	344	336	228	228
q21	3526	3284	2396	2396
q22	1039	1002	965	965
Total cold run time: 106965 ms
Total hot run time: 34137 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5194	4910	4886	4886
q2	333	423	321	321
q3	2108	2694	2294	2294
q4	1337	1751	1282	1282
q5	4189	4350	4534	4350
q6	212	171	136	136
q7	2154	1971	1885	1885
q8	2584	2577	2463	2463
q9	7428	7511	7501	7501
q10	3063	3335	2790	2790
q11	590	514	497	497
q12	673	735	630	630
q13	3464	3916	3292	3292
q14	309	307	292	292
q15	564	515	496	496
q16	924	934	888	888
q17	1146	1416	1423	1416
q18	7877	7683	7470	7470
q19	812	807	786	786
q20	1937	1975	1789	1789
q21	4508	4243	4120	4120
q22	1071	998	968	968
Total cold run time: 52477 ms
Total hot run time: 50552 ms

doris-robot · 2025-11-28T10:48:08Z

TPC-DS: Total hot run time: 181711 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a0f3705f6168c670488561735a883e2d409561b, data reload: false

query1	1066	409	396	396
query2	6585	1191	1158	1158
query3	6755	235	226	226
query4	25394	23361	22900	22900
query5	4875	643	491	491
query6	349	238	225	225
query7	4654	509	304	304
query8	308	265	261	261
query9	8737	2563	2609	2563
query10	511	385	305	305
query11	15462	14764	15027	14764
query12	181	123	114	114
query13	1681	559	424	424
query14	9430	5983	6006	5983
query15	212	199	186	186
query16	7366	679	533	533
query17	1176	740	655	655
query18	2020	446	345	345
query19	219	209	187	187
query20	131	121	123	121
query21	226	141	119	119
query22	3973	3929	3874	3874
query23	32698	31746	31797	31746
query24	8515	2433	2442	2433
query25	656	567	483	483
query26	1256	285	172	172
query27	2743	490	358	358
query28	4404	2160	2133	2133
query29	850	643	532	532
query30	310	247	223	223
query31	836	718	613	613
query32	84	78	77	77
query33	633	385	337	337
query34	849	859	549	549
query35	814	845	743	743
query36	880	941	844	844
query37	126	120	87	87
query38	3815	3874	3762	3762
query39	1486	1413	1420	1413
query40	235	140	141	140
query41	73	67	111	67
query42	125	112	114	112
query43	445	460	424	424
query44	1281	767	755	755
query45	200	192	184	184
query46	868	983	656	656
query47	1666	1720	1628	1628
query48	400	424	321	321
query49	797	511	436	436
query50	656	698	400	400
query51	3805	3901	3878	3878
query52	118	115	105	105
query53	242	265	198	198
query54	310	317	294	294
query55	99	96	98	96
query56	345	326	328	326
query57	1132	1191	1106	1106
query58	295	279	272	272
query59	2240	2434	2337	2337
query60	354	349	346	346
query61	160	161	166	161
query62	794	725	710	710
query63	232	205	198	198
query64	4586	1223	898	898
query65	4088	3980	3969	3969
query66	1174	429	336	336
query67	15227	14994	14786	14786
query68	8221	976	625	625
query69	534	350	311	311
query70	1141	992	987	987
query71	430	347	324	324
query72	5845	4931	4928	4928
query73	666	603	354	354
query74	8778	8733	8566	8566
query75	3073	3005	2528	2528
query76	3320	1129	741	741
query77	539	430	333	333
query78	9398	9490	8889	8889
query79	1946	850	600	600
query80	772	615	516	516
query81	543	276	243	243
query82	431	147	113	113
query83	285	269	256	256
query84	279	120	103	103
query85	935	485	451	451
query86	395	332	285	285
query87	4075	3995	3933	3933
query88	3445	2373	2335	2335
query89	387	332	296	296
query90	1978	230	230	230
query91	172	172	144	144
query92	86	73	66	66
query93	2531	975	659	659
query94	751	467	341	341
query95	512	417	414	414
query96	512	561	287	287
query97	2616	2700	2559	2559
query98	251	222	231	222
query99	1305	1400	1294	1294
Total cold run time: 270365 ms
Total hot run time: 181711 ms

doris-robot · 2025-11-28T10:53:12Z

ClickBench: Total hot run time: 27.59 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a0f3705f6168c670488561735a883e2d409561b, data reload: false

query1	0.06	0.05	0.06
query2	0.10	0.05	0.05
query3	0.26	0.08	0.09
query4	1.62	0.12	0.11
query5	0.26	0.26	0.26
query6	1.16	0.64	0.63
query7	0.03	0.03	0.03
query8	0.06	0.04	0.05
query9	0.60	0.51	0.51
query10	0.56	0.55	0.56
query11	0.16	0.10	0.12
query12	0.15	0.12	0.11
query13	0.63	0.60	0.59
query14	0.98	0.97	0.99
query15	0.82	0.80	0.80
query16	0.40	0.38	0.39
query17	0.98	1.00	1.03
query18	0.24	0.24	0.21
query19	2.00	1.90	1.85
query20	0.02	0.01	0.01
query21	15.44	0.28	0.15
query22	4.76	0.05	0.05
query23	15.90	0.27	0.10
query24	1.36	0.35	0.54
query25	0.09	0.06	0.06
query26	0.15	0.13	0.13
query27	0.09	0.07	0.06
query28	4.30	1.20	1.02
query29	12.60	3.92	3.22
query30	0.30	0.14	0.12
query31	2.82	0.60	0.40
query32	3.23	0.53	0.47
query33	3.02	3.17	3.13
query34	16.88	5.20	4.58
query35	4.59	4.63	4.57
query36	0.67	0.50	0.49
query37	0.10	0.06	0.06
query38	0.08	0.04	0.04
query39	0.05	0.03	0.04
query40	0.18	0.14	0.14
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 97.87 s
Total hot run time: 27.59 s

hello-stephen · 2025-11-28T13:52:54Z

FE Regression Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-11-29T01:52:02Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-11-29T01:52:04Z

PR approved by anyone and no changes requested.

…le storage (#58504) Fix StackOverflowError when executing insert overwrite on Hive tables stored in S3-compatible object storage (OBS, COS, OSS, etc.). The root cause is that S3ObjStorage.listDirectories hardcoded "s3://" scheme when constructing directory paths, causing scheme mismatch (e.g., obs:// becomes s3://) which leads to infinite recursion in HMSTransaction.doRecursiveDeleteFiles. This PR preserves the original scheme from input path。

… S3-compatible storage #58504 (#58534) Cherry-picked from #58504 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

… S3-compatible storage #58504 (#58533) Cherry-picked from #58504 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

Related PR: #58504 hive on glue unsupported create db

Related PR: [#58504](#58504) hive on glue unsupported create db

…le storage (apache#58504) Fix StackOverflowError when executing insert overwrite on Hive tables stored in S3-compatible object storage (OBS, COS, OSS, etc.). The root cause is that S3ObjStorage.listDirectories hardcoded "s3://" scheme when constructing directory paths, causing scheme mismatch (e.g., obs:// becomes s3://) which leads to infinite recursion in HMSTransaction.doRecursiveDeleteFiles. This PR preserves the original scheme from input path。

Related PR: apache#58504 hive on glue unsupported create db

Related PR: [apache#58504](apache#58504) hive on glue unsupported create db

…le storage (apache#5996) pick apache#58504

Related PR: apache#58504 hive on glue unsupported create db

zy-kkk requested a review from CalvinKirs as a code owner November 28, 2025 08:07

zy-kkk added dev/3.1.x dev/4.0.x labels Nov 28, 2025

[fix](hive) Fix StackOverflowError in insert overwrite on S3-compatib…

9a0f370

…le storage

zy-kkk force-pushed the fix_hive branch from 156c879 to 9a0f370 Compare November 28, 2025 08:42

morningman approved these changes Nov 29, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 29, 2025

github-actions bot added the reviewed label Nov 29, 2025

CalvinKirs approved these changes Nov 29, 2025

View reviewed changes

zy-kkk merged commit 2110466 into apache:master Nov 30, 2025
27 of 28 checks passed

zy-kkk deleted the fix_hive branch November 30, 2025 10:43

github-actions bot mentioned this pull request Nov 30, 2025

branch-3.1: [fix](hive) Fix StackOverflowError in insert overwrite on S3-compatible storage #58504 #58533

Merged

github-actions bot mentioned this pull request Nov 30, 2025

branch-4.0: [fix](hive) Fix StackOverflowError in insert overwrite on S3-compatible storage #58504 #58534

Merged

zy-kkk mentioned this pull request Dec 1, 2025

[fix](test) Remove unsupported test cases #58563

Merged

yiguolei pushed a commit that referenced this pull request Dec 2, 2025

branch-4.0: [fix](hive) Fix StackOverflowError in insert overwrite on…

1d83833

… S3-compatible storage #58504 (#58534) Cherry-picked from #58504 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

yiguolei added dev/4.0.2-merged and removed dev/4.0.x labels Dec 2, 2025

morrySnow pushed a commit that referenced this pull request Dec 2, 2025

branch-3.1: [fix](hive) Fix StackOverflowError in insert overwrite on…

a3c7a07

… S3-compatible storage #58504 (#58533) Cherry-picked from #58504 Co-authored-by: zy-kkk <zhongyk10@gmail.com>

morrySnow removed the dev/3.1.x label Dec 2, 2025

morrySnow added the dev/3.1.4-merged label Dec 2, 2025

yiguolei mentioned this pull request Dec 2, 2025

4.0.2 Release Notes #58605

Open

zy-kkk mentioned this pull request Dec 4, 2025

[fix](test) Remove unsupported test cases #58716

Merged

morningman pushed a commit that referenced this pull request Dec 4, 2025

[fix](test) Remove unsupported test cases (#58716)

1732416

Related PR: #58504 hive on glue unsupported create db

github-actions bot pushed a commit that referenced this pull request Dec 4, 2025

[fix](test) Remove unsupported test cases (#58716)

10f529a

Related PR: #58504 hive on glue unsupported create db

github-actions bot pushed a commit that referenced this pull request Dec 4, 2025

[fix](test) Remove unsupported test cases (#58716)

e447227

Related PR: #58504 hive on glue unsupported create db

morningman pushed a commit that referenced this pull request Dec 5, 2025

[fix](test) Remove unsupported test cases (#58563)

ebec0f2

Related PR: [#58504](#58504) hive on glue unsupported create db

nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025

[fix](test) Remove unsupported test cases (apache#58716)

662f94d

Related PR: apache#58504 hive on glue unsupported create db

nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025

[fix](test) Remove unsupported test cases (apache#58563)

6da0b36

Related PR: [apache#58504](apache#58504) hive on glue unsupported create db

seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 30, 2026

[fix](hive) Fix StackOverflowError in insert overwrite on S3-compatib…

f730241

…le storage (apache#5996) pick apache#58504

seawinde pushed a commit to seawinde/doris that referenced this pull request Jan 30, 2026

[fix](test) Remove unsupported test cases (apache#6000)

7c4dbda

Related PR: apache#58504 hive on glue unsupported create db

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](hive) Fix StackOverflowError in insert overwrite on S3-compatible storage#58504

[fix](hive) Fix StackOverflowError in insert overwrite on S3-compatible storage#58504
zy-kkk merged 1 commit intoapache:masterfrom
zy-kkk:fix_hive

zy-kkk commented Nov 28, 2025

Uh oh!

hello-stephen commented Nov 28, 2025

Uh oh!

morningman commented Nov 28, 2025

Uh oh!

hello-stephen commented Nov 28, 2025

Uh oh!

doris-robot commented Nov 28, 2025

Uh oh!

doris-robot commented Nov 28, 2025

Uh oh!

doris-robot commented Nov 28, 2025

Uh oh!

hello-stephen commented Nov 28, 2025

Uh oh!

github-actions bot commented Nov 29, 2025

Uh oh!

github-actions bot commented Nov 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Conversation

zy-kkk commented Nov 28, 2025

Uh oh!

hello-stephen commented Nov 28, 2025

Uh oh!

morningman commented Nov 28, 2025

Uh oh!

hello-stephen commented Nov 28, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Nov 28, 2025

Uh oh!

doris-robot commented Nov 28, 2025

Uh oh!

doris-robot commented Nov 28, 2025

Uh oh!

hello-stephen commented Nov 28, 2025

FE Regression Coverage Report

Uh oh!

github-actions bot commented Nov 29, 2025

Uh oh!

github-actions bot commented Nov 29, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants