[fix](hive) Incorrect location conversion #46362

wuwenchi · 2025-01-03T07:13:16Z

What problem does this PR solve?

Problem Summary:

For the "original_write_path" field, when the location is of the S3 type, it should not be converted into the S3 schema. Instead, the original location should be retained.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-01-03T07:13:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

wuwenchi · 2025-01-03T07:13:27Z

run buildall

doris-robot · 2025-01-03T07:33:04Z

TPC-H: Total hot run time: 32562 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5c5195a28b3ca6dc013fdcd8dc508a7cd9438e51, data reload: false

------ Round 1 ----------------------------------
q1	17635	6084	6059	6059
q2	2054	317	170	170
q3	10533	1252	715	715
q4	10228	849	424	424
q5	7690	2161	1916	1916
q6	204	178	144	144
q7	890	729	599	599
q8	9218	1309	1143	1143
q9	5257	4908	4853	4853
q10	6746	2303	1857	1857
q11	469	284	261	261
q12	343	355	216	216
q13	17770	3656	3102	3102
q14	232	245	210	210
q15	565	491	494	491
q16	612	632	575	575
q17	559	833	322	322
q18	7060	6494	6523	6494
q19	1276	949	532	532
q20	297	311	190	190
q21	2792	2140	1979	1979
q22	359	330	310	310
Total cold run time: 102789 ms
Total hot run time: 32562 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6178	6141	6210	6141
q2	234	322	227	227
q3	2276	2609	2305	2305
q4	1350	1820	1327	1327
q5	4370	4749	4765	4749
q6	182	176	141	141
q7	2157	1994	1797	1797
q8	2549	2769	2645	2645
q9	7368	7279	7160	7160
q10	3043	3292	2784	2784
q11	579	508	507	507
q12	671	768	621	621
q13	3478	3923	3232	3232
q14	288	310	284	284
q15	571	525	500	500
q16	658	687	644	644
q17	1179	1704	1255	1255
q18	7757	7373	7259	7259
q19	771	1164	992	992
q20	1979	2038	1847	1847
q21	5490	4971	4979	4971
q22	630	656	577	577
Total cold run time: 53758 ms
Total hot run time: 51965 ms

doris-robot · 2025-01-03T07:45:16Z

TPC-DS: Total hot run time: 191245 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5c5195a28b3ca6dc013fdcd8dc508a7cd9438e51, data reload: false

query1	986	380	363	363
query2	6511	2426	2312	2312
query3	6701	224	223	223
query4	40717	24197	23464	23464
query5	5904	644	472	472
query6	262	193	191	191
query7	4711	507	306	306
query8	309	249	230	230
query9	6627	2629	2627	2627
query10	479	313	264	264
query11	16081	15286	15254	15254
query12	155	111	109	109
query13	1048	521	415	415
query14	11811	6817	6817	6817
query15	256	210	196	196
query16	8040	599	434	434
query17	1450	724	573	573
query18	2066	408	310	310
query19	235	179	148	148
query20	120	108	109	108
query21	207	126	102	102
query22	4192	4542	4407	4407
query23	34176	33300	33598	33300
query24	5950	2284	2239	2239
query25	481	498	367	367
query26	1221	269	150	150
query27	2095	458	338	338
query28	5447	2450	2414	2414
query29	734	567	410	410
query30	228	180	147	147
query31	980	908	835	835
query32	91	59	64	59
query33	485	347	301	301
query34	776	825	494	494
query35	807	815	742	742
query36	1017	1050	954	954
query37	123	101	73	73
query38	4264	4454	4154	4154
query39	1459	1474	1445	1445
query40	199	114	102	102
query41	47	45	49	45
query42	120	102	100	100
query43	523	511	485	485
query44	1267	807	793	793
query45	175	175	167	167
query46	864	1051	642	642
query47	1972	1933	1886	1886
query48	373	411	317	317
query49	763	473	417	417
query50	616	642	369	369
query51	7176	7099	7037	7037
query52	96	98	98	98
query53	220	247	191	191
query54	466	465	404	404
query55	78	80	75	75
query56	263	257	244	244
query57	1219	1200	1139	1139
query58	242	213	255	213
query59	3086	3231	3245	3231
query60	282	289	249	249
query61	108	106	108	106
query62	884	791	737	737
query63	222	193	189	189
query64	4558	1021	630	630
query65	3288	3181	3273	3181
query66	993	433	316	316
query67	15878	15939	15482	15482
query68	9693	697	516	516
query69	504	289	263	263
query70	1277	1168	1122	1122
query71	482	284	265	265
query72	6167	3768	3898	3768
query73	994	756	351	351
query74	10398	9495	9192	9192
query75	4219	3168	2661	2661
query76	4622	1172	783	783
query77	779	367	270	270
query78	10131	10144	11223	10144
query79	3402	747	591	591
query80	631	516	425	425
query81	463	277	233	233
query82	225	148	122	122
query83	194	168	153	153
query84	290	91	135	91
query85	734	348	292	292
query86	369	330	305	305
query87	4466	4491	4704	4491
query88	4495	2172	2128	2128
query89	391	323	298	298
query90	1944	182	184	182
query91	130	130	111	111
query92	68	55	54	54
query93	941	789	529	529
query94	663	383	288	288
query95	330	261	251	251
query96	485	606	277	277
query97	2968	3044	2878	2878
query98	232	202	195	195
query99	1748	1567	1424	1424
Total cold run time: 299615 ms
Total hot run time: 191245 ms

doris-robot · 2025-01-03T07:50:39Z

ClickBench: Total hot run time: 31.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5c5195a28b3ca6dc013fdcd8dc508a7cd9438e51, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.42	0.42	0.42
query6	1.13	0.65	0.64
query7	0.03	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.50
query10	0.55	0.56	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.12
query13	0.62	0.62	0.59
query14	2.84	2.84	2.71
query15	0.89	0.81	0.82
query16	0.39	0.39	0.37
query17	1.05	1.01	1.04
query18	0.22	0.20	0.20
query19	1.98	1.97	1.90
query20	0.01	0.01	0.02
query21	15.35	0.96	0.59
query22	0.76	1.01	0.70
query23	15.06	1.41	0.53
query24	3.27	1.13	0.97
query25	0.30	0.08	0.06
query26	0.24	0.15	0.14
query27	0.05	0.05	0.04
query28	13.73	1.54	1.06
query29	12.58	3.96	3.28
query30	0.26	0.09	0.08
query31	2.80	0.60	0.37
query32	3.23	0.54	0.46
query33	3.08	3.10	3.17
query34	16.67	5.12	4.49
query35	4.52	4.47	4.50
query36	0.85	0.51	0.50
query37	0.10	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.23 s
Total hot run time: 31.35 s

morningman · 2025-01-03T08:54:18Z

fe/fe-core/src/main/java/org/apache/doris/planner/HiveTableSink.java

            locationParams.setWritePath(storageLocation);
            locationParams.setOriginalWritePath(location);
-            locationParams.setTargetPath(location);
+            locationParams.setOriginalWritePath(sd.getLocation());


duplicate call of setOriginalWritePath

wuwenchi · 2025-01-03T09:01:18Z

run buildall

doris-robot · 2025-01-03T10:48:12Z

TPC-H: Total hot run time: 34032 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 010d2d41b6cab45f8891f8baf7801b10dfcafbc2, data reload: false

------ Round 1 ----------------------------------
q1	17640	6591	6332	6332
q2	2055	313	169	169
q3	10500	1371	816	816
q4	10247	918	456	456
q5	8246	2385	2193	2193
q6	236	194	153	153
q7	985	789	618	618
q8	9241	1559	1378	1378
q9	5528	5177	5266	5177
q10	6831	2342	1886	1886
q11	503	286	262	262
q12	376	397	225	225
q13	17766	3866	3086	3086
q14	237	231	215	215
q15	578	512	497	497
q16	645	633	591	591
q17	601	909	337	337
q18	7274	6416	6532	6416
q19	2380	1112	607	607
q20	334	334	193	193
q21	3169	2349	2108	2108
q22	380	345	317	317
Total cold run time: 105752 ms
Total hot run time: 34032 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6778	6655	6778	6655
q2	255	339	234	234
q3	2306	2800	2368	2368
q4	1482	1941	1395	1395
q5	4513	5157	5019	5019
q6	219	192	146	146
q7	2199	2014	1846	1846
q8	2865	3048	3007	3007
q9	7406	7340	7326	7326
q10	3117	3310	2875	2875
q11	637	526	507	507
q12	695	786	590	590
q13	3627	4027	3303	3303
q14	291	306	272	272
q15	599	522	501	501
q16	669	715	656	656
q17	1333	1895	1308	1308
q18	7853	7544	7467	7467
q19	937	1381	1266	1266
q20	2078	2055	1868	1868
q21	5955	5303	5140	5140
q22	668	633	622	622
Total cold run time: 56482 ms
Total hot run time: 54371 ms

doris-robot · 2025-01-03T11:00:31Z

TPC-DS: Total hot run time: 197517 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 010d2d41b6cab45f8891f8baf7801b10dfcafbc2, data reload: false

query1	1339	993	929	929
query2	6421	2413	2324	2324
query3	10958	4939	4764	4764
query4	33121	23976	23437	23437
query5	3751	614	474	474
query6	269	191	185	185
query7	4009	492	311	311
query8	313	249	245	245
query9	9357	2684	2671	2671
query10	484	319	252	252
query11	17917	15534	15219	15219
query12	166	113	105	105
query13	1578	535	437	437
query14	9856	6888	7386	6888
query15	243	207	193	193
query16	8015	618	452	452
query17	1507	791	593	593
query18	2074	409	320	320
query19	211	184	171	171
query20	125	113	123	113
query21	205	133	111	111
query22	4633	4792	4498	4498
query23	35406	33923	33911	33911
query24	6319	2346	2310	2310
query25	478	443	428	428
query26	729	258	166	166
query27	2083	479	335	335
query28	5246	2483	2483	2483
query29	578	545	450	450
query30	214	188	164	164
query31	1033	920	849	849
query32	75	58	54	54
query33	508	362	308	308
query34	753	864	503	503
query35	821	824	790	790
query36	1027	1064	965	965
query37	121	102	78	78
query38	4328	4302	4090	4090
query39	1570	1497	1493	1493
query40	199	122	106	106
query41	47	46	43	43
query42	126	112	110	110
query43	525	542	505	505
query44	1322	842	848	842
query45	196	176	173	173
query46	909	1085	665	665
query47	2041	1997	1959	1959
query48	398	426	330	330
query49	712	497	424	424
query50	654	683	402	402
query51	7328	7346	7243	7243
query52	103	101	103	101
query53	229	251	189	189
query54	488	494	427	427
query55	91	81	79	79
query56	249	276	247	247
query57	1279	1220	1205	1205
query58	244	232	234	232
query59	3109	3326	3206	3206
query60	270	268	258	258
query61	110	112	110	110
query62	867	827	775	775
query63	232	198	189	189
query64	3270	1029	696	696
query65	3348	3264	3263	3263
query66	824	422	305	305
query67	16653	15912	15628	15628
query68	7406	715	520	520
query69	489	299	264	264
query70	1256	1168	1175	1168
query71	433	288	255	255
query72	6357	3961	3849	3849
query73	645	754	358	358
query74	10479	9163	9022	9022
query75	3916	3164	2684	2684
query76	3579	1176	786	786
query77	751	357	286	286
query78	10190	10157	9384	9384
query79	3295	845	595	595
query80	628	529	430	430
query81	472	276	226	226
query82	587	157	117	117
query83	160	166	145	145
query84	236	95	83	83
query85	878	358	307	307
query86	381	313	275	275
query87	4502	4337	4608	4337
query88	4933	2190	2167	2167
query89	414	346	299	299
query90	1771	195	188	188
query91	132	136	104	104
query92	69	57	55	55
query93	2027	883	534	534
query94	649	375	256	256
query95	333	274	262	262
query96	484	609	280	280
query97	2921	3074	2873	2873
query98	227	207	205	205
query99	1448	1519	1422	1422
Total cold run time: 295712 ms
Total hot run time: 197517 ms

doris-robot · 2025-01-03T11:05:51Z

ClickBench: Total hot run time: 31.42 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 010d2d41b6cab45f8891f8baf7801b10dfcafbc2, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.41	0.42
query6	1.16	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.60	0.51	0.51
query10	0.57	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.59
query14	2.81	2.76	2.74
query15	0.89	0.83	0.82
query16	0.37	0.37	0.37
query17	1.06	1.04	1.07
query18	0.23	0.21	0.20
query19	1.89	1.80	2.02
query20	0.01	0.01	0.02
query21	15.37	0.97	0.60
query22	0.76	0.82	0.66
query23	15.26	1.53	0.58
query24	3.27	1.63	1.09
query25	0.16	0.27	0.08
query26	0.20	0.15	0.14
query27	0.07	0.06	0.04
query28	13.66	1.53	1.05
query29	12.61	3.96	3.29
query30	0.25	0.09	0.07
query31	2.82	0.59	0.37
query32	3.23	0.54	0.45
query33	3.07	3.13	3.13
query34	16.65	5.12	4.46
query35	4.48	4.45	4.43
query36	0.65	0.48	0.51
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.94 s
Total hot run time: 31.42 s

github-actions · 2025-01-07T03:48:13Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-01-07T03:48:15Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Problem Summary: For the "original_write_path" field, when the location is of the S3 type, it should not be converted into the S3 schema. Instead, the original location should be retained.

Cherry-picked from #46362 Co-authored-by: wuwenchi <wuwenchi@selectdb.com>

### What problem does this PR solve? Problem Summary: For the "original_write_path" field, when the location is of the S3 type, it should not be converted into the S3 schema. Instead, the original location should be retained.

bp: #46362

### What problem does this PR solve? Related PR: #46362 Problem Summary: For the "target_path" field, when the location is of the S3 type, it should not be converted into the S3 schema.

Related PR: apache#46362 Problem Summary: For the "target_path" field, when the location is of the S3 type, it should not be converted into the S3 schema.

### What problem does this PR solve? Related PR: apache#46362 Problem Summary: For the "target_path" field, when the location is of the S3 type, it should not be converted into the S3 schema.

fix

5c5195a

morningman added dev/2.1.x dev/3.0.x labels Jan 3, 2025

morningman reviewed Jan 3, 2025

View reviewed changes

fix

010d2d4

morningman approved these changes Jan 7, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 7, 2025

github-actions bot added the reviewed label Jan 7, 2025

CalvinKirs approved these changes Jan 7, 2025

View reviewed changes

morningman merged commit 131742d into apache:master Jan 7, 2025
25 of 26 checks passed

github-actions bot mentioned this pull request Jan 7, 2025

branch-3.0: [fix](hive) Incorrect location conversion #46362 #46515

Merged

github-actions bot mentioned this pull request Jan 7, 2025

branch-2.1: [fix](hive) Incorrect location conversion #46362 #46516

Closed

morningman pushed a commit that referenced this pull request Jan 10, 2025

branch-3.0: [fix](hive) Incorrect location conversion #46362 (#46515)

65af892

Cherry-picked from #46362 Co-authored-by: wuwenchi <wuwenchi@selectdb.com>

morningman added dev/3.0.4-merged and removed dev/3.0.x labels Jan 10, 2025

wuwenchi mentioned this pull request Jan 10, 2025

[fix](hive) Incorrect location conversion for 2.1 (#46362) #46754

Merged

yiguolei pushed a commit that referenced this pull request Jan 10, 2025

[fix](hive) Incorrect location conversion for 2.1 (#46362) (#46754)

6c47898

bp: #46362

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Jan 10, 2025

wuwenchi mentioned this pull request Jan 17, 2025

[fix](hive) Incorrect location conversion for target path #47162

Merged

16 tasks

[fix](hive) Incorrect location conversion #46362

[fix](hive) Incorrect location conversion #46362

Uh oh!

Conversation

wuwenchi commented Jan 3, 2025

What problem does this PR solve?

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 3, 2025

Uh oh!

wuwenchi commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

morningman Jan 3, 2025

Choose a reason for hiding this comment

Uh oh!

wuwenchi commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

github-actions bot commented Jan 7, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants