Skip to content

[fix](paimon) fix not able to read paimon data from hdfs with HA #39806

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Aug 24, 2024
Merged

[fix](paimon) fix not able to read paimon data from hdfs with HA #39806

morningman merged 2 commits into from
Aug 24, 2024

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Aug 22, 2024 •
edited
Loading

When reading Paimon data via JNI, the hadoop properties should be passed to JNI part.
Otherwise, it may miss some properties and return error like:
CAUSED BY: IllegalArgumentException: java.net.UnknownHostException: hdfs-cluster

@doris-robot
Copy link

doris-robot commented Aug 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

morningman commented Aug 22, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 22, 2024

clang-tidy review says "All clean, LGTM! 👍"

@morningman morningman added the usercase Important user case type label label Aug 22, 2024
@doris-robot
Copy link

doris-robot commented Aug 22, 2024

TPC-H: Total hot run time: 38053 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a38b076414ce308cebd0422748f30e893934f615, data reload: false

------ Round 1 ----------------------------------
q1	18010	4760	4329	4329
q2	2031	175	185	175
q3	11646	986	1109	986
q4	10272	767	665	665
q5	7786	2885	2863	2863
q6	229	141	136	136
q7	989	597	593	593
q8	9333	2082	2109	2082
q9	7122	6572	6574	6572
q10	6990	2281	2195	2195
q11	470	254	253	253
q12	395	216	224	216
q13	17766	3051	3025	3025
q14	280	232	250	232
q15	527	485	492	485
q16	509	414	395	395
q17	1007	715	719	715
q18	7223	6803	6805	6803
q19	1395	1136	1000	1000
q20	673	334	345	334
q21	4012	3010	3240	3010
q22	1084	989	1000	989
Total cold run time: 109749 ms
Total hot run time: 38053 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4353	4348	4315	4315
q2	380	302	281	281
q3	2842	2658	2699	2658
q4	1951	1725	1663	1663
q5	5643	5716	5711	5711
q6	229	140	134	134
q7	2246	1835	1805	1805
q8	3287	3432	3453	3432
q9	8850	8872	8922	8872
q10	3522	3407	3373	3373
q11	610	503	526	503
q12	838	685	670	670
q13	15671	3280	3089	3089
q14	332	287	287	287
q15	550	480	498	480
q16	505	454	460	454
q17	1859	1560	1530	1530
q18	8058	7858	7851	7851
q19	1788	1677	1509	1509
q20	2127	1927	1887	1887
q21	5786	5639	5534	5534
q22	1197	1100	1069	1069
Total cold run time: 72624 ms
Total hot run time: 57107 ms

@doris-robot
Copy link

doris-robot commented Aug 22, 2024

TPC-DS: Total hot run time: 192332 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a38b076414ce308cebd0422748f30e893934f615, data reload: false

query1	1263	890	873	873
query2	6208	1968	1938	1938
query3	10617	3974	3991	3974
query4	59544	27094	23189	23189
query5	5372	502	500	500
query6	419	160	167	160
query7	5770	287	287	287
query8	283	209	221	209
query9	8970	2495	2483	2483
query10	498	277	263	263
query11	18184	15118	15410	15118
query12	174	104	113	104
query13	1556	408	398	398
query14	11444	7415	7440	7415
query15	249	169	185	169
query16	7194	470	488	470
query17	1113	563	572	563
query18	2030	307	307	307
query19	290	166	153	153
query20	116	116	112	112
query21	199	106	104	104
query22	4637	4522	4371	4371
query23	34121	33635	33645	33635
query24	5911	2851	2816	2816
query25	534	373	377	373
query26	685	159	160	159
query27	1818	277	276	276
query28	3732	2054	2088	2054
query29	652	410	418	410
query30	241	151	158	151
query31	898	781	754	754
query32	86	55	52	52
query33	460	281	281	281
query34	833	467	482	467
query35	845	710	733	710
query36	1052	956	921	921
query37	144	86	85	85
query38	4043	3898	3887	3887
query39	1457	1390	1399	1390
query40	223	118	117	117
query41	48	51	46	46
query42	123	98	103	98
query43	509	491	467	467
query44	1099	741	744	741
query45	192	167	167	167
query46	1086	745	740	740
query47	1870	1805	1804	1804
query48	380	300	297	297
query49	772	437	443	437
query50	803	425	417	417
query51	7235	7109	7037	7037
query52	105	88	92	88
query53	257	184	179	179
query54	566	512	456	456
query55	80	74	75	74
query56	272	272	252	252
query57	1173	1056	1045	1045
query58	214	231	242	231
query59	3110	2762	2797	2762
query60	295	268	267	267
query61	105	97	97	97
query62	763	672	663	663
query63	214	189	187	187
query64	4145	2261	1743	1743
query65	3228	3178	3197	3178
query66	673	328	331	328
query67	15369	15082	15168	15082
query68	3028	566	579	566
query69	402	290	285	285
query70	1150	1118	1080	1080
query71	385	273	275	273
query72	2588	2100	2072	2072
query73	701	320	319	319
query74	9136	8890	8881	8881
query75	3366	2714	2711	2711
query76	1398	1006	957	957
query77	530	315	320	315
query78	10392	9245	9057	9057
query79	1053	550	536	536
query80	693	512	520	512
query81	512	229	229	229
query82	280	147	137	137
query83	169	149	147	147
query84	266	78	79	78
query85	683	289	281	281
query86	301	302	305	302
query87	4505	4208	4262	4208
query88	2964	2297	2307	2297
query89	379	285	284	284
query90	1946	194	195	194
query91	124	168	102	102
query92	60	51	56	51
query93	1045	536	546	536
query94	709	304	269	269
query95	330	267	268	267
query96	586	268	268	268
query97	3163	3051	3083	3051
query98	223	210	210	210
query99	1712	1277	1276	1276
Total cold run time: 303330 ms
Total hot run time: 192332 ms

@doris-robot
Copy link

doris-robot commented Aug 22, 2024

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a38b076414ce308cebd0422748f30e893934f615, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.04	0.05
query4	1.69	0.07	0.07
query5	0.50	0.50	0.48
query6	1.12	0.74	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.16	0.12	0.12
query13	0.62	0.61	0.58
query14	0.76	0.80	0.79
query15	0.85	0.82	0.82
query16	0.35	0.39	0.38
query17	1.03	1.06	1.05
query18	0.21	0.21	0.20
query19	1.85	1.84	1.85
query20	0.01	0.01	0.02
query21	15.39	0.69	0.68
query22	4.08	7.96	1.70
query23	18.26	1.40	1.28
query24	2.09	0.22	0.23
query25	0.17	0.09	0.08
query26	0.26	0.18	0.18
query27	0.08	0.07	0.08
query28	13.21	1.01	1.00
query29	12.64	3.35	3.34
query30	0.24	0.06	0.05
query31	2.87	0.39	0.40
query32	3.26	0.48	0.47
query33	2.96	2.99	3.01
query34	17.03	4.41	4.37
query35	4.41	4.43	4.42
query36	0.66	0.50	0.47
query37	0.19	0.15	0.17
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.05	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.18 s
Total hot run time: 30.67 s

@github-actions
Copy link
Contributor

github-actions bot commented Aug 23, 2024

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes Aug 23, 2024
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 23, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 23, 2024

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 47e45bf into apache:master Aug 24, 2024
morningman added a commit to morningman/doris that referenced this pull request Aug 24, 2024
@morningman
[fix](paimon) fix not able to read paimon data from hdfs with HA (apa…
2399721 
…che#39806)

When reading Paimon data via JNI, the hadoop properties should be passed
to JNI part.
Otherwise, it may miss some properties and return error like:
`CAUSED BY: IllegalArgumentException: java.net.UnknownHostException:
hdfs-cluster`
@morningman morningman mentioned this pull request Aug 24, 2024
Merged
morningman added a commit that referenced this pull request Aug 24, 2024
@morningman
[fix](paimon) fix not able to read paimon data from hdfs with HA (#39806
14a2a66 
) (#39876)

bp #39806
dataroaring pushed a commit that referenced this pull request Aug 26, 2024
@morningman @dataroaring
[fix](paimon) fix not able to read paimon data from hdfs with HA (#39806
af6c4ac 
)

When reading Paimon data via JNI, the hadoop properties should be passed
to JNI part.
Otherwise, it may miss some properties and return error like:
`CAUSED BY: IllegalArgumentException: java.net.UnknownHostException:
hdfs-cluster`
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kaka11chen kaka11chen kaka11chen approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @doris-robot @kaka11chen @wuwenchi @dataroaring