Skip to content

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Feb 6, 2025

cherry-pick: #46912

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

…e#46912)

Related PR: apache#45806

Problem Summary:

In the previous BE processing of JDBC Driver, the Driver jar will be downloaded to the local cache directory first, and the cached jar package will be provided to the JVM for loading. This will cause two problems
1. The jar package in the cache may fail to load due to duplication
2. Frequent repeated loading of the driver by the JVM will cause Compressed class space OOM

In order to fix these two problems, this PR has the following changes
1. Remove the logic of BE downloading the Driver Jar to the local cache directory, and directly hand over the original path to Java's Classloader for processing
2. Treat the jar packages with the same name in the same path as one and cache them in the map to avoid repeated loading and cause Compressed class space OOM
@zy-kkk zy-kkk marked this pull request as ready for review February 25, 2025 02:26
@zy-kkk zy-kkk requested a review from dataroaring as a code owner February 25, 2025 02:27
@zy-kkk
Copy link
Member Author

zy-kkk commented Feb 25, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.36% (1060/1287)
Line Coverage: 65.88% (17581/26686)
Region Coverage: 65.38% (8661/13248)
Branch Coverage: 55.22% (4665/8448)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aab07ab7821df1266e197f5c37fa93e2c4889942_aab07ab7821df1266e197f5c37fa93e2c4889942_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 40877 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aab07ab7821df1266e197f5c37fa93e2c4889942, data reload: false

------ Round 1 ----------------------------------
q1	17896	7488	7262	7262
q2	2074	178	175	175
q3	10611	1102	1235	1102
q4	10464	723	760	723
q5	7773	2851	2844	2844
q6	234	143	141	141
q7	991	615	603	603
q8	9348	1970	2067	1970
q9	6658	6437	6458	6437
q10	6991	2324	2382	2324
q11	468	276	270	270
q12	399	213	207	207
q13	17789	3039	3011	3011
q14	251	205	214	205
q15	564	523	528	523
q16	659	588	597	588
q17	998	525	573	525
q18	7248	6764	6600	6600
q19	1385	1122	1016	1016
q20	478	208	201	201
q21	4246	3175	3147	3147
q22	1114	1035	1003	1003
Total cold run time: 108639 ms
Total hot run time: 40877 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7259	7193	7198	7193
q2	332	240	232	232
q3	2957	2903	2958	2903
q4	2075	1810	1819	1810
q5	5777	5733	5754	5733
q6	223	136	140	136
q7	2218	1822	1831	1822
q8	3391	3568	3577	3568
q9	8775	8957	8875	8875
q10	3646	3565	3576	3565
q11	598	489	499	489
q12	812	626	604	604
q13	9729	3228	3184	3184
q14	309	269	303	269
q15	566	522	522	522
q16	741	644	634	634
q17	1856	1631	1605	1605
q18	8353	7789	7649	7649
q19	1677	1587	1532	1532
q20	2110	1878	1862	1862
q21	5545	5315	5424	5315
q22	1137	1066	1034	1034
Total cold run time: 70086 ms
Total hot run time: 60536 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197957 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aab07ab7821df1266e197f5c37fa93e2c4889942, data reload: false

query1	1360	922	913	913
query2	6217	2003	2005	2003
query3	10952	4335	4306	4306
query4	66458	28341	23587	23587
query5	4996	440	437	437
query6	424	175	178	175
query7	5535	317	315	315
query8	299	214	230	214
query9	8629	2722	2713	2713
query10	449	282	254	254
query11	17162	15260	15706	15260
query12	154	107	104	104
query13	1447	447	420	420
query14	10401	7465	8158	7465
query15	206	182	184	182
query16	7082	516	487	487
query17	1061	600	593	593
query18	1698	326	339	326
query19	217	173	177	173
query20	114	111	114	111
query21	207	106	108	106
query22	5005	4699	4462	4462
query23	34794	33939	34544	33939
query24	6157	2975	2970	2970
query25	514	399	403	399
query26	659	174	166	166
query27	1874	359	353	353
query28	4274	2506	2460	2460
query29	677	441	446	441
query30	244	167	170	167
query31	1017	803	846	803
query32	69	54	55	54
query33	401	284	280	280
query34	909	507	529	507
query35	883	721	749	721
query36	1087	976	964	964
query37	109	64	70	64
query38	4105	3925	3961	3925
query39	1520	1460	1471	1460
query40	208	99	100	99
query41	53	54	50	50
query42	114	99	103	99
query43	549	505	498	498
query44	1179	856	838	838
query45	188	164	169	164
query46	1142	727	728	727
query47	2047	1960	1951	1951
query48	488	386	378	378
query49	735	414	401	401
query50	864	428	437	428
query51	7314	7273	7075	7075
query52	98	88	84	84
query53	272	180	183	180
query54	550	450	455	450
query55	76	78	81	78
query56	247	256	252	252
query57	1260	1101	1123	1101
query58	222	203	204	203
query59	3000	2962	2941	2941
query60	289	256	268	256
query61	111	105	104	104
query62	822	725	719	719
query63	220	190	187	187
query64	1388	657	644	644
query65	3253	3253	3192	3192
query66	719	329	308	308
query67	16191	15560	15622	15560
query68	3325	593	596	593
query69	434	267	274	267
query70	1186	1117	1152	1117
query71	354	263	251	251
query72	6082	4206	4199	4199
query73	753	356	352	352
query74	10327	9225	8933	8933
query75	3346	2645	2698	2645
query76	1962	984	1006	984
query77	524	312	282	282
query78	10504	9661	9505	9505
query79	1864	601	612	601
query80	1139	418	416	416
query81	522	239	238	238
query82	214	88	85	85
query83	166	138	144	138
query84	284	72	75	72
query85	989	299	287	287
query86	400	285	283	283
query87	4410	4388	4235	4235
query88	3890	2387	2366	2366
query89	400	293	290	290
query90	2054	191	189	189
query91	199	148	146	146
query92	69	49	52	49
query93	2265	548	554	548
query94	908	299	298	298
query95	362	258	262	258
query96	621	287	290	287
query97	3358	3232	3190	3190
query98	214	205	199	199
query99	1748	1445	1408	1408
Total cold run time: 317838 ms
Total hot run time: 197957 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aab07ab7821df1266e197f5c37fa93e2c4889942, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.23	0.06	0.06
query4	1.64	0.09	0.08
query5	0.52	0.53	0.51
query6	1.14	0.74	0.74
query7	0.03	0.01	0.04
query8	0.06	0.04	0.04
query9	0.54	0.51	0.50
query10	0.56	0.57	0.59
query11	0.16	0.12	0.12
query12	0.16	0.12	0.13
query13	0.60	0.60	0.59
query14	2.87	2.74	2.75
query15	0.90	0.85	0.84
query16	0.39	0.36	0.38
query17	1.08	0.98	1.03
query18	0.19	0.18	0.20
query19	1.88	1.89	2.04
query20	0.02	0.01	0.01
query21	15.35	0.65	0.66
query22	3.78	7.11	2.21
query23	18.26	1.39	1.29
query24	2.22	0.25	0.22
query25	0.15	0.08	0.08
query26	0.28	0.18	0.17
query27	0.09	0.08	0.08
query28	13.17	1.16	1.14
query29	12.64	3.35	3.32
query30	0.24	0.06	0.06
query31	2.89	0.41	0.39
query32	3.25	0.49	0.49
query33	3.01	3.02	3.06
query34	16.73	4.52	4.57
query35	4.63	4.62	4.57
query36	0.67	0.50	0.50
query37	0.20	0.17	0.17
query38	0.18	0.15	0.15
query39	0.06	0.04	0.04
query40	0.17	0.14	0.13
query41	0.11	0.06	0.05
query42	0.07	0.05	0.05
query43	0.06	0.05	0.05
Total cold run time: 111.3 s
Total hot run time: 33.76 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit b718d91 into apache:branch-3.0 Feb 25, 2025
21 of 23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants