Skip to content

[branch-2.0][opt](jdbc catalog) opt oceanbase jdbc client #34936

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 17, 2024
Merged

[branch-2.0][opt](jdbc catalog) opt oceanbase jdbc client #34936

morningman merged 1 commit into from
May 17, 2024

Conversation

@zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented May 15, 2024 •
edited
Loading

Proposed changes

Issue Number: close #xxx

Removed the OB Client proxy and used the factory directly

mysql> select * from ob_mysql.test.all_types;
+-----------+------------+-------------+-------+----------+-----------+-----------+----------+---------+---------+----------+------+-----------+------+--------+------------+--------------------------+---------------------+----------+------------------+---------+------+---------+--------------+-------+--------------------------+-----------------------------------------------+-----------------+------+----------------------------+--------------+--------+
| tinyint_u | smallint_u | mediumint_u | int_u | bigint_u | decimal_u | double_u  | float_u  | boolean | tinyint | smallint | year | mediumint | int  | bigint | date       | timestamp                | datetime            | float    | double           | decimal | char | varchar | time         | text  | blob                     | json                                          | set             | bit  | binary                     | varbinary    | enum   |
+-----------+------------+-------------+-------+----------+-----------+-----------+----------+---------+---------+----------+------+-----------+------+--------+------------+--------------------------+---------------------+----------+------------------+---------+------+---------+--------------+-------+--------------------------+-----------------------------------------------+-----------------+------+----------------------------+--------------+--------+
|       201 |        301 |         401 |   501 | 601      |   3.14159 | 4.1415926 | 5.141592 |       1 |    -123 |     -301 | 2012 |      -401 | -501 |   -601 | 2012-10-30 | 2012-10-25 12:05:36.3457 | 2012-10-25 08:08:08 | -4.14145 |    -5.1400000001 | -6.1400 | row1 | line1   | 09:09:09.567 | text1 | 0x48656C6C6F20576F726C64 | {"age":30,"city":"London","name":"Alice"}     | Option1,Option3 | 0x2A | 0x48656C6C6F00000000000000 | 0x48656C6C6F | Value2 |
|       202 |        302 |         402 |   502 | 602      |   4.14159 | 5.1415926 | 6.141592 |       0 |    -124 |     -302 | 2013 |      -402 | -502 |   -602 | 2012-11-01 | 2012-10-26 02:08:39.3457 | 2013-10-26 08:09:18 | -5.14145 |    -6.1400000001 | -7.1400 | row2 | line2   | 09:11:09.567 | text2 | 0xE86F6C6C6F20576F726C67 | {"age":18,"city":"ChongQing","name":"Gaoxin"} | Option1,Option2 | 0x2F | 0x58676C6C6F00000000000000 | 0x88656C6C9F | Value3 |
|      NULL |        302 |        NULL |   502 | 602      |   4.14159 |      NULL | 6.141592 |    NULL |    -124 |     -302 | 2013 |      -402 | -502 |   -602 | NULL       | 2012-10-26 02:08:39.3457 | 2013-10-26 08:09:18 | -5.14145 |             NULL | -7.1400 | row2 | NULL    | 09:11:09.567 | text2 | 0xE86F6C6C6F20576F726C67 | NULL                                          | NULL            | 0x2F | NULL                       | 0x88656C6C9F | Value3 |
|       203 |        303 |         403 |   503 | 603      |   7.14159 | 8.1415926 | 9.141592 |       0 |    NULL |     -402 | 2017 |      -602 | -902 |  -1102 | 2012-11-02 | NULL                     | 2013-10-27 08:11:18 | -5.14145 | -6.1400000000001 | -7.1400 | row3 | line3   | 09:11:09.567 | text3 | 0xE86F6C6C6F20576F726C67 | {"age":24,"city":"ChongQing","name":"ChenQi"} | Option2         | 0x2F | 0x58676C6C6F00000000000000 | NULL         | Value1 |
+-----------+------------+-------------+-------+----------+-----------+-----------+----------+---------+---------+----------+------+-----------+------+--------+------------+--------------------------+---------------------+----------+------------------+---------+------+---------+--------------+-------+--------------------------+-----------------------------------------------+-----------------+------+----------------------------+--------------+--------+
4 rows in set (0.97 sec)

mysql> select * from ob_oracle.ZYK.TEST_ALL_TYPES;
+------+------+------+------------+------+------+--------+---------+------+--------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+---------+--------+---------+-------+--------+-------+-----------+---------------------+-------------------------+----------------------------+----------------------------+----------------------------+-------+-------------------+
| ID   | N1   | N2   | N3         | N4   | N5   | N6     | N7      | N8   | N9     | TINYINT_VALUE1 | SMALLINT_VALUE1 | INT_VALUE1 | BIGINT_VALUE1      | TINYINT_VALUE2 | SMALLINT_VALUE2 | INT_VALUE2 | BIGINT_VALUE2       | COUNTRY | CITY   | ADDRESS | NAME  | NUM1   | NUM2  | NUM4      | T1                  | T2                      | T3                         | T4                         | T5                         | T6    | T7                |
+------+------+------+------------+------+------+--------+---------+------+--------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+---------+--------+---------+-------+--------+-------+-----------+---------------------+-------------------------+----------------------------+----------------------------+----------------------------+-------+-------------------+
| 1    | 111  | 123  | 7456123.89 | 573  | 34   | 673.43 | 34.1264 |   60 | 23.231 |             99 |            9999 |  999999999 | 999999999999999999 |            999 |           99999 | 9999999999 | 9999999999999999999 | 1       | china  | beijing | alice | 123.45 | 12300 | 0.0012345 | 2022-01-21 05:23:01 | 2019-11-12 20:33:57.999 | 2019-11-12 20:33:57.999998 | 2019-11-12 20:33:57.999996 | 2019-11-12 20:33:57.999997 | 223-9 | 12 10:23:1.123457 |
| 2    | NULL | NULL |       NULL | NULL | NULL |   NULL |    NULL | NULL |   NULL |           NULL |            NULL |       NULL |               NULL |           NULL |            NULL |       NULL | NULL                | NULL    | NULL   | NULL    | NULL  |   NULL |  NULL |      NULL | NULL                | NULL                    | NULL                       | NULL                       | NULL                       | NULL  | NULL              |
+------+------+------+------------+------+------+--------+---------+------+--------+----------------+-----------------+------------+--------------------+----------------+-----------------+------------+---------------------+---------+--------+---------+-------+--------+-------+-----------+---------------------+-------------------------+----------------------------+----------------------------+----------------------------+-------+-------------------+
2 rows in set (1.16 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zy-kkk
Copy link
Member Author

zy-kkk commented May 15, 2024

run buildall

@zy-kkk zy-kkk changed the title [opt](jdbc catalog) opt oceanbase jdbc client [branch-2.0][opt](jdbc catalog) opt oceanbase jdbc client May 15, 2024
@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-H: Total hot run time: 49348 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 83cb4b36a3f42bc22ea73efc37f3c047215317e8, data reload: false

------ Round 1 ----------------------------------
q1	18011	4313	4292	4292
q2	2039	158	140	140
q3	10425	1935	1921	1921
q4	10338	1248	1306	1248
q5	8239	3874	3908	3874
q6	238	125	123	123
q7	2014	1612	1588	1588
q8	9495	2715	2700	2700
q9	12035	10335	10100	10100
q10	8614	3480	3518	3480
q11	423	241	247	241
q12	460	299	295	295
q13	18355	3970	3997	3970
q14	349	323	334	323
q15	493	461	456	456
q16	674	562	565	562
q17	1139	947	934	934
q18	7319	6858	6918	6858
q19	1701	1587	1549	1549
q20	525	332	294	294
q21	4394	4113	4014	4014
q22	493	409	386	386
Total cold run time: 117773 ms
Total hot run time: 49348 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4315	4268	4285	4268
q2	315	228	217	217
q3	4162	4182	4130	4130
q4	2748	2728	2740	2728
q5	7122	7091	7064	7064
q6	238	118	120	118
q7	3215	2879	2869	2869
q8	4358	4448	4508	4448
q9	16882	16784	16625	16625
q10	4250	4260	4265	4260
q11	751	699	662	662
q12	1053	844	847	844
q13	6812	3796	3753	3753
q14	458	445	426	426
q15	515	460	457	457
q16	730	687	681	681
q17	3857	3820	3881	3820
q18	8828	8880	8988	8880
q19	1761	1702	1693	1693
q20	2372	2139	2168	2139
q21	8503	8527	8484	8484
q22	1006	955	919	919
Total cold run time: 84251 ms
Total hot run time: 79485 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

TeamCity be ut coverage result:
Function Coverage: 37.78% (8073/21366)
Line Coverage: 29.44% (65902/223870)
Region Coverage: 28.91% (33925/117359)
Branch Coverage: 24.77% (17413/70296)
Coverage Report: http://coverage.selectdb-in.cc/coverage/83cb4b36a3f42bc22ea73efc37f3c047215317e8_83cb4b36a3f42bc22ea73efc37f3c047215317e8/report/index.html

@doris-robot
Copy link

doris-robot commented May 15, 2024

TPC-DS: Total hot run time: 202120 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 83cb4b36a3f42bc22ea73efc37f3c047215317e8, data reload: false

query1	962	405	391	391
query2	6745	2652	2827	2652
query3	8867	219	209	209
query4	21671	18261	17895	17895
query5	19739	6514	6472	6472
query6	411	223	237	223
query7	5211	305	297	297
query8	232	237	233	233
query9	3120	2686	2611	2611
query10	428	306	313	306
query11	11391	10727	10710	10710
query12	122	75	69	69
query13	5603	706	682	682
query14	19108	13395	13524	13395
query15	363	213	226	213
query16	6434	275	257	257
query17	1328	1554	853	853
query18	2287	410	408	408
query19	213	147	147	147
query20	80	75	85	75
query21	188	93	95	93
query22	5213	4997	4993	4993
query23	32356	32085	31626	31626
query24	6801	6545	6441	6441
query25	511	427	433	427
query26	466	158	157	157
query27	1192	300	290	290
query28	5707	2362	2333	2333
query29	2872	2818	2875	2818
query30	236	166	161	161
query31	865	743	737	737
query32	63	64	60	60
query33	340	284	238	238
query34	799	470	489	470
query35	1057	933	945	933
query36	1296	1069	1160	1069
query37	87	62	61	61
query38	3042	2923	2910	2910
query39	1361	1347	1318	1318
query40	208	93	91	91
query41	46	36	36	36
query42	85	92	88	88
query43	735	721	648	648
query44	1064	709	708	708
query45	238	228	224	224
query46	1209	959	1003	959
query47	1830	1831	1743	1743
query48	992	732	701	701
query49	612	376	377	376
query50	849	582	600	582
query51	4795	4725	4626	4626
query52	97	82	79	79
query53	444	313	312	312
query54	2648	2444	2464	2444
query55	90	80	75	75
query56	225	201	202	201
query57	1235	1071	1019	1019
query58	211	207	202	202
query59	4192	4270	4031	4031
query60	210	210	198	198
query61	90	83	84	83
query62	812	441	505	441
query63	479	334	337	334
query64	2187	1353	1411	1353
query65	3635	3540	3569	3540
query66	781	370	374	370
query67	15769	15948	14863	14863
query68	8861	652	637	637
query69	568	330	346	330
query70	1532	1387	1385	1385
query71	423	299	321	299
query72	6449	3428	3474	3428
query73	754	343	324	324
query74	6303	5850	5866	5850
query75	5282	3708	3661	3661
query76	5241	1145	1181	1145
query77	943	260	254	254
query78	12708	11557	11855	11557
query79	6906	660	651	651
query80	1003	403	395	395
query81	486	233	230	230
query82	1628	98	95	95
query83	182	133	127	127
query84	260	71	70	70
query85	817	309	296	296
query86	330	296	294	294
query87	3204	3035	2990	2990
query88	4930	2349	2337	2337
query89	344	281	298	281
query90	1932	217	199	199
query91	168	140	136	136
query92	56	54	54	54
query93	2988	561	565	561
query94	710	211	208	208
query95	1139	1066	1060	1060
query96	654	326	329	326
query97	6445	6407	6297	6297
query98	193	176	173	173
query99	2913	910	915	910
Total cold run time: 310887 ms
Total hot run time: 202120 ms

@doris-robot
Copy link

doris-robot commented May 15, 2024

ClickBench: Total hot run time: 30.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 83cb4b36a3f42bc22ea73efc37f3c047215317e8, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.08	0.07
query5	0.54	0.52	0.51
query6	1.29	0.62	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.51	0.46	0.49
query10	0.54	0.52	0.53
query11	0.12	0.09	0.08
query12	0.12	0.09	0.09
query13	0.62	0.63	0.62
query14	0.78	0.77	0.79
query15	0.79	0.76	0.76
query16	0.35	0.37	0.39
query17	1.00	0.97	1.00
query18	0.25	0.23	0.26
query19	1.86	1.81	1.73
query20	0.01	0.00	0.00
query21	15.47	0.54	0.53
query22	1.91	2.67	1.63
query23	17.37	1.08	0.99
query24	5.46	1.03	1.21
query25	0.39	0.09	0.04
query26	0.71	0.17	0.17
query27	0.04	0.05	0.03
query28	7.40	0.74	0.80
query29	12.65	2.40	2.23
query30	0.58	0.50	0.51
query31	2.80	0.40	0.39
query32	3.38	0.51	0.49
query33	3.11	3.05	3.07
query34	15.27	4.80	4.82
query35	4.87	4.83	4.83
query36	1.06	1.01	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.02
query43	0.03	0.01	0.01
Total cold run time: 103.83 s
Total hot run time: 30.62 s

@doris-robot
Copy link

doris-robot commented May 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 83cb4b36a3f42bc22ea73efc37f3c047215317e8 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

morningman
morningman approved these changes May 17, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 8eeee84 into apache:branch-2.0 May 17, 2024
@zy-kkk zy-kkk deleted the ob_opt branch May 18, 2024 10:51
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zy-kkk
[opt](jdbc catalog) opt oceanbase jdbc client (apache#34936)
81b6a1c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

Assignees

No one assigned

Labels

dev/2.0.12-merged

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zy-kkk @doris-robot @morningman