Skip to content

[conf](mysql) opt mysql network timeout to 600s #32545

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 3 commits into from
Mar 22, 2024
Merged

[conf](mysql) opt mysql network timeout to 600s #32545

yiguolei merged 3 commits into from
Mar 22, 2024

Conversation

@cambyzju
Copy link
Contributor

@cambyzju cambyzju commented Mar 20, 2024 •
edited
Loading

Proposed changes

Now netWriteTimeout and netReadTimeout only used in MysqlChannel, from pr: #28808

First of all, mysql client may be in different location as FE, the network maybe slow, not like FE and BE always have a fast network.
Secondly, the max packet size is 16MB in Mysql Protocol, not very small.

So we should enlarge this timeout.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@cambyzju
Copy link
Contributor Author

cambyzju commented Mar 20, 2024

issue:
#28807
#32539

@cambyzju
Copy link
Contributor Author

cambyzju commented Mar 20, 2024

run buildall

@cambyzju
Copy link
Contributor Author

cambyzju commented Mar 21, 2024

run external

@cambyzju
Copy link
Contributor Author

cambyzju commented Mar 21, 2024

run buildall

@cambyzju
Copy link
Contributor Author

cambyzju commented Mar 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 22, 2024

TPC-H: Total hot run time: 37953 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18255d2bf23718391459378701749505760c4301, data reload: false

------ Round 1 ----------------------------------
q1	17911	4392	4207	4207
q2	2102	153	156	153
q3	11087	1154	1272	1154
q4	10372	748	799	748
q5	7575	3032	2986	2986
q6	205	125	122	122
q7	1037	572	562	562
q8	9344	2018	2003	2003
q9	7060	6520	6475	6475
q10	8303	3322	3534	3322
q11	444	224	222	222
q12	371	199	191	191
q13	17789	2846	2855	2846
q14	226	204	213	204
q15	506	456	454	454
q16	437	359	362	359
q17	976	547	591	547
q18	7342	6507	6692	6507
q19	1557	1443	1483	1443
q20	553	258	264	258
q21	3573	2933	2880	2880
q22	371	310	312	310
Total cold run time: 109141 ms
Total hot run time: 37953 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4101	4083	4054	4054
q2	323	226	225	225
q3	2979	2839	2838	2838
q4	1851	1547	1571	1547
q5	5249	5270	5277	5270
q6	191	114	115	114
q7	2228	1838	1836	1836
q8	3148	3280	3302	3280
q9	8615	8589	8560	8560
q10	3735	3694	3680	3680
q11	560	445	443	443
q12	725	556	535	535
q13	16931	2874	2812	2812
q14	284	255	257	255
q15	494	464	452	452
q16	453	412	415	412
q17	1741	1487	1461	1461
q18	7431	7170	7043	7043
q19	1619	1580	1451	1451
q20	1916	1729	1715	1715
q21	4850	4691	4785	4691
q22	517	449	432	432
Total cold run time: 69941 ms
Total hot run time: 53106 ms

@doris-robot
Copy link

doris-robot commented Mar 22, 2024

TPC-DS: Total hot run time: 181138 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18255d2bf23718391459378701749505760c4301, data reload: false

query1	934	378	354	354
query2	7401	2114	1941	1941
query3	6703	216	210	210
query4	31459	20762	20996	20762
query5	4375	424	400	400
query6	267	176	175	175
query7	4621	299	285	285
query8	228	164	168	164
query9	8860	2314	2282	2282
query10	474	242	251	242
query11	17126	14129	14179	14129
query12	138	87	84	84
query13	1648	408	426	408
query14	12491	11317	10517	10517
query15	282	208	188	188
query16	8242	265	263	263
query17	1973	564	540	540
query18	2105	291	275	275
query19	347	158	156	156
query20	92	86	91	86
query21	199	127	127	127
query22	4723	4452	4460	4452
query23	32040	31211	31217	31211
query24	10714	2785	2830	2785
query25	638	374	392	374
query26	1367	153	155	153
query27	2978	350	362	350
query28	7487	1926	1895	1895
query29	879	687	624	624
query30	301	145	144	144
query31	938	731	725	725
query32	97	56	55	55
query33	777	260	251	251
query34	1037	466	474	466
query35	844	608	598	598
query36	990	889	851	851
query37	115	75	71	71
query38	3562	3443	3418	3418
query39	1411	1379	1415	1379
query40	214	116	109	109
query41	49	45	45	45
query42	99	98	97	97
query43	492	449	439	439
query44	1175	730	726	726
query45	258	242	264	242
query46	1099	687	692	687
query47	1658	1624	1612	1612
query48	430	352	362	352
query49	1098	331	329	329
query50	756	375	375	375
query51	6796	6600	6571	6571
query52	110	89	92	89
query53	335	283	280	280
query54	296	245	238	238
query55	85	74	77	74
query56	238	257	223	223
query57	1082	1026	1015	1015
query58	236	201	210	201
query59	2662	2700	2506	2506
query60	269	251	256	251
query61	107	94	93	93
query62	580	418	417	417
query63	302	272	275	272
query64	5515	3779	3822	3779
query65	3046	3000	2972	2972
query66	980	348	340	340
query67	14854	14680	14379	14379
query68	5466	512	505	505
query69	548	367	364	364
query70	1193	1104	1207	1104
query71	422	289	278	278
query72	6330	2792	2620	2620
query73	705	316	328	316
query74	7950	6578	6627	6578
query75	3473	2852	2793	2793
query76	3537	874	905	874
query77	443	276	260	260
query78	10438	9530	9654	9530
query79	8621	513	519	513
query80	2062	426	417	417
query81	551	217	214	214
query82	1388	218	201	201
query83	320	154	160	154
query84	288	73	75	73
query85	1664	324	304	304
query86	488	310	308	308
query87	3713	3538	3533	3533
query88	5052	2316	2286	2286
query89	537	362	367	362
query90	1992	174	176	174
query91	184	136	135	135
query92	60	47	46	46
query93	6921	504	503	503
query94	1232	185	178	178
query95	433	338	329	329
query96	597	261	272	261
query97	3071	2871	2891	2871
query98	215	211	204	204
query99	1086	747	794	747
Total cold run time: 306664 ms
Total hot run time: 181138 ms

@doris-robot
Copy link

doris-robot commented Mar 22, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 18255d2bf23718391459378701749505760c4301 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       20.4 seconds inserted 10000000 Rows, about 490K ops/s

xy720
xy720 approved these changes Mar 22, 2024
View reviewed changes
Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 22, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 734473c into apache:master Mar 22, 2024
yiguolei pushed a commit that referenced this pull request Mar 24, 2024
@cambyzju @Doris-Extras
[conf](mysql) opt mysql network timeout to 600s #32545
5077f7d
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Mar 25, 2024
@cambyzju
[conf](mysql) opt mysql network timeout to 600s apache#32545
2ee81fd
@cambyzju cambyzju mentioned this pull request Mar 25, 2024
Merged
cambyzju added a commit to cambyzju/incubator-doris that referenced this pull request Mar 25, 2024
@cambyzju
[conf](mysql) opt mysql network timeout to 600s apache#32545
b7f1f64
@cambyzju cambyzju mentioned this pull request Mar 25, 2024
Merged
cambyzju added a commit that referenced this pull request Mar 29, 2024
@cambyzju
[conf](mysql) opt mysql network timeout to 600s #32545 (#32739)
2ae85e4
cambyzju added a commit that referenced this pull request Mar 29, 2024
@cambyzju @fornaix
[pick-1.2][fix](mysql) fix mysql channel infinite blocking (#32741)
74123e4 
* [fix](mysql) fix mysql channel infinite blocking (#28808)

Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.

* [conf](mysql) opt mysql network timeout to 600s #32545

---------

Co-authored-by: fornaix <foxnaix@gmail.com>
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@cambyzju
[conf](mysql) opt mysql network timeout to 600s apache#32545 (apache#…
8302b5d 
…32739)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@xy720 xy720 xy720 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/1.2.9-merged dev/2.0.8-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cambyzju @doris-robot @yiguolei @xy720