Skip to content

Conversation

@fornaix
Copy link
Contributor

@fornaix fornaix commented Dec 21, 2023

Proposed changes

Issue Number: close #28807

Further comments

Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.

@yiguolei
Copy link
Contributor

@fornaix Hi hi, could you please describe the problem?
Because I meet this problem very frequently "The last packet successfully received from the server was 1,401,847 milliseconds ago. The last packet sent successfully to the server was 1,401,848 milliseconds ago.
"

Does it relate with this PR?

@fornaix
Copy link
Contributor Author

fornaix commented Dec 26, 2023

@yiguolei hi, sure. I found this problem after killing a running query. The query is aborted and there are no running instances in be. But the mysql thread that receives the query is stuck there forever.

I meet this problem very frequently

Is this problem also stuck in channel blocking write or read?

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 26, 2023
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei
Copy link
Contributor

@fornaix actually I do not know. because I could not get the jstack.

I found this problem after killing a running query. The query is aborted and there are no running instances in be. But the mysql thread that receives the query is stuck there forever.

If the query is aborted, and the mysql thread maybe hang in getNext and there is a timeout here. I think I will throw timeout exception.

Could you please explain it more detail?

My wechat number is monkgrandpa . Could you please add me, we could talk about it at wechat?

@yiguolei
Copy link
Contributor

run buildall

@yiguolei yiguolei added usercase Important user case type label dev/2.0.4 labels Dec 28, 2023
@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.2 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17187890961 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit cd0d62333d704c88b96744c2adc1c4e094971f33, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4987	4622	4682	4622
q2	360	152	139	139
q3	1474	1293	1248	1248
q4	1130	932	890	890
q5	3203	3156	3157	3156
q6	246	128	128	128
q7	1068	497	493	493
q8	2287	2266	2255	2255
q9	6702	6689	6723	6689
q10	3208	3309	3293	3293
q11	338	210	205	205
q12	349	207	208	207
q13	4163	3437	3387	3387
q14	239	210	216	210
q15	569	531	531	531
q16	443	379	392	379
q17	1042	769	591	591
q18	7074	6897	6858	6858
q19	1777	1646	1628	1628
q20	567	325	297	297
q21	3195	2690	2680	2680
q22	368	306	297	297
Total cold run time: 44789 ms
Total hot run time: 40183 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4596	4568	4575	4568
q2	269	164	167	164
q3	3384	3398	3373	3373
q4	2232	2211	2200	2200
q5	5735	5736	5732	5732
q6	243	117	118	117
q7	2374	1822	1848	1822
q8	3604	3619	3616	3616
q9	8999	8982	8965	8965
q10	3822	3896	3905	3896
q11	491	366	380	366
q12	760	616	614	614
q13	3899	3187	3189	3187
q14	288	256	262	256
q15	580	516	529	516
q16	489	454	447	447
q17	1984	1958	1949	1949
q18	8681	8119	8172	8119
q19	1760	1768	1769	1768
q20	2240	1936	1946	1936
q21	6093	5744	5713	5713
q22	561	452	442	442
Total cold run time: 63084 ms
Total hot run time: 59766 ms

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit 36b45b5 into apache:master Dec 29, 2023
@xiaokang
Copy link
Contributor

@fornaix can you submit a PR to branch-2.0 ?

@fornaix
Copy link
Contributor Author

fornaix commented Jan 2, 2024

@fornaix can you submit a PR to branch-2.0 ?

@xiaokang sure, #29374

@xiaokang
Copy link
Contributor

xiaokang commented Jan 2, 2024

@fornaix can you submit a PR to branch-2.0 ?

@xiaokang sure, #29374

tks!

HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.
cambyzju pushed a commit to cambyzju/incubator-doris that referenced this pull request Mar 25, 2024
Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.
cambyzju added a commit that referenced this pull request Mar 29, 2024
* [fix](mysql) fix mysql channel infinite blocking (#28808)

Call the Channels blocking method with timeout instead.

Using session variables net_write_timeout and net_read_timeout as the timeout parameter.

* [conf](mysql) opt mysql network timeout to 600s #32545

---------

Co-authored-by: fornaix <foxnaix@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/1.2.9-merged dev/2.0.4-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] MySQL channel infinite blocking

6 participants