[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

CalvinKirs · 2023-10-27T10:00:05Z

What Happend

mysql-load-fail-record-cleaner thread crazy increase

Proposed changes

MySqlLoad will create a new thread every time checkpoint is made, set it's only daemon thread can be create new thread

Test

Add a unified thread name setting method

CalvinKirs · 2023-10-30T04:43:34Z

run buildall

doris-robot · 2023-10-30T05:10:29Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.25 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162273247 Bytes

doris-robot · 2023-10-30T05:23:07Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162109524 Bytes

dataroaring

LGTM

github-actions · 2023-10-30T05:53:23Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-10-30T05:53:25Z

PR approved by anyone and no changes requested.

…t mysql load (apache#26031) Add a unified thread name setting method (cherry picked from commit eb2cbae)

…hread every time checkpoint is made #26031 (#26136)

…26164) 1. FE could not compile because below error. Intro by PR #25933 ``` [INFO] --- exec:3.1.0:java (doc) @ fe-core --- ... Failed to generate doc for ignoreRuntimeFilterIds ``` 2. fix UT bugs intro by below PRs > - #25951 > - #26031 3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR #25933 merged. So this PR revert them to fix FE UT > - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (#26145)" > This reverts commit 8d7abf6. > - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (#26130)" > This reverts commit 19122b5. > - Revert "[Performance](Nereids): optimize GroupExpressionMatching (#26084)" > This reverts commit 0d956e9.

…y time checkpoint mysql load (#26031) (#26139)

…t mysql load (apache#26031) Add a unified thread name setting method

…pache#26164) 1. FE could not compile because below error. Intro by PR apache#25933 ``` [INFO] --- exec:3.1.0:java (doc) @ fe-core --- ... Failed to generate doc for ignoreRuntimeFilterIds ``` 2. fix UT bugs intro by below PRs > - apache#25951 > - apache#26031 3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT > - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)" > This reverts commit 8d7abf6. > - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)" > This reverts commit 19122b5. > - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)" > This reverts commit 0d956e9.

…t mysql load (apache#26031) Add a unified thread name setting method

…pache#26164) 1. FE could not compile because below error. Intro by PR apache#25933 ``` [INFO] --- exec:3.1.0:java (doc) @ fe-core --- ... Failed to generate doc for ignoreRuntimeFilterIds ``` 2. fix UT bugs intro by below PRs > - apache#25951 > - apache#26031 3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT > - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)" > This reverts commit 8d7abf6. > - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)" > This reverts commit 19122b5. > - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)" > This reverts commit 0d956e9.

…y time checkpoint mysql load (apache#26031) (apache#26139)

…t mysql load (apache#26031) Add a unified thread name setting method

…pache#26164) 1. FE could not compile because below error. Intro by PR apache#25933 ``` [INFO] --- exec:3.1.0:java (doc) @ fe-core --- ... Failed to generate doc for ignoreRuntimeFilterIds ``` 2. fix UT bugs intro by below PRs > - apache#25951 > - apache#26031 3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT > - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)" > This reverts commit 8d7abf6. > - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)" > This reverts commit 19122b5. > - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)" > This reverts commit 0d956e9.

CalvinKirs changed the title ~~[Improve](thread)Add a unified thread name setting method~~ [Improve](thread)Fixed the MySqlLoad will create a new thread every time checkpoint is made Oct 30, 2023

CalvinKirs changed the title ~~[Improve](thread)Fixed the MySqlLoad will create a new thread every time checkpoint is made~~ [Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made Oct 30, 2023

CalvinKirs force-pushed the master-test-thread-increase branch from 588b1b0 to 719bd47 Compare October 30, 2023 04:34

Fix meaningless thread creation every time checkpoint mysql load

2ec8440

Add a unified thread name setting method

CalvinKirs force-pushed the master-test-thread-increase branch from 8a7f484 to 2ec8440 Compare October 30, 2023 04:43

dataroaring added the dev/2.0.3 label Oct 30, 2023

dataroaring approved these changes Oct 30, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 30, 2023

github-actions bot added the reviewed label Oct 30, 2023

bobhan1 approved these changes Oct 30, 2023

View reviewed changes

dataroaring merged commit eb2cbae into apache:master Oct 30, 2023

dataroaring added the usercase Important user case type label label Oct 30, 2023

wm1581066 added p0_c p0_b and removed p0_c labels Oct 30, 2023

CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Oct 31, 2023

[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…

e5b48e5

…t mysql load (apache#26031) Add a unified thread name setting method (cherry picked from commit eb2cbae)

CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Oct 31, 2023

[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…

4408526

…t mysql load (apache#26031) Add a unified thread name setting method (cherry picked from commit eb2cbae)

CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Oct 31, 2023

[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…

ee7a524

…t mysql load (apache#26031) Add a unified thread name setting method (cherry picked from commit eb2cbae)

xiaokang pushed a commit that referenced this pull request Oct 31, 2023

(pick-2.0.2.1)[Fix](MySqlLoad)Fixed the MySqlLoad will create a new t…

ab3de52

…hread every time checkpoint is made #26031 (#26136)

morrySnow mentioned this pull request Oct 31, 2023

[fix](compile) fe compile failed when generate doc and FE UT failed #26164

Merged

xiaokang pushed a commit that referenced this pull request Nov 1, 2023

(pick-2.0-26031)[Fix](MySqlLoad) Fix meaningless thread creation ever…

0242f87

…y time checkpoint mysql load (#26031) (#26139)

xiaokang mentioned this pull request Nov 1, 2023

(pick-2.0-26031)[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoint mysql load (#26031) #26139

Merged

xiaokang added dev/2.0.3-merged and removed dev/2.0.3 labels Nov 1, 2023

dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023

[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…

c2977cf

…t mysql load (apache#26031) Add a unified thread name setting method

seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023

[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…

18207c5

…t mysql load (apache#26031) Add a unified thread name setting method

gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023

(pick-2.0-26031)[Fix](MySqlLoad) Fix meaningless thread creation ever…

7b2e5b9

…y time checkpoint mysql load (apache#26031) (apache#26139)

xiaokang mentioned this pull request Dec 4, 2023

Release Note 2.0.3 #27909

Closed

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023

[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…

2994f13

…t mysql load (apache#26031) Add a unified thread name setting method

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

Uh oh!

CalvinKirs commented Oct 27, 2023 •

edited

Loading

Uh oh!

CalvinKirs commented Oct 30, 2023

Uh oh!

doris-robot commented Oct 30, 2023

Uh oh!

doris-robot commented Oct 30, 2023

Uh oh!

dataroaring left a comment

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

Uh oh!

Conversation

CalvinKirs commented Oct 27, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Happend

Proposed changes

Test

Uh oh!

CalvinKirs commented Oct 30, 2023

Uh oh!

doris-robot commented Oct 30, 2023

Uh oh!

doris-robot commented Oct 30, 2023

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

github-actions bot commented Oct 30, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

CalvinKirs commented Oct 27, 2023 •

edited

Loading