Skip to content

[fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
gavinchou merged 2 commits into from
Jun 25, 2025
Merged

[fix](cloud) Fix ConcurrentModificationException in cloud rebalance #52013

gavinchou merged 2 commits into from
Jun 25, 2025

Conversation

@deardeng
Copy link
Contributor

@deardeng deardeng commented Jun 20, 2025 

2025-06-19 15:58:44,191 ERROR (cloud tablet rebalancer|277) [Daemon.run():121] daemon thread got exception. name: cloud tablet rebalancer
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.checkInflghtWarmUpCacheAsync(CloudTabletRebalancer.java:421) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.runAfterCatalogReady(CloudTabletRebalancer.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@deardeng
[fix](cloud) Fix ConcurrentModificationException in cloud rebalance
afde75b 
```
2025-06-19 15:58:44,191 ERROR (cloud tablet rebalancer|277) [Daemon.run():121] daemon thread got exception. name: cloud tablet rebalancer
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.checkInflghtWarmUpCacheAsync(CloudTabletRebalancer.java:421) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.runAfterCatalogReady(CloudTabletRebalancer.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
```
@deardeng
Copy link
Contributor Author

deardeng commented Jun 20, 2025

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring added dev/3.0.x dev/3.1.x usercase Important user case type label labels Jun 20, 2025
@deardeng
Copy link
Contributor Author

deardeng commented Jun 22, 2025

run buildall

@deardeng
Copy link
Contributor Author

deardeng commented Jun 25, 2025

run feut

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 25, 2025

FE UT Coverage Report

Increment line coverage 0.00% (0/13) 🎉
Increment coverage report
Complete coverage report

dataroaring
dataroaring approved these changes Jun 25, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 25, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2025

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit 1d37391 into apache:master Jun 25, 2025
25 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 25, 2025
@deardeng
[fix](cloud) Fix ConcurrentModificationException in cloud rebalance (…
6beef18 
…#52013)

```
2025-06-19 15:58:44,191 ERROR (cloud tablet rebalancer|277) [Daemon.run():121] daemon thread got exception. name: cloud tablet rebalancer
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.checkInflghtWarmUpCacheAsync(CloudTabletRebalancer.java:421) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.runAfterCatalogReady(CloudTabletRebalancer.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
```
@github-actions github-actions bot mentioned this pull request Jun 25, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jun 25, 2025
@deardeng
[fix](cloud) Fix ConcurrentModificationException in cloud rebalance (…
99bcf97 
…#52013)

```
2025-06-19 15:58:44,191 ERROR (cloud tablet rebalancer|277) [Daemon.run():121] daemon thread got exception. name: cloud tablet rebalancer
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.checkInflghtWarmUpCacheAsync(CloudTabletRebalancer.java:421) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.runAfterCatalogReady(CloudTabletRebalancer.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
```
@github-actions github-actions bot mentioned this pull request Jun 25, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 26, 2025
@github-actions @deardeng
branch-3.0: [fix](cloud) Fix ConcurrentModificationException in clo…
1c02fdb 
…ud rebalance #52013 (#52309)

Cherry-picked from #52013

Co-authored-by: deardeng <dengxin@selectdb.com>
morrySnow pushed a commit that referenced this pull request Jun 26, 2025
@github-actions @deardeng
branch-3.1: [fix](cloud) Fix ConcurrentModificationException in clo…
512944b 
…ud rebalance #52013 (#52310)

Cherry-picked from #52013

Co-authored-by: deardeng <dengxin@selectdb.com>
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@github-actions @deardeng @koarz
branch-3.0: [fix](cloud) Fix ConcurrentModificationException in clo…
45cbd48 
…ud rebalance apache#52013 (apache#52309)

Cherry-picked from apache#52013

Co-authored-by: deardeng <dengxin@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Sep 1, 2025
@deardeng @gavinchou
[fix](cloud) Fix ConcurrentModificationException in cloud rebalance… (
ef697cc 
apache#4141)

… (apache#52013)

```
2025-06-19 15:58:44,191 ERROR (cloud tablet rebalancer|277) [Daemon.run():121] daemon thread got exception. name: cloud tablet rebalancer
java.util.ConcurrentModificationException: null
        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1597) ~[?:?]
        at java.util.HashMap$KeyIterator.next(HashMap.java:1620) ~[?:?]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.checkInflghtWarmUpCacheAsync(CloudTabletRebalancer.java:421) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.cloud.catalog.CloudTabletRebalancer.runAfterCatalogReady(CloudTabletRebalancer.java:257) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.MasterDaemon.runOneCycle(MasterDaemon.java:58) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.common.util.Daemon.run(Daemon.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. cloud dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@deardeng @hello-stephen @gavinchou @dataroaring @morrySnow