[Fix](Catalog) Close system resources when dropping catalog #49621
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
TPC-H: Total hot run time: 34055 ms |
TPC-DS: Total hot run time: 182811 ms |
ClickBench: Total hot run time: 30.81 s |
### Description: This PR ensures that system resources, such as thread pools, are properly closed when a catalog is dropped. Previously, these resources were not explicitly released, which could lead to potential resource leaks. ### Changes: Implemented close() method in the catalog to release system resources. Ensured that thread pools and other managed resources are properly shut down when dropping a catalog. Added necessary cleanup logic in the dropCatalog method.
CalvinKirs
force-pushed
the
master-ctl-close
branch
from
6a1fd71 to
4ad16db
Compare
morningman
previously approved these changes
Mar 28, 2025
Member
Author
|
run buildall |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 34068 ms |
TPC-DS: Total hot run time: 192466 ms |
ClickBench: Total hot run time: 31.44 s |
github-actions
bot
removed
the
approved
Member
Author
|
run buildall |
TPC-H: Total hot run time: 33615 ms |
zy-kkk
previously approved these changes
Mar 31, 2025
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
TPC-DS: Total hot run time: 191834 ms |
ClickBench: Total hot run time: 31.4 s |
There was a problem hiding this comment.
Pull Request Overview
This PR introduces changes to ensure system resources are properly released when a catalog is dropped. Key changes include renaming and refactoring methods (from onRefresh to resetToUninitialized) for catalog reset and closure, and adding onClose implementations in several catalog-related classes to improve resource management.
Reviewed Changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/AccessControllerManager.java | Added blank string check for removal of access controllers |
| fe/fe-core/src/main/java/org/apache/doris/datasource/jdbc/JdbcExternalCatalog.java | Renamed onRefresh to resetToUninitialized for cleanup consistency |
| fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java | Updated close() method now only nullifies the catalog reference |
| fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java | Introduced onClose to nullify the catalog reference after calling super |
| fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java | Updated resetToUninitialized and added onClose to close executors and ops |
| fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java | Refactored resetToUninitialized with added javadoc and onClose enhancements |
| fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java | Updated to call resetToUninitialized instead of onRefresh |
| fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogIf.java | Changed onRefresh call to resetToUninitialized for consistency |
Comments suppressed due to low confidence (1)
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalog.java:104
- Like in IcebergMetadataOps, simply setting the catalog to null in the onClose() method may not release underlying resources. Consider invoking a dedicated close or cleanup method on the catalog if available.
if (null != catalog) {
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataOps.java
Show resolved
Hide resolved
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
hubgeter
approved these changes
Mar 31, 2025
wuwenchi
approved these changes
Mar 31, 2025
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Apr 10, 2025
apache#49621) apache#49621 (cherry picked from commit 902b95b)
CalvinKirs
added a commit
to CalvinKirs/incubator-doris
that referenced
this pull request
Apr 10, 2025
…og (apache#49621) (apache#49621) (cherry picked from commit 902b95b)
yiguolei
pushed a commit
that referenced
this pull request
Apr 10, 2025
dataroaring
pushed a commit
that referenced
this pull request
Apr 24, 2025
Closed
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…9621) ### Description: This PR ensures that system resources, such as thread pools, are properly closed when a catalog is dropped. Previously, these resources were not explicitly released, which could lead to potential resource leaks. ### Changes: Implemented close() method in the catalog to release system resources. Ensured that thread pools and other managed resources are properly shut down when dropping a catalog. Added necessary cleanup logic in the dropCatalog method.
morningman
pushed a commit
that referenced
this pull request
Nov 5, 2025
since #49621 Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.
github-actions bot
pushed a commit
that referenced
this pull request
Nov 5, 2025
since #49621 Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.
github-actions bot
pushed a commit
that referenced
this pull request
Nov 5, 2025
since #49621 Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.
wyxxxcat
pushed a commit
to wyxxxcat/doris
that referenced
this pull request
Nov 18, 2025
…57680) since apache#49621 Since ExternalCatalog no longer calls methods in the interface when onClose, we need to manually clean up the values in refreshmanager when deleting the catalog.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
9 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This PR ensures that system resources, such as thread pools, are properly closed when a catalog is dropped. Previously, these resources were not explicitly released, which could lead to potential resource leaks.
Changes:
Implemented close() method in the catalog to release system resources.
Ensured that thread pools and other managed resources are properly shut down when dropping a catalog.
Added necessary cleanup logic in the dropCatalog method.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)