[feat](hive) add catalog level partition cache property #50724
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34618 ms |
TPC-DS: Total hot run time: 189623 ms |
ClickBench: Total hot run time: 28.79 s |
There was a problem hiding this comment.
Pull Request Overview
This PR adds support for configuring the hive partition cache at the catalog level via a new property ("partition.cache.ttl-second") and updates existing cache-related configurations. Key changes include:
- Removing a drop command in one of the Hive test suites.
- Introducing new tests and modifying existing ones for both file meta cache and partition cache.
- Updating cache initialization logic and configuration validations in the FE code.
Reviewed Changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/external_table_p0/hive/test_hive_star_qualifier.groovy | Commenting out a drop catalog command in the test suite. |
| regression-test/suites/external_table_p0/hive/test_hive_meta_cache.groovy | Adding new tests for invalid TTL values and proper caching behavior. |
| regression-test/suites/external_table_p0/export/hive_read/orc/test_hive_read_orc.groovy | Changing test suite tags to include additional external docker tags. |
| regression-test/data/external_table_p0/hive/test_hive_meta_cache.out | Updating generated expected output for the meta cache tests. |
| fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java | Changing method access modifiers and updating cache initialization logic. |
| fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java | Introducing new constants and validations for the partition cache TTL property. |
| fe/fe-common/src/main/java/org/apache/doris/common/Config.java | Updating configuration descriptions to include an additional table type. |
Comments suppressed due to low confidence (2)
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java:134
- The access level of init() has been changed from private to public. Please ensure that this change is intended and document its external usage if applicable.
public void init() {
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java:173
- The method setNewFileCache() has been changed from public to private; ensure that this does not affect any intended external calls or testing, and update the documentation if necessary.
private void setNewFileCache() {
wuwenchi
approved these changes
May 9, 2025
Contributor
Author
|
run buildall |
Contributor
|
PR approved by anyone and no changes requested. |
TPC-H: Total hot run time: 33648 ms |
TPC-DS: Total hot run time: 191154 ms |
ClickBench: Total hot run time: 29.15 s |
hubgeter
approved these changes
May 9, 2025
CalvinKirs
approved these changes
May 9, 2025
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions bot
pushed a commit
that referenced
this pull request
May 9, 2025
### What problem does this PR solve? Problem Summary: Support enable or disable hive partition cache at Catalog level for hive catalog. Previously, if user want to disable the hive partition cache, they can only set the `max_hive_partition_table_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `partition.cache.ttl-second`. If set to 0, the hive partition cache will be disabled, so if new partitioned is added, Doris will read the new partition immediately.
github-actions bot
pushed a commit
that referenced
this pull request
May 9, 2025
### What problem does this PR solve? Problem Summary: Support enable or disable hive partition cache at Catalog level for hive catalog. Previously, if user want to disable the hive partition cache, they can only set the `max_hive_partition_table_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `partition.cache.ttl-second`. If set to 0, the hive partition cache will be disabled, so if new partitioned is added, Doris will read the new partition immediately.
yiguolei
pushed a commit
that referenced
this pull request
May 9, 2025
Closed
dataroaring
pushed a commit
that referenced
this pull request
May 13, 2025
16 tasks
morningman
added a commit
that referenced
this pull request
May 19, 2025
### What problem does this PR solve? Problem Summary: Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
github-actions bot
pushed a commit
that referenced
this pull request
May 19, 2025
### What problem does this PR solve? Problem Summary: Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
### What problem does this PR solve? Problem Summary: Support enable or disable hive partition cache at Catalog level for hive catalog. Previously, if user want to disable the hive partition cache, they can only set the `max_hive_partition_table_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `partition.cache.ttl-second`. If set to 0, the hive partition cache will be disabled, so if new partitioned is added, Doris will read the new partition immediately.
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
### What problem does this PR solve? Problem Summary: Just same as apache#50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
Closed
github-actions bot
pushed a commit
that referenced
this pull request
Jun 24, 2025
### What problem does this PR solve? Problem Summary: Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
Support enable or disable hive partition cache at Catalog level for hive catalog.
Previously, if user want to disable the hive partition cache, they can only set the
max_hive_partition_table_cache_num=0in fe.conf and restart FE.And this config will effect all catalogs.
In this PR, I add a new catalog property
partition.cache.ttl-second.If set to 0, the hive partition cache will be disabled, so if new partitioned is added,
Doris will read the new partition immediately.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)