[feat](hive) add catalog level schema cache property #50958
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33754 ms |
TPC-DS: Total hot run time: 192839 ms |
ClickBench: Total hot run time: 29.24 s |
morningman
force-pushed
the
schema_cache_off
branch
from
18e7d8c to
fdb5aa7
Compare
morningman
changed the title
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 34170 ms |
TPC-DS: Total hot run time: 186719 ms |
ClickBench: Total hot run time: 29.61 s |
wuwenchi
approved these changes
May 19, 2025
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
github-actions bot
pushed a commit
that referenced
this pull request
May 19, 2025
### What problem does this PR solve? Problem Summary: Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
16 tasks
morningman
added a commit
that referenced
this pull request
May 21, 2025
…get_schema_from_table" mode (#51057) ### What problem does this PR solve? Related PR: #50958 Problem Summary: In #50958, I introduced the catalog level schema cache switch. But if use also add `"get_schema_from_table" = "true"`, the schema will be fetched from `Hive Table` instead of schema cache, so the schema cache config is broken. This PR fix it: 1. Always get new `Hive Table` instance to get schema from table. To make schema change config work. 2. For `show create table xxx`, it will always show the latest schema of a hive table.
morningman
added a commit
that referenced
this pull request
May 21, 2025
…get_schema_from_table" mode (#51057) Related PR: #50958 Problem Summary: In #50958, I introduced the catalog level schema cache switch. But if use also add `"get_schema_from_table" = "true"`, the schema will be fetched from `Hive Table` instead of schema cache, so the schema cache config is broken. This PR fix it: 1. Always get new `Hive Table` instance to get schema from table. To make schema change config work. 2. For `show create table xxx`, it will always show the latest schema of a hive table.
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
### What problem does this PR solve? Problem Summary: Just same as apache#50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
koarz
pushed a commit
to koarz/doris
that referenced
this pull request
Jun 4, 2025
…get_schema_from_table" mode (apache#51057) ### What problem does this PR solve? Related PR: apache#50958 Problem Summary: In apache#50958, I introduced the catalog level schema cache switch. But if use also add `"get_schema_from_table" = "true"`, the schema will be fetched from `Hive Table` instead of schema cache, so the schema cache config is broken. This PR fix it: 1. Always get new `Hive Table` instance to get schema from table. To make schema change config work. 2. For `show create table xxx`, it will always show the latest schema of a hive table.
dataroaring
pushed a commit
that referenced
this pull request
Jun 24, 2025
github-actions bot
pushed a commit
that referenced
this pull request
Jun 24, 2025
### What problem does this PR solve? Problem Summary: Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.
morningman
added a commit
that referenced
this pull request
Jun 24, 2025
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jun 24, 2025
…get_schema_from_table" mode (apache#51057) Related PR: apache#50958 Problem Summary: In apache#50958, I introduced the catalog level schema cache switch. But if use also add `"get_schema_from_table" = "true"`, the schema will be fetched from `Hive Table` instead of schema cache, so the schema cache config is broken. This PR fix it: 1. Always get new `Hive Table` instance to get schema from table. To make schema change config work. 2. For `show create table xxx`, it will always show the latest schema of a hive table.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Problem Summary:
Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs.
Previously, if user want to disable the schema cache, they can only set the
max_external_schema_cache_num=0in fe.conf and restart FE.And this config will effect all catalogs.
In this PR, I add a new catalog property
schema.cache.ttl-second.If set to 0, the schema cache will be disabled, so if schema is changed
Doris will read the new schema immediately.
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)