[refactor](catalog) set "use_meta_cache" default to true #38244
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40290 ms |
TPC-DS: Total hot run time: 174555 ms |
ClickBench: Total hot run time: 30.57 s |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 40184 ms |
TPC-DS: Total hot run time: 173428 ms |
ClickBench: Total hot run time: 30.68 s |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
dataroaring
pushed a commit
that referenced
this pull request
Jul 24, 2024
This is PR #33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jul 25, 2024
This is PR apache#33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
morningman
added a commit
that referenced
this pull request
Jul 25, 2024
Followup #38244 fix some unstable fe ut after set `use_meta_cache` as true
morningman
pushed a commit
that referenced
this pull request
Aug 1, 2024
Brought by: #38244 how to reproduce: create a hms catalog named hive with property use_meta_cache = false, after that restart fe, and run sql `switch hive; show databases;`, will get error： ```sql ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.metacache.MetaCache.listNames()" because "this.metaCache" is null ``` reason, property use_meta_cache of newly created hms catalog after pr #38244 was force convert to true, but field `useMetaCache` of class ExternalCatalog is not converted at the same time.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Aug 1, 2024
This is PR apache#33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Aug 1, 2024
…#38619) Brought by: apache#38244 how to reproduce: create a hms catalog named hive with property use_meta_cache = false, after that restart fe, and run sql `switch hive; show databases;`, will get error： ```sql ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.metacache.MetaCache.listNames()" because "this.metaCache" is null ``` reason, property use_meta_cache of newly created hms catalog after pr apache#38244 was force convert to true, but field `useMetaCache` of class ExternalCatalog is not converted at the same time.
morningman
added a commit
that referenced
this pull request
Aug 2, 2024
feiniaofeiafei
pushed a commit
to feiniaofeiafei/doris
that referenced
this pull request
Aug 9, 2024
…#38619) Brought by: apache#38244 how to reproduce: create a hms catalog named hive with property use_meta_cache = false, after that restart fe, and run sql `switch hive; show databases;`, will get error： ```sql ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.metacache.MetaCache.listNames()" because "this.metaCache" is null ``` reason, property use_meta_cache of newly created hms catalog after pr apache#38244 was force convert to true, but field `useMetaCache` of class ExternalCatalog is not converted at the same time.
dataroaring
pushed a commit
that referenced
this pull request
Aug 11, 2024
Followup #38244 fix some unstable fe ut after set `use_meta_cache` as true
dataroaring
pushed a commit
that referenced
this pull request
Aug 11, 2024
Brought by: #38244 how to reproduce: create a hms catalog named hive with property use_meta_cache = false, after that restart fe, and run sql `switch hive; show databases;`, will get error： ```sql ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.metacache.MetaCache.listNames()" because "this.metaCache" is null ``` reason, property use_meta_cache of newly created hms catalog after pr #38244 was force convert to true, but field `useMetaCache` of class ExternalCatalog is not converted at the same time.
dataroaring
pushed a commit
that referenced
this pull request
Aug 16, 2024
Followup #38244 fix some unstable fe ut after set `use_meta_cache` as true
dataroaring
pushed a commit
that referenced
this pull request
Aug 16, 2024
Brought by: #38244 how to reproduce: create a hms catalog named hive with property use_meta_cache = false, after that restart fe, and run sql `switch hive; show databases;`, will get error： ```sql ERROR 1105 (HY000): NullPointerException, msg: java.lang.NullPointerException: Cannot invoke "org.apache.doris.datasource.metacache.MetaCache.listNames()" because "this.metaCache" is null ``` reason, property use_meta_cache of newly created hms catalog after pr #38244 was force convert to true, but field `useMetaCache` of class ExternalCatalog is not converted at the same time.
morningman
pushed a commit
that referenced
this pull request
Aug 23, 2024
…using meta cache and add parameters to the Hive catalog. (#39239) before #15401 and #38244 ## Proposed changes 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Aug 23, 2024
…using meta cache and add parameters to the Hive catalog. (apache#39239) before apache#15401 and apache#38244 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
dataroaring
pushed a commit
that referenced
this pull request
Aug 26, 2024
…using meta cache and add parameters to the Hive catalog. (#39239) before #15401 and #38244 ## Proposed changes 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
morningman
added a commit
that referenced
this pull request
Oct 17, 2024
Followup #38244 When setting `use_meta_cache=true`, the `include_database_list` and `exclude_database_list` will have no effect. This PR fix it. Also fix a bug, related to #40479. if `include_database_list` and `exclude_database_list` is set, and `use` a database which is excluded, Doris should return "database not found" error
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 18, 2024
…#41939) Followup apache#38244 When setting `use_meta_cache=true`, the `include_database_list` and `exclude_database_list` will have no effect. This PR fix it. Also fix a bug, related to apache#40479. if `include_database_list` and `exclude_database_list` is set, and `use` a database which is excluded, Doris should return "database not found" error
morningman
added a commit
to morningman/doris
that referenced
this pull request
Oct 21, 2024
…#41939) Followup apache#38244 When setting `use_meta_cache=true`, the `include_database_list` and `exclude_database_list` will have no effect. This PR fix it. Also fix a bug, related to apache#40479. if `include_database_list` and `exclude_database_list` is set, and `use` a database which is excluded, Doris should return "database not found" error
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is PR #33610 introduce a new feature of
use_meta_cache=true.Now I will set this property default to true.
For all newly created catalog, is this property is not specified, it will be true.
For all previously created catalog, this property is still false.