[feature-wip](multi-catalog) support automatic sync hive metastore events #15401
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
TeamCity pipeline, clickbench performance test result: |
added 13 commits
December 28, 2022 12:07
morningman
reviewed
Jan 2, 2023
fe/fe-core/src/main/java/org/apache/doris/datasource/HMSExternalCatalog.java
Outdated
Show resolved
Hide resolved
| LOG.info("Event id not updated when pulling events on catalog [{}]", hmsExternalCatalog.getName()); | ||
| return null; | ||
| } | ||
| return client.getNextNotification(lastSyncedEventId, Config.hms_events_batch_size_per_rpc, null); |
Contributor
There was a problem hiding this comment.
should we use currentEventId here instead of lastSyncedEventId?
Contributor
Author
There was a problem hiding this comment.
The currentEventId represents the latest event ID. If we use the currentEventId to get events, we will never get events
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventFactory.java
Outdated
Show resolved
Hide resolved
zddr
changed the title
zddr
changed the title
morningman
changed the title
morningman
approved these changes
Jan 3, 2023
| LOG.info("Event id not updated when pulling events on catalog [{}]", hmsExternalCatalog.getName()); | ||
| return null; | ||
| } | ||
| return client.getNextNotification(lastSyncedEventId, Config.hms_events_batch_size_per_rpc, null); |
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
I found that just |
morningman
changed the title
Contributor
This is a WIP pr, not finished |
eldenmoon
pushed a commit
to eldenmoon/incubator-doris
that referenced
this pull request
Jan 5, 2023
…ents (apache#15401) Poll metastore for create/alter/drop operations on database, table, partition events at a given frequency. By observing such events, we can take appropriate action on the (refresh/invalidate/add/remove) so that represents the latest information available in metastore. We keep track of the last synced event id in each polling iteration so the next batch can be requested appropriately.
morningman
pushed a commit
that referenced
this pull request
Jan 16, 2023
…ents (#15401) Poll metastore for create/alter/drop operations on database, table, partition events at a given frequency. By observing such events, we can take appropriate action on the (refresh/invalidate/add/remove) so that represents the latest information available in metastore. We keep track of the last synced event id in each polling iteration so the next batch can be requested appropriately.
morningman
pushed a commit
that referenced
this pull request
Aug 23, 2024
…using meta cache and add parameters to the Hive catalog. (#39239) before #15401 and #38244 ## Proposed changes 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
morningman
pushed a commit
to morningman/doris
that referenced
this pull request
Aug 23, 2024
…using meta cache and add parameters to the Hive catalog. (apache#39239) before apache#15401 and apache#38244 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
dataroaring
pushed a commit
that referenced
this pull request
Aug 26, 2024
…using meta cache and add parameters to the Hive catalog. (#39239) before #15401 and #38244 ## Proposed changes 1. Add the parameter `hive.enable_hms_events_incremental_sync` to the hive catalog, which is used to switch the catalog to read hive notification events (default is false). The default value is `enable_hms_events_incremental_sync` in fe.conf 2. Add the parameter `hive.hms_events_batch_size_per_rpc` to the hive catalog, which is used to set the size of notification events read by the catalog each time. The default value is `hms_events_batch_size_per_rpc` in fe.conf (default is 500) 3. append hms event notification case . 4. Remove the `use_meta_cache` setting in catalog that is forced to true. Example : ``` create catalog if not exists catalog_name properties ( "type"="hms", 'hive.metastore.uris' = 'thrift://externalEnvIp:hms_port', "hive.enable_hms_events_incremental_sync" ="true", "hive.hms_events_batch_size_per_rpc" = "1000" ); ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Issue Number: close #xxx
Problem summary
Poll metastore for create/alter/drop operations on database, table, partition events at a given frequency. By observing such events, we can take appropriate action on the (refresh/invalidate/add/remove) so that represents the latest information
available in metastore. We keep track of the last synced event id in each polling
iteration so the next batch can be requested appropriately.
Checklist(Required)
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...