[opt](catalog) support using loading cache for db/table list in external catalog #33610
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 38961 ms |
TPC-DS: Total hot run time: 188477 ms |
ClickBench: Total hot run time: 30.46 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 38357 ms |
TPC-DS: Total hot run time: 186087 ms |
ClickBench: Total hot run time: 31.26 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 39540 ms |
TPC-DS: Total hot run time: 183980 ms |
ClickBench: Total hot run time: 30.44 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 38547 ms |
TPC-DS: Total hot run time: 185592 ms |
ClickBench: Total hot run time: 30.92 s |
|
Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' |
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 38393 ms |
TPC-DS: Total hot run time: 185352 ms |
morningman
added a commit
to morningman/doris
that referenced
this pull request
May 9, 2024
…nal catalog (apache#33610) 1. **Master FE** uniformly retrieves table information and generates the corresponding `id -> name` mapping. 2. The `id -> name` mapping is stored in Doris's metadata and persisted. 3. **Master FE** synchronizes this information with other FEs via **EditLog**. 4. To update the table information, a `refresh` command must be executed or the metadata synchronized through an **HMS event**. - **Advantage**: All FEs can see a consistent list of tables as the information is uniformly obtained from the Master FE, preventing discrepancies in table visibility across different FEs. - **Disadvantage**: There is an inability to promptly perceive changes in the tables. For example, a new table on the Hive side is not immediately visible on the Doris side and requires a refresh or periodic metadata refresh for visibility. - **Catalog** adds a new property `use_meta_cache`. Default is `false`. If set to `true`, it will use an independent caching method to synchronize table information. - Once enabled, table information will no longer be uniformly obtained by Master FE but will instead be independently fetched by each FE. - Each FE has its own cache of the Database and Table list, implemented using the **Caffeine library**. - This cache synchronously loads table information when accessed. If the cache does not exist, it will directly access HMS for table information. - **Behaviors**: - Different FEs may see different table information due to different loading times, but they will eventually be consistent. - New tables created on the Hive side can be queried directly in Doris, but may not be visible in `show databases` or `show tables`. - Tables deleted on the Hive side will still appear in `show databases/tables` but will be inaccessible. - All caches will refresh at most every 10 minutes. - **Compatibility**: - For already created catalog, after upgrade, the `use_meta_cache` is `false`. - For newly created catalog, if `use_meta_cache` is not set, set it as `false`. - Can not modify `use_meta_cache` after being created. - **MetaCache**: - A general Cache class responsible for caching Database/Table information, including two LoadingCaches for storing "name lists" and "name-to-object" caches. - **ID Generation Rules**: - As table information is no longer uniformly fetched by the Master FE, a consistent rule must exist to ensure that each FE generates the same ID for the same table. Here, we use the absolute value of the top 8 bits of the sha256 hash of the table name as the object's ID. This way, the same name generates the same ID, but the ID is no longer globally unique and is unique only within the Catalog or Database level. - Remove some unused methods such as `getIdToTable()` and `getIdToDb()` I have run the p0 for both `use_meta_cache = true` and `use_meta_cache=false`
morningman
added a commit
to morningman/doris
that referenced
this pull request
May 9, 2024
…4517) Followup apache#33610 Refactor the logic of schema cache for external table. Before, the ExternalSchemaCache only caches the schema(which is `List<Column>`) of the table. But actually, some other objects are also being refreshed when schema cache refreshing, such as `partition columns`, etc. So This PR mainly changes: 1. Add a new class `SchemaCacheValue` as a parent class for all kinds of cache value in ExternalSchemaCache 2. It contains `List<Column>` basically, and sub class can extend from it to add more cached objects. 3. Move all cached objects into `SchemaCacheValue`, so that they will be refreshed along with schema together. Additional: 1. Remove unused warn log in AcceptListener.java 2. Fix unclear error msg if trino connector's table does not exist.
morningman
added a commit
to morningman/doris
that referenced
this pull request
May 9, 2024
…4517) Followup apache#33610 Refactor the logic of schema cache for external table. Before, the ExternalSchemaCache only caches the schema(which is `List<Column>`) of the table. But actually, some other objects are also being refreshed when schema cache refreshing, such as `partition columns`, etc. So This PR mainly changes: 1. Add a new class `SchemaCacheValue` as a parent class for all kinds of cache value in ExternalSchemaCache 2. It contains `List<Column>` basically, and sub class can extend from it to add more cached objects. 3. Move all cached objects into `SchemaCacheValue`, so that they will be refreshed along with schema together. Additional: 1. Remove unused warn log in AcceptListener.java 2. Fix unclear error msg if trino connector's table does not exist.
ByteYue
pushed a commit
to ByteYue/doris
that referenced
this pull request
May 15, 2024
…nal catalog (apache#33610) ## Proposed changes ### Previous Implementation Logic in Doris with External Catalog #### Steps 1. **Master FE** uniformly retrieves table information and generates the corresponding `id -> name` mapping. 2. The `id -> name` mapping is stored in Doris's metadata and persisted. 3. **Master FE** synchronizes this information with other FEs via **EditLog**. 4. To update the table information, a `refresh` command must be executed or the metadata synchronized through an **HMS event**. #### Advantages and Disadvantages - **Advantage**: All FEs can see a consistent list of tables as the information is uniformly obtained from the Master FE, preventing discrepancies in table visibility across different FEs. - **Disadvantage**: There is an inability to promptly perceive changes in the tables. For example, a new table on the Hive side is not immediately visible on the Doris side and requires a refresh or periodic metadata refresh for visibility. ### Modifications in this PR #### 1. User Interface - **Catalog** adds a new property `use_meta_cache`. Default is `false`. If set to `true`, it will use an independent caching method to synchronize table information. #### 2. Enabling `use_meta_cache` - Once enabled, table information will no longer be uniformly obtained by Master FE but will instead be independently fetched by each FE. - Each FE has its own cache of the Database and Table list, implemented using the **Caffeine library**. - This cache synchronously loads table information when accessed. If the cache does not exist, it will directly access HMS for table information. - **Behaviors**: - Different FEs may see different table information due to different loading times, but they will eventually be consistent. - New tables created on the Hive side can be queried directly in Doris, but may not be visible in `show databases` or `show tables`. - Tables deleted on the Hive side will still appear in `show databases/tables` but will be inaccessible. - All caches will refresh at most every 10 minutes. - **Compatibility**: - For already created catalog, after upgrade, the `use_meta_cache` is `false`. - For newly created catalog, if `use_meta_cache` is not set, set it as `false`. - Can not modify `use_meta_cache` after being created. #### 3. Code Implementation - **MetaCache**: - A general Cache class responsible for caching Database/Table information, including two LoadingCaches for storing "name lists" and "name-to-object" caches. - **ID Generation Rules**: - As table information is no longer uniformly fetched by the Master FE, a consistent rule must exist to ensure that each FE generates the same ID for the same table. Here, we use the absolute value of the top 8 bits of the sha256 hash of the table name as the object's ID. This way, the same name generates the same ID, but the ID is no longer globally unique and is unique only within the Catalog or Database level. - Remove some unused methods such as `getIdToTable()` and `getIdToDb()` ## Further comments I have run the p0 for both `use_meta_cache = true` and `use_meta_cache=false`
ByteYue
pushed a commit
to ByteYue/doris
that referenced
this pull request
May 15, 2024
…4517) Followup apache#33610 Refactor the logic of schema cache for external table. Before, the ExternalSchemaCache only caches the schema(which is `List<Column>`) of the table. But actually, some other objects are also being refreshed when schema cache refreshing, such as `partition columns`, etc. So This PR mainly changes: 1. Add a new class `SchemaCacheValue` as a parent class for all kinds of cache value in ExternalSchemaCache 2. It contains `List<Column>` basically, and sub class can extend from it to add more cached objects. 3. Move all cached objects into `SchemaCacheValue`, so that they will be refreshed along with schema together. Additional: 1. Remove unused warn log in AcceptListener.java 2. Fix unclear error msg if trino connector's table does not exist.
morningman
added a commit
that referenced
this pull request
May 30, 2024
```
Caused by: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:1.8.0_301]
at org.apache.doris.catalog.ResourceMgr.getResource(ResourceMgr.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.catalogResource(CatalogProperty.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.getOrDefault(CatalogProperty.java:77) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.setDefaultPropsIfMissing(ExternalCatalog.java:173) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.hive.HMSExternalCatalog.setDefaultPropsIfMissing(HMSExternalCatalog.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.gsonPostProcess(ExternalCatalog.java:687) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT]
at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?]
at org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:289) ~[doris-fe.jar:1.2-SNAPSHOT]
```
Introduced from #33610.
When read meta image, the `resource` maybe null, we should ignore it.
morningman
added a commit
to morningman/doris
that referenced
this pull request
May 30, 2024
```
Caused by: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:1.8.0_301]
at org.apache.doris.catalog.ResourceMgr.getResource(ResourceMgr.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.catalogResource(CatalogProperty.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.getOrDefault(CatalogProperty.java:77) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.setDefaultPropsIfMissing(ExternalCatalog.java:173) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.hive.HMSExternalCatalog.setDefaultPropsIfMissing(HMSExternalCatalog.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.gsonPostProcess(ExternalCatalog.java:687) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT]
at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?]
at org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:289) ~[doris-fe.jar:1.2-SNAPSHOT]
```
Introduced from apache#33610.
When read meta image, the `resource` maybe null, we should ignore it.
yiguolei
pushed a commit
that referenced
this pull request
May 30, 2024
```
Caused by: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:1.8.0_301]
at org.apache.doris.catalog.ResourceMgr.getResource(ResourceMgr.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.catalogResource(CatalogProperty.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.getOrDefault(CatalogProperty.java:77) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.setDefaultPropsIfMissing(ExternalCatalog.java:173) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.hive.HMSExternalCatalog.setDefaultPropsIfMissing(HMSExternalCatalog.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.gsonPostProcess(ExternalCatalog.java:687) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT]
at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?]
at org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:289) ~[doris-fe.jar:1.2-SNAPSHOT]
```
Introduced from #33610.
When read meta image, the `resource` maybe null, we should ignore it.
dataroaring
pushed a commit
that referenced
this pull request
May 31, 2024
```
Caused by: java.lang.NullPointerException
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:1.8.0_301]
at org.apache.doris.catalog.ResourceMgr.getResource(ResourceMgr.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.catalogResource(CatalogProperty.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.CatalogProperty.getOrDefault(CatalogProperty.java:77) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.setDefaultPropsIfMissing(ExternalCatalog.java:173) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.hive.HMSExternalCatalog.setDefaultPropsIfMissing(HMSExternalCatalog.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.datasource.ExternalCatalog.gsonPostProcess(ExternalCatalog.java:687) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT]
at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?]
at org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:289) ~[doris-fe.jar:1.2-SNAPSHOT]
```
Introduced from #33610.
When read meta image, the `resource` maybe null, we should ignore it.
morningman
added a commit
that referenced
this pull request
Jun 19, 2024
This is PR #33610 introduce a new feature of `use_meta_cache=true`. And made a wrong check when checking this config. And if user enable the hive metastore even listener for hive catalog, it may causing FE unable to restart due to meta data replay error: ``` 2024-06-19 14:25:32,536 ERROR (stateListener|118) [EditLog.loadJournal():1231] Operation Type 325 java.util.NoSuchElementException: No value present at java.util.Optional.get(Optional.java:135) ~[?:1.8.0_341] at org.apache.doris.datasource.ExternalCatalog.replayInitCatalog(ExternalCatalog.java:594) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.CatalogMgr.replayInitCatalog(CatalogMgr.java:584) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:1012) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayJournal(Env.java:2779) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.transferToMaster(Env.java:1473) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.access$1400(Env.java:324) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2670) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] ``` TODO: add hive event listener test suit in external p0
dataroaring
pushed a commit
that referenced
this pull request
Jun 21, 2024
This is PR #33610 introduce a new feature of `use_meta_cache=true`. And made a wrong check when checking this config. And if user enable the hive metastore even listener for hive catalog, it may causing FE unable to restart due to meta data replay error: ``` 2024-06-19 14:25:32,536 ERROR (stateListener|118) [EditLog.loadJournal():1231] Operation Type 325 java.util.NoSuchElementException: No value present at java.util.Optional.get(Optional.java:135) ~[?:1.8.0_341] at org.apache.doris.datasource.ExternalCatalog.replayInitCatalog(ExternalCatalog.java:594) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.datasource.CatalogMgr.replayInitCatalog(CatalogMgr.java:584) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.persist.EditLog.loadJournal(EditLog.java:1012) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.replayJournal(Env.java:2779) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.transferToMaster(Env.java:1473) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env.access$1400(Env.java:324) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.catalog.Env$5.runOneCycle(Env.java:2670) ~[doris-fe.jar:1.2-SNAPSHOT] at org.apache.doris.common.util.Daemon.run(Daemon.java:116) ~[doris-fe.jar:1.2-SNAPSHOT] ``` TODO: add hive event listener test suit in external p0
morningman
added a commit
that referenced
this pull request
Jul 15, 2024
1. Redundant hashCode of FunctionCallExpr.java
This line is redundant, and will cause the amount of hashcode
calculation to grow exponentially.
2. A potential deadlock of external catalog
The following case may causing this deadlock:
- high frequency load.
- querying external table join inner table on non-master FE.
- refresh external catalog frequently.
3. A workaround to avoid FE restart failure because db does not found
introduced from #33610 and fixed in #36530.
This PR is to make previous metadata restart successfully.
seawinde
pushed a commit
to seawinde/doris
that referenced
this pull request
Jul 17, 2024
1. Redundant hashCode of FunctionCallExpr.java
This line is redundant, and will cause the amount of hashcode
calculation to grow exponentially.
2. A potential deadlock of external catalog
The following case may causing this deadlock:
- high frequency load.
- querying external table join inner table on non-master FE.
- refresh external catalog frequently.
3. A workaround to avoid FE restart failure because db does not found
introduced from apache#33610 and fixed in apache#36530.
This PR is to make previous metadata restart successfully.
dataroaring
pushed a commit
that referenced
this pull request
Jul 17, 2024
1. Redundant hashCode of FunctionCallExpr.java
This line is redundant, and will cause the amount of hashcode
calculation to grow exponentially.
2. A potential deadlock of external catalog
The following case may causing this deadlock:
- high frequency load.
- querying external table join inner table on non-master FE.
- refresh external catalog frequently.
3. A workaround to avoid FE restart failure because db does not found
introduced from #33610 and fixed in #36530.
This PR is to make previous metadata restart successfully.
morningman
added a commit
that referenced
this pull request
Jul 24, 2024
This is PR #33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
dataroaring
pushed a commit
that referenced
this pull request
Jul 24, 2024
This is PR #33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Jul 25, 2024
This is PR apache#33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
morningman
added a commit
to morningman/doris
that referenced
this pull request
Aug 1, 2024
This is PR apache#33610 introduce a new feature of `use_meta_cache=true`. Now I will set this property default to true. For all newly created catalog, is this property is not specified, it will be true. For all previously created catalog, this property is still false.
Contributor
|
public static long genIdByName(String... names) {
return Math.abs(sha256long(String.join(".", names)));
}The name is a combination of catalog and database. Then, the ID is generated by the name. So, the |
Contributor
|
How to ensure the problem that the different FEs may see different table information due to different loading times, but they will eventually be consistent? @morningman |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
Previous Implementation Logic in Doris with External Catalog
Steps
id -> namemapping.id -> namemapping is stored in Doris's metadata and persisted.refreshcommand must be executed or the metadata synchronized through an HMS event.Advantages and Disadvantages
Modifications in this PR
1. User Interface
use_meta_cache. Default isfalse. If set totrue, it will use an independent caching method to synchronize table information.2. Enabling
use_meta_cacheOnce enabled, table information will no longer be uniformly obtained by Master FE but will instead be independently fetched by each FE.
Each FE has its own cache of the Database and Table list, implemented using the Caffeine library.
This cache synchronously loads table information when accessed. If the cache does not exist, it will directly access HMS for table information.
Behaviors:
show databasesorshow tables.show databases/tablesbut will be inaccessible.Compatibility:
use_meta_cacheisfalse.use_meta_cacheis not set, set it asfalse.use_meta_cacheafter being created.3. Code Implementation
MetaCache:
ID Generation Rules:
Remove some unused methods such as
getIdToTable()andgetIdToDb()Further comments
I have run the p0 for both
use_meta_cache = trueanduse_meta_cache=false