[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable #41659

hubgeter · 2024-10-10T08:33:26Z

Proposed changes

Because ExternalTable will initialize the previously uninitialized table when getCachedRowCount(), which is unnecessary. So for the uninitialized table, we directly return -1.
This will increase the speed of our query information_schema.tables.

doris-robot · 2024-10-10T08:33:31Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

hubgeter · 2024-10-10T08:33:37Z

run buildall

hubgeter · 2024-10-10T10:59:51Z

run buildall

hubgeter · 2024-10-10T14:18:26Z

run buildall

morningman · 2024-10-11T04:01:50Z

fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java

+        // ExternalTable.getRowCount(), but this is not very meaningful and time-consuming.
+        // The getCachedRowCount() function is only used when `show table` and querying `information_schema.tables`.
+        if (!checkInitialized()) {
+            return -2;


i will fix it

morningman · 2024-10-11T04:02:16Z

fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalTable.java

        return false;
    }

+    public boolean checkInitialized() {


No need this method, we can use check objectCreated

i will remove this method.

morningman · 2024-10-11T04:04:18Z

fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java

    private boolean enableHmsEventsIncrementalSync = false;

+    //for "type" = "hms" , but is iceberg table.
+    HiveCatalog icebergHiveCatalog;


Suggested change

HiveCatalog icebergHiveCatalog;

private HiveCatalog icebergHiveCatalog;

i will fix it

morningman · 2024-10-11T04:06:39Z

fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java

    }

+    public Catalog getIcebergHiveCatalog() {
+        if (icebergHiveCatalog == null) {


This method may be called concurrently

for prevent this, i will move icebergHiveCatalog init to initLocalObjectsImpl()

morningman · 2024-10-11T04:14:00Z

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergMetadataCache.java

-        HiveCatalog hiveCatalog = new HiveCatalog();
-        hiveCatalog.setConf(conf);
-
-        if (props.containsKey(HMSExternalCatalog.BIND_BROKER_NAME)) {


This part is missing?

I kept this part and removed the else part. As it seemed to me that the two were duplicates. The else part just initialize hivecatalog use uriproperty.

…ount when reading ExternalTable

hubgeter · 2024-10-11T06:42:15Z

run buildall

morningman

LGTM

github-actions · 2024-10-11T10:18:41Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-11T10:18:43Z

PR approved by anyone and no changes requested.

wuwenchi · 2024-10-11T10:33:51Z

fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergUtils.java

+        HiveCatalog hiveCatalog = new org.apache.iceberg.hive.HiveCatalog();
+        hiveCatalog.setConf(externalCatalog.getConfiguration());
+
+        Map<String, String> catalogProperties = externalCatalog.getProperties();
+        String metastoreUris = catalogProperties.getOrDefault(HMSProperties.HIVE_METASTORE_URIS, "");
+        catalogProperties.put(CatalogProperties.URI, metastoreUris);
+
+        hiveCatalog.initialize(name, catalogProperties);
+        return hiveCatalog;


Is same as IcebergHMSExternalCatalog.initCatalog ?

yes . so now IcebergHMSExternalCatalog.initCatalog() method use this method.

you can ref : https://github.com/apache/doris/pull/41659/files#diff-e6e9c0730301ce99354da7a68c322f1c94d656033b25ab525eab6f17eb9dce7dL41

…ount when reading ExternalTable (apache#41659) ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (apache#41659) Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (apache#41659) ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (#41659) (#41959) bp #41659 ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

…ount when reading ExternalTable (#41659) (#41962) bp #41659 ## Proposed changes Because ExternalTable will initialize the previously uninitialized table when `getCachedRowCount()`, which is unnecessary. So for the uninitialized table, we directly return -1. This will increase the speed of our query `information_schema.tables`.

hubgeter force-pushed the opt_infomation_schema_tables branch from 90d6893 to c3535c9 Compare October 10, 2024 14:18

morningman added dev/2.1.x dev/3.0.x labels Oct 11, 2024

morningman reviewed Oct 11, 2024

View reviewed changes

hubgeter force-pushed the opt_infomation_schema_tables branch from c3535c9 to 3d68a83 Compare October 11, 2024 06:40

hubgeter added 3 commits October 11, 2024 14:42

[Enhancement](ExternalTable)Optimize the performance of getCachedRowC…

089629d

…ount when reading ExternalTable

fix case

d92304c

fix comment

7526b02

hubgeter force-pushed the opt_infomation_schema_tables branch from 3d68a83 to 7526b02 Compare October 11, 2024 06:42

morningman approved these changes Oct 11, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 11, 2024

wuwenchi reviewed Oct 11, 2024

View reviewed changes

wuwenchi approved these changes Oct 11, 2024

View reviewed changes

morningman merged commit 127ac8d into apache:master Oct 13, 2024

hubgeter mentioned this pull request Oct 16, 2024

[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable (#41659) #41959

Merged

hubgeter mentioned this pull request Oct 16, 2024

[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable (#41659) #41962

Merged

yiguolei added dev/2.1.7-merged and removed dev/2.1.x labels Oct 17, 2024

morningman added dev/3.0.3-merged and removed dev/3.0.x labels Oct 18, 2024

gavinchou mentioned this pull request Nov 26, 2024

Release Note 3.0.3 #44522

Closed

	HiveCatalog icebergHiveCatalog;
	private HiveCatalog icebergHiveCatalog;

[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable #41659

[Enhancement](ExternalTable)Optimize the performance of getCachedRowCount when reading ExternalTable #41659

Uh oh!

Conversation

hubgeter commented Oct 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Oct 10, 2024

Uh oh!

hubgeter commented Oct 10, 2024

Uh oh!

hubgeter commented Oct 10, 2024

Uh oh!

hubgeter commented Oct 10, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubgeter Oct 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Oct 11, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

hubgeter commented Oct 10, 2024 •

edited

Loading

hubgeter Oct 11, 2024 •

edited

Loading