[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #22702

hubgeter · 2023-08-08T02:02:33Z

Proposed changes

add information_schema.metadata_name_idsfor quickly get catlogs,db,table.

table struct :

mysql> desc  internal.information_schema.metadata_name_ids;
+---------------+--------------+------+-------+---------+-------+
| Field         | Type         | Null | Key   | Default | Extra |
+---------------+--------------+------+-------+---------+-------+
| CATALOG_ID    | BIGINT       | Yes  | false | NULL    |       |
| CATALOG_NAME  | VARCHAR(512) | Yes  | false | NULL    |       |
| DATABASE_ID   | BIGINT       | Yes  | false | NULL    |       |
| DATABASE_NAME | VARCHAR(64)  | Yes  | false | NULL    |       |
| TABLE_ID      | BIGINT       | Yes  | false | NULL    |       |
| TABLE_NAME    | VARCHAR(64)  | Yes  | false | NULL    |       |
+---------------+--------------+------+-------+---------+-------+
6 rows in set (0.00 sec) 


mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G;
*************************** 1. row ***************************
   CATALOG_ID: 113008
 CATALOG_NAME: hive1
  DATABASE_ID: 113042
DATABASE_NAME: ssb1_parquet
     TABLE_ID: 114009
   TABLE_NAME: dates
1 row in set (0.07 sec)

when you create / drop catalog , need not refresh catalog .

mysql> select count(*) from internal.information_schema.metadata_name_ids\G; 
*************************** 1. row ***************************
count(*): 21301
1 row in set (0.34 sec)


mysql> drop catalog hive2;
Query OK, 0 rows affected (0.01 sec)

mysql> select count(*) from internal.information_schema.metadata_name_ids\G; 
*************************** 1. row ***************************
count(*): 10665
1 row in set (0.04 sec) 


mysql> create catalog hive3 ... 
mysql> select count(*) from internal.information_schema.metadata_name_ids\G;                                                                        
*************************** 1. row ***************************
count(*): 21301
1 row in set (0.32 sec)

create / drop table , need not refresh catalog .

mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ;


mysql> select count(*) from internal.information_schema.metadata_name_ids\G; 
*************************** 1. row ***************************
count(*): 10666
1 row in set (0.04 sec)

mysql> drop table demo.example_tbl;
Query OK, 0 rows affected (0.01 sec)

mysql> select count(*) from internal.information_schema.metadata_name_ids\G; 
*************************** 1. row ***************************
count(*): 10665
1 row in set (0.04 sec)

you can set query time , prevent queries from taking too long .


fe.conf :  query_metadata_name_ids_timeout 

the time used to obtain all tables in one database

add information_schema.profiling in order to Compatible with mysql

mysql> select * from information_schema.profiling;
Empty set (0.07 sec)

mysql> set profiling=1;                                                                                 
Query OK, 0 rows affected (0.01 sec)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions · 2023-08-08T02:09:16Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-08T04:11:39Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

please add regression test cases

morningman · 2023-08-09T15:56:12Z

fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java


+    //query  internal.information_schema.simple_tables time
+    @VariableMgr.VarAttr(name = QUERY_SIMPLE_TABLES_TIMEOUT)
+    public int querySimpleTablesTimeout = 3;


I think it can be a FE config, not session variable.
Because we always set it globally

ok good idea

morningman · 2023-08-09T15:57:28Z

gensrc/thrift/Descriptors.thrift

    SCH_COLUMN_STATISTICS,
-    SCH_PARAMETERS;
+    SCH_PARAMETERS,
+	SCH_SIMPLE_TABLES,


Suggested change

SCH_SIMPLE_TABLES,

SCH_SIMPLE_TABLES,

morningman · 2023-08-09T15:57:38Z

gensrc/thrift/Descriptors.thrift

-    SCH_PARAMETERS;
+    SCH_PARAMETERS,
+	SCH_SIMPLE_TABLES,
+	SCH_PROFILING;


Suggested change

SCH_PROFILING;

SCH_PROFILING;

morningman · 2023-08-09T15:58:11Z

gensrc/thrift/FrontendService.thrift

 }

+struct TSimpleTableStatus {
+	1: required string name


use 4 spaces instead of tab.
and use optional instead of required

ok good idea

morningman · 2023-08-09T15:58:16Z

gensrc/thrift/FrontendService.thrift

+	2: required i64 id 
+}
+struct TListSimpleTableStatusResult {
+	1: required list<TSimpleTableStatus> tables 


morningman · 2023-08-09T16:00:10Z

fe/fe-core/src/main/java/org/apache/doris/catalog/SchemaTable.java

+                            .column("ROUTINE_TYPE", ScalarType.createVarchar(64))
+                            .column("DATA_TYPEDTD_IDENDS", ScalarType.createVarchar(64))
+                            .build()))
+            .put("simple_tables", new SchemaTable(SystemIdGenerator.getNextId(), "simple_tables", TableType.SCHEMA,


how about just name it as metadata_name_ids?

ok good idea

github-actions · 2023-08-11T05:53:27Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-08-11T08:06:51Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-08-18T07:11:21Z

run buildall

github-actions · 2023-08-18T07:13:26Z

clang-tidy review says "All clean, LGTM! 👍"

hubgeter · 2023-08-18T11:31:57Z

run buildall

github-actions · 2023-08-18T11:37:51Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-18T12:27:53Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.7 seconds
stream load tsv: 535 seconds loaded 74807831229 Bytes, about 133 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.3 seconds inserted 10000000 Rows, about 341K ops/s
storage size: 17162394725 Bytes

morningman

LGTM

github-actions · 2023-08-20T14:26:21Z

PR approved by at least one committer and no changes requested.

github-actions · 2023-08-20T14:26:23Z

PR approved by anyone and no changes requested.

github-actions · 2023-08-23T06:43:20Z

clang-tidy review says "All clean, LGTM! 👍"

morningman

LGTM

hubgeter · 2023-08-25T04:28:00Z

run buildall

github-actions · 2023-08-25T04:35:26Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-25T05:23:24Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.38 seconds
stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17162092058 Bytes

hubgeter · 2023-08-25T13:38:20Z

run buildall

github-actions · 2023-08-25T13:44:59Z

clang-tidy review says "All clean, LGTM! 👍"

hello-stephen · 2023-08-25T14:36:04Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.5 seconds
stream load tsv: 537 seconds loaded 74807831229 Bytes, about 132 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 64 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 31 seconds loaded 861443392 Bytes, about 26 MB/s
insert into select: 29.4 seconds inserted 10000000 Rows, about 340K ops/s
storage size: 17162023308 Bytes

morningman

LGTM

github-actions · 2023-08-25T16:06:46Z

PR approved by at least one committer and no changes requested.

…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```

…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)" This reverts commit 39c339c.

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)" (#23928) This reverts commit 39c339c.

…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```

…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" This reverts commit 01ee595.

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" (#24002) This reverts commit 01ee595.

…tlogs,db,table and add profiling table in order to Compatible with mysql (apache#22702) add information_schema.metadata_name_idsfor quickly get catlogs,db,table. 1. table struct : ```mysql mysql> desc internal.information_schema.metadata_name_ids; +---------------+--------------+------+-------+---------+-------+ | Field | Type | Null | Key | Default | Extra | +---------------+--------------+------+-------+---------+-------+ | CATALOG_ID | BIGINT | Yes | false | NULL | | | CATALOG_NAME | VARCHAR(512) | Yes | false | NULL | | | DATABASE_ID | BIGINT | Yes | false | NULL | | | DATABASE_NAME | VARCHAR(64) | Yes | false | NULL | | | TABLE_ID | BIGINT | Yes | false | NULL | | | TABLE_NAME | VARCHAR(64) | Yes | false | NULL | | +---------------+--------------+------+-------+---------+-------+ 6 rows in set (0.00 sec) mysql> select * from internal.information_schema.metadata_name_ids where CATALOG_NAME="hive1" limit 1 \G; *************************** 1. row *************************** CATALOG_ID: 113008 CATALOG_NAME: hive1 DATABASE_ID: 113042 DATABASE_NAME: ssb1_parquet TABLE_ID: 114009 TABLE_NAME: dates 1 row in set (0.07 sec) ``` 2. when you create / drop catalog , need not refresh catalog . ```mysql mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.34 sec) mysql> drop catalog hive2; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) mysql> create catalog hive3 ... mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 21301 1 row in set (0.32 sec) ``` 3. create / drop table , need not refresh catalog . ```mysql mysql> CREATE TABLE IF NOT EXISTS demo.example_tbl ... ; mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10666 1 row in set (0.04 sec) mysql> drop table demo.example_tbl; Query OK, 0 rows affected (0.01 sec) mysql> select count(*) from internal.information_schema.metadata_name_ids\G; *************************** 1. row *************************** count(*): 10665 1 row in set (0.04 sec) ``` 4. you can set query time , prevent queries from taking too long . ``` fe.conf : query_metadata_name_ids_timeout the time used to obtain all tables in one database ``` 5. add information_schema.profiling in order to Compatible with mysql ```mysql mysql> select * from information_schema.profiling; Empty set (0.07 sec) mysql> set profiling=1; Query OK, 0 rows affected (0.01 sec) ```

hubgeter changed the title ~~add information_schema.simple_tables for quickly get catlogs,db,table.~~ [feature](information_schema)add simple_tables for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql Aug 8, 2023

morningman added the dev/2.0.1 label Aug 9, 2023

morningman reviewed Aug 9, 2023

View reviewed changes

hubgeter force-pushed the tables branch from afa864b to 9a5f1df Compare August 11, 2023 05:44

hubgeter force-pushed the tables branch from 1094ef3 to 37c6cbe Compare August 18, 2023 07:06

morningman previously approved these changes Aug 20, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 20, 2023

github-actions bot added the reviewed label Aug 20, 2023

hubgeter dismissed morningman’s stale review via 1022c1e August 23, 2023 06:35

hubgeter force-pushed the tables branch from efe4ce8 to 1022c1e Compare August 23, 2023 06:35

morningman previously approved these changes Aug 24, 2023

View reviewed changes

hubgeter dismissed morningman’s stale review via b97a572 August 25, 2023 04:27

hubgeter force-pushed the tables branch from 1022c1e to b97a572 Compare August 25, 2023 04:27

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 25, 2023

fix regression test

b450f3c

hubgeter force-pushed the tables branch from b97a572 to b450f3c Compare August 25, 2023 13:36

morningman approved these changes Aug 25, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 25, 2023

xiaokang added dev/2.0.2 and removed dev/2.0.1 labels Aug 30, 2023

BePPPower approved these changes Aug 31, 2023

View reviewed changes

morningman merged commit e680d42 into apache:master Aug 31, 2023

xiaokang added the merge_conflict label Aug 31, 2023

xiaokang pushed a commit that referenced this pull request Sep 5, 2023

[feature](information_schema)add metadata_name_ids for quickly get ca…

39c339c

…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)

BiteTheDDDDt added a commit that referenced this pull request Sep 5, 2023

Revert "[feature](information_schema)add metadata_name_ids for quickl…

e6e33d7

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23753)" This reverts commit 39c339c.

BiteTheDDDDt mentioned this pull request Sep 5, 2023

[Chore](revert) revert 23753 on 2.0 #23928

Merged

hubgeter mentioned this pull request Sep 6, 2023

[feature](information_schema)add metadata_name_ids for quickly get c… #23980

Merged

xiaokang pushed a commit that referenced this pull request Sep 6, 2023

[feature](information_schema)add metadata_name_ids for quickly get ca…

01ee595

…tlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)

xiaokang added a commit that referenced this pull request Sep 6, 2023

Revert "[feature](information_schema)add metadata_name_ids for quickl…

e8744ff

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" This reverts commit 01ee595.

xiaokang added a commit that referenced this pull request Sep 6, 2023

Revert "[feature](information_schema)add metadata_name_ids for quickl…

7cc1d65

…y get catlogs,db,table and add profiling table in order to Compatible with mysql (#22702) (#23980)" (#24002) This reverts commit 01ee595.

morningman mentioned this pull request Sep 8, 2023

[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #24059

Merged

morningman added dev/2.0.2-merged and removed dev/2.0.2 labels Sep 8, 2023

xiaokang mentioned this pull request Sep 30, 2023

Release Note 2.0.2 #25011

Closed

xiaokang mentioned this pull request Dec 3, 2023

Release Note 2.0.3 #27909

Closed

[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #22702

[feature](information_schema)add metadata_name_ids for quickly get catlogs,db,table and add profiling table in order to Compatible with mysql #22702

Uh oh!

Conversation

hubgeter commented Aug 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Further comments

Uh oh!

github-actions bot commented Aug 8, 2023

Uh oh!

github-actions bot commented Aug 8, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 11, 2023

Uh oh!

github-actions bot commented Aug 11, 2023

Uh oh!

hubgeter commented Aug 18, 2023

Uh oh!

github-actions bot commented Aug 18, 2023

Uh oh!

hubgeter commented Aug 18, 2023

Uh oh!

github-actions bot commented Aug 18, 2023

Uh oh!

hello-stephen commented Aug 18, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 20, 2023

Uh oh!

github-actions bot commented Aug 20, 2023

Uh oh!

github-actions bot commented Aug 23, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

hubgeter commented Aug 25, 2023

Uh oh!

github-actions bot commented Aug 25, 2023

Uh oh!

hello-stephen commented Aug 25, 2023

Uh oh!

hubgeter commented Aug 25, 2023

Uh oh!

github-actions bot commented Aug 25, 2023

Uh oh!

hello-stephen commented Aug 25, 2023

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 25, 2023

Uh oh!

Reviewers

hubgeter commented Aug 8, 2023 •

edited

Loading