[fix](multi-catalog) fix table not found error after catalog creation #45521

liutang123 · 2024-12-17T07:46:37Z

If we set lower_case_table_names to 2, and then create a catalog with meta cache. The lowerCaseToTableName in ExternalDatabse will be empty because listTableNames method is invoked in meta cache. Now, if we directly execute getTableNullable, we will get null.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #38227

Problem Summary:
If we directly query a table after the catalog creation, we will get a AnalysisException with table not found message.

2024-12-17 11:38:43,574 WARN (mysql-nio-pool-16|8552) [StmtExecutor.executeByNereids():807] Nereids plan query failed:
select * from hudi_new_1.hudi_db.hudi_table_col13
org.apache.doris.nereids.exceptions.AnalysisException: Table [hudi_table_col13] does not exist in database [hudi_db].
        at org.apache.doris.nereids.util.RelationUtil.getDbAndTable(RelationUtil.java:156) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.util.RelationUtil.getTable(RelationUtil.java:133) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.analysis.BindRelation.bind(BindRelation.java:203) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.analysis.BindRelation.doBindRelation(BindRelation.java:147) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.rules.analysis.BindRelation.lambda$build$0(BindRelation.java:123) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.pattern.PatternMatcher$1.transform(PatternMatcher.java:92) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteJob.rewrite(PlanTreeRewriteJob.java:57) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteBottomUpJob.rewriteThis(PlanTreeRewriteBottomUpJob.java:91) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.PlanTreeRewriteBottomUpJob.execute(PlanTreeRewriteBottomUpJob.java:75) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.scheduler.SimpleJobScheduler.executeJobPool(SimpleJobScheduler.java:44) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.rewrite.RootPlanTreeRewriteJob.execute(RootPlanTreeRewriteJob.java:66) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.executor.AbstractBatchJobExecutor.execute(AbstractBatchJobExecutor.java:139) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.jobs.executor.Analyzer.analyze(Analyzer.java:104) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.lambda$analyze$2(NereidsPlanner.java:313) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.keepOrShowPlanProcess(NereidsPlanner.java:727) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.analyze(NereidsPlanner.java:313) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithoutLock(NereidsPlanner.java:223) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.planWithLock(NereidsPlanner.java:205) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.nereids.NereidsPlanner.plan(NereidsPlanner.java:129) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.executeByNereids(StmtExecutor.java:801) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:560) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.StmtExecutor.execute(StmtExecutor.java:536) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.executeQuery(ConnectProcessor.java:344) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:225) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.handleQuery(MysqlConnectProcessor.java:285) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.dispatch(MysqlConnectProcessor.java:313) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.qe.MysqlConnectProcessor.processOnce(MysqlConnectProcessor.java:483) ~[doris-fe.jar:1.2-SNAPSHOT]
        at org.apache.doris.mysql.ReadListener.lambda$handleEvent$0(ReadListener.java:52) ~[doris-fe.jar:1.2-SNAPSHOT]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:829) ~[?:?]

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2024-12-17T07:46:43Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

liutang123 · 2024-12-17T09:12:50Z

run buildall

liutang123 · 2024-12-18T01:57:09Z

run feut

liutang123 · 2024-12-18T03:36:01Z

run feut

If we set lower_case_table_names to 2, and then create a catalog with meta cache. The `lowerCaseToTableName` in `ExternalDatabse` will be empty because `listTableNames` method is invoked in meta cache. Now, if we directly execute `getTableNullable`, we will get null.

liutang123 · 2024-12-18T07:21:41Z

run buildall

github-actions · 2024-12-18T07:22:07Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

doris-robot · 2024-12-18T07:53:19Z

TPC-H: Total hot run time: 39344 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 143d7bb62112ddb8b608f77f2b6cf8d73cb2441e, data reload: false

------ Round 1 ----------------------------------
q1	17645	7374	7256	7256
q2	2048	178	174	174
q3	10647	1087	1121	1087
q4	10573	760	757	757
q5	7617	2661	2690	2661
q6	234	147	146	146
q7	968	624	599	599
q8	9252	1800	1805	1800
q9	6603	6373	6341	6341
q10	7015	2260	2316	2260
q11	471	265	258	258
q12	408	218	236	218
q13	17774	2926	2974	2926
q14	254	212	213	212
q15	563	509	495	495
q16	663	608	600	600
q17	978	572	502	502
q18	7122	6559	6631	6559
q19	1348	1003	935	935
q20	466	179	182	179
q21	3963	3117	3071	3071
q22	370	308	309	308
Total cold run time: 106982 ms
Total hot run time: 39344 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7221	7167	7347	7167
q2	332	225	236	225
q3	2886	2805	2894	2805
q4	2071	1823	1784	1784
q5	5678	5645	5637	5637
q6	225	143	145	143
q7	2249	1851	1811	1811
q8	3334	3523	3474	3474
q9	8842	8903	8913	8903
q10	3569	3497	3488	3488
q11	595	505	494	494
q12	875	613	585	585
q13	14158	3035	3177	3035
q14	326	282	280	280
q15	572	515	504	504
q16	666	652	653	652
q17	1847	1605	1620	1605
q18	8421	7807	7820	7807
q19	2224	1558	1613	1558
q20	2051	1866	1830	1830
q21	5622	5329	5467	5329
q22	660	560	592	560
Total cold run time: 74424 ms
Total hot run time: 59676 ms

doris-robot · 2024-12-18T08:05:41Z

TPC-DS: Total hot run time: 196014 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 143d7bb62112ddb8b608f77f2b6cf8d73cb2441e, data reload: false

query1	1306	978	915	915
query2	6247	2373	2282	2282
query3	10977	4485	4605	4485
query4	32936	23330	23457	23330
query5	4239	450	454	450
query6	282	204	174	174
query7	4023	301	307	301
query8	301	237	237	237
query9	9460	2693	2685	2685
query10	480	241	236	236
query11	17755	15186	15037	15037
query12	153	102	105	102
query13	1549	398	397	397
query14	9896	7267	7035	7035
query15	270	196	202	196
query16	8228	466	499	466
query17	1604	646	559	559
query18	2204	316	313	313
query19	375	167	166	166
query20	117	110	118	110
query21	209	113	106	106
query22	4945	4723	4799	4723
query23	34604	33601	35496	33601
query24	11354	2572	2456	2456
query25	536	396	420	396
query26	712	151	150	150
query27	2613	329	337	329
query28	7370	2445	2453	2445
query29	660	410	429	410
query30	226	149	152	149
query31	1038	834	831	831
query32	124	56	53	53
query33	748	321	284	284
query34	1168	522	540	522
query35	903	784	774	774
query36	1109	952	982	952
query37	117	77	80	77
query38	4328	4053	4098	4053
query39	1503	1486	1453	1453
query40	209	100	101	100
query41	53	45	44	44
query42	120	107	102	102
query43	535	500	487	487
query44	1289	841	841	841
query45	188	171	165	165
query46	1192	710	729	710
query47	2060	1933	1951	1933
query48	426	329	327	327
query49	1046	411	412	411
query50	856	402	425	402
query51	7345	7217	7196	7196
query52	108	94	93	93
query53	261	198	185	185
query54	1296	428	443	428
query55	82	83	82	82
query56	270	244	243	243
query57	1256	1159	1144	1144
query58	230	216	227	216
query59	3288	3258	3166	3166
query60	274	259	237	237
query61	107	103	104	103
query62	868	715	709	709
query63	219	195	195	195
query64	3885	683	639	639
query65	3267	3262	3246	3246
query66	789	357	308	308
query67	16486	15469	15533	15469
query68	6001	545	555	545
query69	504	254	249	249
query70	1201	1121	1126	1121
query71	516	262	254	254
query72	6288	4186	4143	4143
query73	790	359	368	359
query74	10039	8833	8823	8823
query75	3925	2785	2593	2593
query76	4430	1017	1132	1017
query77	691	277	278	277
query78	10224	9764	9420	9420
query79	2069	604	610	604
query80	923	434	430	430
query81	511	244	231	231
query82	274	112	117	112
query83	191	143	143	143
query84	289	72	72	72
query85	1049	303	308	303
query86	363	303	297	297
query87	4697	4332	4343	4332
query88	4400	2212	2187	2187
query89	472	285	296	285
query90	2162	184	186	184
query91	142	101	106	101
query92	65	48	54	48
query93	4717	540	539	539
query94	916	280	254	254
query95	354	249	252	249
query96	646	276	282	276
query97	2886	2676	2664	2664
query98	215	190	204	190
query99	1609	1328	1291	1291
Total cold run time: 308233 ms
Total hot run time: 196014 ms

doris-robot · 2024-12-18T08:11:12Z

ClickBench: Total hot run time: 32.86 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 143d7bb62112ddb8b608f77f2b6cf8d73cb2441e, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.60	0.11	0.11
query5	0.43	0.42	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.49	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.13	0.11	0.10
query13	0.61	0.63	0.59
query14	2.72	2.73	2.78
query15	0.89	0.82	0.82
query16	0.37	0.39	0.36
query17	1.05	1.04	1.02
query18	0.22	0.21	0.21
query19	1.93	1.85	2.00
query20	0.01	0.01	0.02
query21	15.36	0.61	0.59
query22	3.19	2.33	2.56
query23	16.80	1.04	0.90
query24	3.13	0.67	2.26
query25	0.29	0.25	0.04
query26	0.33	0.14	0.13
query27	0.04	0.05	0.05
query28	9.95	1.11	1.06
query29	12.58	3.33	3.27
query30	0.25	0.07	0.06
query31	2.85	0.39	0.38
query32	3.25	0.47	0.46
query33	2.98	3.06	3.08
query34	17.04	4.44	4.52
query35	4.46	4.47	4.50
query36	0.64	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.14	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.45 s
Total hot run time: 32.86 s

liutang123 · 2024-12-18T09:21:54Z

run feut

liutang123 · 2024-12-19T01:15:19Z

run feut

liutang123 · 2024-12-19T06:17:17Z

run feut

liutang123 · 2025-01-13T08:30:45Z

#41510 fix this problem

… creation apache#45521

…rror after catalog creation apache#45521" due to apache#41510 has fixed this bug. This reverts commit 9f72b3d534f1b6eb53b5198cd3f0f757e71d35f8.

… creation apache#45521

…rror after catalog creation apache#45521" due to apache#41510 has fixed this bug. This reverts commit 9f72b3d534f1b6eb53b5198cd3f0f757e71d35f8.

liutang123 requested a review from zy-kkk December 17, 2024 07:46

liutang123 changed the title ~~[fix](multicatalog) fix table not found error after catalog creation~~ [fix](multi-catalog) fix table not found error after catalog creation Dec 17, 2024

liutang123 added 2 commits December 18, 2024 15:21

fix ut

143d7bb

liutang123 force-pushed the fix-external-table-cache branch from 1fdf46b to 143d7bb Compare December 18, 2024 07:21

liutang123 mentioned this pull request Dec 20, 2024

[improvement](jdbc catalog) Optimize JdbcCatalog case mapping stability #41510

Merged

16 tasks

liutang123 closed this Jan 13, 2025

liutang123 pushed a commit to liutang123/doris that referenced this pull request Apr 29, 2025

[Tencent][fix](multi-catalog) fix table not found error after catalog…

5976628

… creation apache#45521

cambyzju pushed a commit to cambyzju/incubator-doris that referenced this pull request Jul 17, 2025

[Tencent][fix](multi-catalog) fix table not found error after catalog…

3a7b500

… creation apache#45521

[fix](multi-catalog) fix table not found error after catalog creation #45521

[fix](multi-catalog) fix table not found error after catalog creation #45521

Uh oh!

Conversation

liutang123 commented Dec 17, 2024 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 17, 2024

Uh oh!

liutang123 commented Dec 17, 2024

Uh oh!

liutang123 commented Dec 18, 2024

Uh oh!

liutang123 commented Dec 18, 2024

Uh oh!

liutang123 commented Dec 18, 2024

Uh oh!

github-actions bot commented Dec 18, 2024

sh-checker report

Uh oh!

doris-robot commented Dec 18, 2024

Uh oh!

doris-robot commented Dec 18, 2024

Uh oh!

doris-robot commented Dec 18, 2024

Uh oh!

liutang123 commented Dec 18, 2024

Uh oh!

liutang123 commented Dec 19, 2024

Uh oh!

liutang123 commented Dec 19, 2024

Uh oh!

liutang123 commented Jan 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

liutang123 commented Dec 17, 2024 •

edited by morningman

Loading

`sh-checker report`