branch-3.0: [feat](hive) add catalog level schema cache property #50958 #51057 #51011

github-actions · 2025-05-19T03:37:57Z

Cherry-picked from #50958 #51057

### What problem does this PR solve? Problem Summary: Just same as #50724, support enable or disable schema cache at Catalog level for all kinds of external catalogs. Previously, if user want to disable the schema cache, they can only set the `max_external_schema_cache_num=0` in fe.conf and restart FE. And this config will effect all catalogs. In this PR, I add a new catalog property `schema.cache.ttl-second`. If set to 0, the schema cache will be disabled, so if schema is changed Doris will read the new schema immediately.

hello-stephen · 2025-05-19T03:38:05Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hello-stephen · 2025-05-19T03:38:08Z

run buildall

doris-robot · 2025-05-19T04:12:13Z

TPC-H: Total hot run time: 39801 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be6a790d5424611c006d4413e1205abf8ff38569, data reload: false

------ Round 1 ----------------------------------
q1	17576	6721	6564	6564
q2	2085	165	161	161
q3	10797	1067	1116	1067
q4	10548	745	677	677
q5	7750	2917	2739	2739
q6	218	133	128	128
q7	957	616	610	610
q8	9358	1923	2024	1923
q9	6652	6361	6416	6361
q10	7045	2269	2302	2269
q11	465	260	264	260
q12	406	219	213	213
q13	17791	2963	2959	2959
q14	240	205	204	204
q15	501	467	467	467
q16	659	591	583	583
q17	990	571	553	553
q18	7138	6695	6673	6673
q19	1504	1055	1083	1055
q20	483	209	212	209
q21	4114	3122	3307	3122
q22	1136	1023	1004	1004
Total cold run time: 108413 ms
Total hot run time: 39801 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6651	6572	6574	6572
q2	328	235	237	235
q3	2861	2795	2896	2795
q4	2027	1803	1767	1767
q5	5787	5730	5693	5693
q6	215	128	124	124
q7	2224	1808	1864	1808
q8	3345	3597	3840	3597
q9	9028	8817	8902	8817
q10	3549	3509	3505	3505
q11	599	508	495	495
q12	783	605	624	605
q13	8266	3167	3182	3167
q14	286	289	280	280
q15	505	473	462	462
q16	673	659	654	654
q17	1827	1620	1598	1598
q18	8120	7814	7807	7807
q19	1690	1607	1558	1558
q20	2042	1821	1840	1821
q21	5513	5387	5327	5327
q22	1096	1060	1052	1052
Total cold run time: 67415 ms
Total hot run time: 59739 ms

doris-robot · 2025-05-19T04:24:52Z

TPC-DS: Total hot run time: 198549 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be6a790d5424611c006d4413e1205abf8ff38569, data reload: false

query1	1277	905	917	905
query2	6320	2013	1942	1942
query3	10803	4402	4177	4177
query4	61119	29594	23814	23814
query5	5215	483	468	468
query6	398	174	173	173
query7	5440	308	306	306
query8	301	228	211	211
query9	8161	2581	2573	2573
query10	453	276	259	259
query11	17734	15187	15687	15187
query12	163	105	104	104
query13	1466	451	455	451
query14	10654	7736	7465	7465
query15	210	182	185	182
query16	7116	494	506	494
query17	1214	593	633	593
query18	1937	322	321	321
query19	220	161	173	161
query20	122	113	112	112
query21	202	101	107	101
query22	4730	4559	4585	4559
query23	35717	34155	34797	34155
query24	6223	2990	3063	2990
query25	520	383	406	383
query26	659	169	168	168
query27	1937	365	359	359
query28	4296	2423	2418	2418
query29	713	464	431	431
query30	240	167	163	163
query31	1020	830	837	830
query32	66	59	54	54
query33	465	301	302	301
query34	928	542	509	509
query35	872	736	722	722
query36	1097	949	949	949
query37	114	64	74	64
query38	4066	4014	3992	3992
query39	1536	1477	1505	1477
query40	211	102	106	102
query41	54	51	51	51
query42	126	106	109	106
query43	551	490	500	490
query44	1229	830	829	829
query45	192	173	176	173
query46	1180	775	771	771
query47	2010	1905	1924	1905
query48	493	390	399	390
query49	752	429	434	429
query50	837	441	449	441
query51	7519	7203	7239	7203
query52	106	90	89	89
query53	272	194	194	194
query54	585	460	475	460
query55	78	74	76	74
query56	271	257	257	257
query57	1264	1150	1142	1142
query58	239	233	219	219
query59	3187	3012	3017	3012
query60	296	263	263	263
query61	136	121	107	107
query62	790	706	702	702
query63	235	201	211	201
query64	1405	671	627	627
query65	3272	3201	3237	3201
query66	635	300	291	291
query67	16018	15560	15546	15546
query68	4129	577	580	577
query69	415	268	269	268
query70	1152	1134	1142	1134
query71	322	265	255	255
query72	6127	4220	4208	4208
query73	762	351	355	351
query74	10085	8938	9211	8938
query75	3374	2691	2757	2691
query76	1906	1001	1015	1001
query77	544	277	275	275
query78	10607	9684	9595	9595
query79	2078	606	608	606
query80	1384	442	425	425
query81	526	240	238	238
query82	1242	90	88	88
query83	279	140	160	140
query84	280	77	82	77
query85	1052	302	294	294
query86	377	309	298	298
query87	4445	4261	4306	4261
query88	3699	2392	2369	2369
query89	414	290	295	290
query90	1931	186	188	186
query91	191	146	151	146
query92	60	51	51	51
query93	2947	565	552	552
query94	751	308	291	291
query95	364	265	254	254
query96	631	276	280	276
query97	3325	3172	3128	3128
query98	225	201	197	197
query99	1608	1291	1310	1291
Total cold run time: 316899 ms
Total hot run time: 198549 ms

doris-robot · 2025-05-19T04:30:20Z

ClickBench: Total hot run time: 31.74 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be6a790d5424611c006d4413e1205abf8ff38569, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.53	0.53	0.51
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.50	0.51
query10	0.57	0.55	0.59
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	2.78	2.76	2.78
query15	0.88	0.82	0.81
query16	0.39	0.39	0.40
query17	1.07	1.06	1.02
query18	0.25	0.21	0.21
query19	1.96	1.90	2.00
query20	0.01	0.01	0.02
query21	15.35	0.60	0.59
query22	2.97	1.69	1.80
query23	16.93	1.09	0.90
query24	3.43	2.03	0.59
query25	0.24	0.15	0.22
query26	0.33	0.13	0.13
query27	0.04	0.04	0.03
query28	9.61	0.49	0.44
query29	12.60	3.19	3.16
query30	0.25	0.06	0.06
query31	2.89	0.39	0.38
query32	3.26	0.46	0.46
query33	2.97	2.99	3.04
query34	17.11	4.48	4.49
query35	4.54	4.54	4.47
query36	0.65	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.71 s
Total hot run time: 31.74 s

morningman · 2025-05-20T02:20:35Z

Do not merge now

…get_schema_from_table" mode (#51057) Related PR: #50958 Problem Summary: In #50958, I introduced the catalog level schema cache switch. But if use also add `"get_schema_from_table" = "true"`, the schema will be fetched from `Hive Table` instead of schema cache, so the schema cache config is broken. This PR fix it: 1. Always get new `Hive Table` instance to get schema from table. To make schema change config work. 2. For `show create table xxx`, it will always show the latest schema of a hive table.

dataroaring · 2025-06-11T03:17:14Z

run buildall

doris-robot · 2025-06-11T04:29:06Z

TPC-H: Total hot run time: 39773 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 108443ccec317f4c15349c999894f2b32deb8890, data reload: false

------ Round 1 ----------------------------------
q1	17561	6833	6604	6604
q2	2055	162	181	162
q3	10573	1103	1240	1103
q4	10574	801	714	714
q5	7757	2888	2872	2872
q6	219	139	133	133
q7	985	616	609	609
q8	9361	1959	2100	1959
q9	6600	6437	6453	6437
q10	7027	2298	2279	2279
q11	483	270	258	258
q12	396	216	212	212
q13	17794	2962	3008	2962
q14	223	215	212	212
q15	491	472	475	472
q16	493	385	380	380
q17	996	559	547	547
q18	7311	6601	6674	6601
q19	1394	995	1104	995
q20	476	197	203	197
q21	3896	3261	3119	3119
q22	1090	1006	946	946
Total cold run time: 107755 ms
Total hot run time: 39773 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6553	6553	6585	6553
q2	335	233	223	223
q3	2853	2804	2820	2804
q4	2008	1870	1837	1837
q5	5798	5717	5738	5717
q6	208	129	132	129
q7	2258	1770	1839	1770
q8	3387	3592	3534	3534
q9	8934	8753	8936	8753
q10	3605	3537	3497	3497
q11	596	502	489	489
q12	807	625	616	616
q13	9601	3191	3142	3142
q14	289	279	282	279
q15	521	463	483	463
q16	482	455	442	442
q17	1848	1646	1620	1620
q18	8377	7772	7716	7716
q19	1723	1638	1542	1542
q20	2132	1899	1868	1868
q21	5053	5011	5113	5011
q22	1122	1033	1016	1016
Total cold run time: 68490 ms
Total hot run time: 59021 ms

doris-robot · 2025-06-11T04:41:39Z

TPC-DS: Total hot run time: 196144 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 108443ccec317f4c15349c999894f2b32deb8890, data reload: false

query1	1266	919	890	890
query2	6296	1922	1859	1859
query3	11091	4414	4494	4414
query4	61352	28779	23821	23821
query5	5231	440	443	440
query6	403	174	178	174
query7	5473	315	302	302
query8	311	220	217	217
query9	8696	2609	2590	2590
query10	473	260	257	257
query11	17960	15115	15665	15115
query12	165	99	107	99
query13	1454	432	414	414
query14	10384	6598	6563	6563
query15	199	177	188	177
query16	7096	488	523	488
query17	1148	576	574	574
query18	1890	324	332	324
query19	226	167	172	167
query20	119	113	124	113
query21	205	100	102	100
query22	4673	4409	4618	4409
query23	34223	34252	33917	33917
query24	6141	2863	2906	2863
query25	526	424	412	412
query26	652	167	164	164
query27	1814	357	367	357
query28	4010	2196	2131	2131
query29	695	478	450	450
query30	237	161	157	157
query31	986	843	861	843
query32	73	54	57	54
query33	406	293	296	293
query34	931	497	538	497
query35	872	723	720	720
query36	1091	969	940	940
query37	105	62	70	62
query38	4002	3970	3942	3942
query39	1532	1465	1456	1456
query40	206	99	110	99
query41	48	48	47	47
query42	113	98	97	97
query43	509	482	492	482
query44	1175	825	814	814
query45	182	173	178	173
query46	1158	740	717	717
query47	2025	1879	1906	1879
query48	488	371	378	371
query49	738	412	399	399
query50	828	428	433	428
query51	7370	7330	7311	7311
query52	101	85	89	85
query53	257	191	181	181
query54	566	458	465	458
query55	79	79	80	79
query56	282	270	241	241
query57	1286	1137	1158	1137
query58	213	206	213	206
query59	3225	3088	3017	3017
query60	276	256	247	247
query61	111	107	107	107
query62	755	687	657	657
query63	221	181	181	181
query64	1355	648	620	620
query65	3270	3179	3193	3179
query66	706	290	291	290
query67	15818	15539	15510	15510
query68	4267	574	555	555
query69	438	267	255	255
query70	1167	1120	1110	1110
query71	339	254	247	247
query72	6399	4015	3996	3996
query73	744	344	362	344
query74	10128	9015	9065	9015
query75	3358	2664	2695	2664
query76	1967	1034	1029	1029
query77	487	270	266	266
query78	10605	9658	9608	9608
query79	2248	594	594	594
query80	1331	411	431	411
query81	500	219	217	217
query82	943	86	84	84
query83	162	142	137	137
query84	279	75	75	75
query85	982	319	290	290
query86	373	265	284	265
query87	4509	4259	4218	4218
query88	4104	2397	2371	2371
query89	421	289	279	279
query90	1989	181	184	181
query91	173	146	144	144
query92	60	47	51	47
query93	2602	571	559	559
query94	795	287	280	280
query95	351	263	250	250
query96	617	279	281	279
query97	3289	3131	3172	3131
query98	208	210	188	188
query99	1584	1304	1278	1278
Total cold run time: 315187 ms
Total hot run time: 196144 ms

doris-robot · 2025-06-11T04:46:58Z

ClickBench: Total hot run time: 30.56 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 108443ccec317f4c15349c999894f2b32deb8890, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.53	0.50	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.56	0.55	0.54
query11	0.14	0.10	0.10
query12	0.15	0.12	0.11
query13	0.61	0.60	0.59
query14	0.77	0.78	0.82
query15	0.85	0.83	0.82
query16	0.39	0.37	0.40
query17	1.05	1.05	1.03
query18	0.22	0.21	0.21
query19	1.78	1.86	1.74
query20	0.01	0.01	0.01
query21	15.41	0.59	0.56
query22	2.52	2.39	1.64
query23	16.91	1.01	0.86
query24	3.45	1.83	1.68
query25	0.27	0.10	0.08
query26	0.67	0.13	0.13
query27	0.04	0.04	0.04
query28	8.84	0.53	0.51
query29	12.55	3.19	3.17
query30	0.24	0.07	0.05
query31	2.86	0.37	0.39
query32	3.26	0.46	0.45
query33	2.99	2.97	2.99
query34	17.10	4.53	4.49
query35	4.49	4.50	4.53
query36	0.66	0.48	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.52 s
Total hot run time: 30.56 s

dataroaring

LGTM

github-actions bot requested a review from dataroaring as a code owner May 19, 2025 03:37

dataroaring closed this May 19, 2025

dataroaring reopened this May 19, 2025

morningman changed the title ~~branch-3.0: [feat](hive) add catalog level schema cache property #50958~~ branch-3.0: [feat](hive) add catalog level schema cache property #50958 #51057 May 21, 2025

dataroaring closed this Jun 11, 2025

dataroaring reopened this Jun 11, 2025

dataroaring approved these changes Jun 24, 2025

View reviewed changes

dataroaring merged commit a160dd2 into branch-3.0 Jun 24, 2025
39 of 43 checks passed

github-actions bot deleted the auto-pick-50958-branch-3.0 branch June 24, 2025 07:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

branch-3.0: [feat](hive) add catalog level schema cache property #50958 #51057 #51011

branch-3.0: [feat](hive) add catalog level schema cache property #50958 #51057 #51011

Uh oh!

github-actions bot commented May 19, 2025 •

edited by morningman

Loading

Uh oh!

hello-stephen commented May 19, 2025

Uh oh!

hello-stephen commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

morningman commented May 20, 2025

Uh oh!

dataroaring commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

dataroaring left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

branch-3.0: [feat](hive) add catalog level schema cache property #50958 #51057 #51011

branch-3.0: [feat](hive) add catalog level schema cache property #50958 #51057 #51011

Uh oh!

Conversation

github-actions bot commented May 19, 2025 • edited by morningman Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hello-stephen commented May 19, 2025

Uh oh!

hello-stephen commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

morningman commented May 20, 2025

Uh oh!

dataroaring commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

doris-robot commented Jun 11, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented May 19, 2025 •

edited by morningman

Loading