Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #50958

### What problem does this PR solve?

Problem Summary:
Just same as #50724, support enable or disable schema cache at Catalog
level for all kinds of external catalogs.

Previously, if user want to disable the schema cache, they can only set
the
`max_external_schema_cache_num=0` in fe.conf and restart FE.
And this config will effect all catalogs.

In this PR, I add a new catalog property `schema.cache.ttl-second`.
If set to 0, the schema cache will be disabled, so if schema is changed
Doris will read the new schema immediately.
@github-actions github-actions bot requested a review from morrySnow as a code owner June 24, 2025 07:23
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jun 24, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39605 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9023b83e416c8c5a8db4e70708f9cba863f13860, data reload: false

------ Round 1 ----------------------------------
q1	17662	6814	6588	6588
q2	2079	173	161	161
q3	10617	1142	1144	1142
q4	10225	724	796	724
q5	7729	2868	2784	2784
q6	220	136	136	136
q7	975	630	620	620
q8	9356	1948	1994	1948
q9	6620	6377	6345	6345
q10	7065	2240	2265	2240
q11	460	261	258	258
q12	414	210	214	210
q13	17841	3002	2976	2976
q14	235	201	208	201
q15	496	474	468	468
q16	475	375	370	370
q17	983	556	587	556
q18	7353	6621	6652	6621
q19	1324	1025	931	931
q20	460	202	201	201
q21	3919	3139	3263	3139
q22	1086	986	1018	986
Total cold run time: 107594 ms
Total hot run time: 39605 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6832	6623	6642	6623
q2	325	235	237	235
q3	2942	2780	2884	2780
q4	2068	1836	1849	1836
q5	5712	5735	5634	5634
q6	205	125	124	124
q7	2185	1798	1813	1798
q8	3334	3514	3534	3514
q9	8943	8756	8796	8756
q10	3559	3556	3501	3501
q11	592	486	513	486
q12	828	626	607	607
q13	9664	3200	3165	3165
q14	315	263	263	263
q15	505	470	468	468
q16	510	427	427	427
q17	1842	1639	1578	1578
q18	8108	7621	7630	7621
q19	1643	1573	1474	1474
q20	2090	1841	1853	1841
q21	5318	4825	4766	4766
q22	1069	1014	984	984
Total cold run time: 68589 ms
Total hot run time: 58481 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190912 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9023b83e416c8c5a8db4e70708f9cba863f13860, data reload: false

query1	985	372	371	371
query2	6529	1892	1854	1854
query3	6704	215	232	215
query4	33935	23443	23789	23443
query5	4409	475	448	448
query6	293	207	188	188
query7	4631	334	324	324
query8	301	234	222	222
query9	9648	2582	2562	2562
query10	512	277	264	264
query11	18073	15206	15178	15178
query12	150	100	103	100
query13	1656	423	423	423
query14	9037	7220	7161	7161
query15	257	173	175	173
query16	8101	455	489	455
query17	1635	579	563	563
query18	2113	315	317	315
query19	288	158	160	158
query20	122	110	107	107
query21	205	105	105	105
query22	4418	4348	4301	4301
query23	34466	33592	34101	33592
query24	11685	2919	2898	2898
query25	673	416	432	416
query26	1511	173	174	173
query27	2792	354	350	350
query28	7724	2107	2116	2107
query29	739	448	444	444
query30	330	163	170	163
query31	992	792	806	792
query32	99	60	65	60
query33	802	301	313	301
query34	910	506	538	506
query35	952	711	732	711
query36	1053	932	925	925
query37	129	70	70	70
query38	4001	3835	3857	3835
query39	1469	1418	1415	1415
query40	288	104	105	104
query41	54	49	51	49
query42	119	106	105	105
query43	526	465	481	465
query44	1238	802	802	802
query45	187	171	170	170
query46	1158	736	712	712
query47	1916	1809	1826	1809
query48	438	337	341	337
query49	1297	420	418	418
query50	812	428	426	426
query51	7304	7248	7197	7197
query52	104	96	95	95
query53	272	188	193	188
query54	1267	475	503	475
query55	77	79	76	76
query56	276	250	251	250
query57	1269	1147	1157	1147
query58	235	209	213	209
query59	3009	2871	2778	2778
query60	288	264	259	259
query61	115	160	113	113
query62	876	672	677	672
query63	213	192	192	192
query64	5199	663	636	636
query65	3276	3209	3187	3187
query66	1421	307	330	307
query67	15896	15738	15660	15660
query68	4884	601	581	581
query69	425	266	262	262
query70	1179	1135	1055	1055
query71	321	266	258	258
query72	6201	4080	4129	4080
query73	742	352	352	352
query74	10743	9254	8943	8943
query75	3368	2676	2643	2643
query76	2846	1099	1041	1041
query77	393	276	287	276
query78	10510	9486	9527	9486
query79	2061	607	607	607
query80	1177	429	428	428
query81	537	221	221	221
query82	897	91	89	89
query83	221	149	143	143
query84	240	82	83	82
query85	1353	314	296	296
query86	429	296	305	296
query87	4429	4206	4230	4206
query88	3651	2398	2347	2347
query89	422	299	293	293
query90	2169	193	190	190
query91	147	111	109	109
query92	66	53	55	53
query93	1586	567	562	562
query94	1053	302	302	302
query95	369	260	263	260
query96	604	281	279	279
query97	3286	3147	3149	3147
query98	210	202	192	192
query99	1534	1351	1307	1307
Total cold run time: 301512 ms
Total hot run time: 190912 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9023b83e416c8c5a8db4e70708f9cba863f13860, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.51	0.50	0.50
query6	1.14	0.74	0.72
query7	0.04	0.02	0.01
query8	0.03	0.03	0.03
query9	0.57	0.50	0.50
query10	0.54	0.56	0.54
query11	0.15	0.09	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	0.78	0.78	0.80
query15	0.85	0.82	0.82
query16	0.38	0.38	0.38
query17	1.03	1.06	0.99
query18	0.23	0.22	0.23
query19	1.88	1.81	1.80
query20	0.01	0.02	0.01
query21	15.41	0.55	0.57
query22	2.38	1.71	2.36
query23	16.83	1.00	0.80
query24	3.28	1.65	1.43
query25	0.15	0.06	0.08
query26	0.62	0.14	0.13
query27	0.05	0.04	0.04
query28	9.20	0.52	0.52
query29	12.60	3.19	3.18
query30	0.25	0.06	0.06
query31	2.86	0.39	0.40
query32	3.23	0.45	0.46
query33	2.98	2.98	3.04
query34	17.16	4.50	4.48
query35	4.52	4.51	4.52
query36	0.66	0.49	0.50
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.18	0.12	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.53 s
Total hot run time: 30.41 s

@morningman morningman merged commit d10f242 into branch-3.1 Jun 24, 2025
21 of 23 checks passed
@github-actions github-actions bot deleted the auto-pick-50958-branch-3.1 branch June 24, 2025 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants