Skip to content

branch-3.0: [feat](hive) add catalog level partition cache property #50724 #50761

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
May 13, 2025
Merged

branch-3.0: [feat](hive) add catalog level partition cache property #50724 #50761

dataroaring merged 1 commit into from
May 13, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented May 9, 2025

Cherry-picked from #50724

@morningman
[feat](hive) add catalog level partition cache property (#50724)
1244acf 
### What problem does this PR solve?

Problem Summary:
Support enable or disable hive partition cache at Catalog level for hive
catalog.

Previously, if user want to disable the hive partition cache, they can
only set the
`max_hive_partition_table_cache_num=0` in fe.conf and restart FE.
And this config will effect all catalogs.

In this PR, I add a new catalog property `partition.cache.ttl-second`.
If set to 0, the hive partition cache will be disabled, so if new
partitioned is added,
Doris will read the new partition immediately.
@github-actions github-actions bot requested a review from dataroaring as a code owner May 9, 2025 09:06
@hello-stephen
Copy link
Contributor

hello-stephen commented May 9, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this May 9, 2025
@dataroaring dataroaring reopened this May 9, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented May 9, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 9, 2025

TPC-H: Total hot run time: 40031 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1244acfc059b849cde943bc11f788b5fdd070390, data reload: false

------ Round 1 ----------------------------------
q1	17628	6689	6593	6593
q2	2066	171	166	166
q3	10663	1090	1179	1090
q4	10565	771	705	705
q5	7713	2890	2791	2791
q6	223	137	139	137
q7	966	614	609	609
q8	9359	1949	2093	1949
q9	6518	6391	6431	6391
q10	7005	2266	2327	2266
q11	451	257	266	257
q12	405	219	211	211
q13	17829	2988	3005	2988
q14	245	208	218	208
q15	512	471	464	464
q16	662	585	612	585
q17	975	518	588	518
q18	7293	6699	6688	6688
q19	1391	1160	1074	1074
q20	483	210	204	204
q21	4020	3149	3200	3149
q22	1074	988	996	988
Total cold run time: 108046 ms
Total hot run time: 40031 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6568	6545	6603	6545
q2	321	244	238	238
q3	2897	2726	2877	2726
q4	2031	1873	1758	1758
q5	5737	5781	5732	5732
q6	215	130	132	130
q7	2233	1806	1869	1806
q8	3374	3562	3534	3534
q9	8932	8807	8797	8797
q10	3576	3524	3464	3464
q11	599	498	486	486
q12	833	617	638	617
q13	9838	3171	3154	3154
q14	302	269	285	269
q15	520	472	472	472
q16	694	645	667	645
q17	1823	1601	1611	1601
q18	8263	7928	7696	7696
q19	1700	1571	1461	1461
q20	2040	1819	1874	1819
q21	5555	5367	5288	5288
q22	1106	1099	1026	1026
Total cold run time: 69157 ms
Total hot run time: 59264 ms

@doris-robot
Copy link

doris-robot commented May 9, 2025

TPC-DS: Total hot run time: 197722 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1244acfc059b849cde943bc11f788b5fdd070390, data reload: false

query1	1279	919	892	892
query2	6277	1984	1963	1963
query3	10802	4304	4309	4304
query4	61072	29545	23412	23412
query5	5172	462	453	453
query6	399	177	173	173
query7	5462	318	305	305
query8	302	233	217	217
query9	8628	2683	2593	2593
query10	460	276	265	265
query11	17499	15212	15769	15212
query12	170	108	109	108
query13	1425	459	434	434
query14	9815	7171	7275	7171
query15	195	183	185	183
query16	7255	463	481	463
query17	1127	590	572	572
query18	1938	320	308	308
query19	218	158	149	149
query20	115	108	112	108
query21	213	104	104	104
query22	4719	4409	4700	4409
query23	34528	34111	33929	33929
query24	6147	2967	2945	2945
query25	528	419	424	419
query26	666	188	182	182
query27	1888	370	369	369
query28	4220	2457	2415	2415
query29	705	472	456	456
query30	245	160	169	160
query31	980	808	856	808
query32	73	61	61	61
query33	429	290	333	290
query34	943	527	520	520
query35	848	767	746	746
query36	1088	947	970	947
query37	120	70	73	70
query38	4017	3973	4036	3973
query39	1682	1689	1476	1476
query40	202	112	105	105
query41	52	53	51	51
query42	118	105	104	104
query43	524	488	479	479
query44	1176	825	831	825
query45	189	179	176	176
query46	1139	728	711	711
query47	2026	1901	1904	1901
query48	519	419	419	419
query49	736	417	413	413
query50	845	451	432	432
query51	7285	7310	7203	7203
query52	103	92	92	92
query53	266	190	191	190
query54	583	493	501	493
query55	84	85	85	85
query56	293	286	268	268
query57	1290	1171	1148	1148
query58	231	227	224	224
query59	3258	3109	3110	3109
query60	279	253	262	253
query61	133	135	109	109
query62	786	686	695	686
query63	225	190	198	190
query64	1409	701	685	685
query65	3266	3220	3213	3213
query66	685	297	305	297
query67	15960	15681	15726	15681
query68	4191	585	578	578
query69	434	270	269	269
query70	1196	1116	1114	1114
query71	345	254	257	254
query72	6346	4103	4040	4040
query73	785	350	363	350
query74	10335	9027	9084	9027
query75	3390	2623	2657	2623
query76	1998	1075	966	966
query77	504	275	277	275
query78	10603	9643	9613	9613
query79	1442	595	590	590
query80	876	436	428	428
query81	507	240	240	240
query82	1246	97	88	88
query83	242	142	155	142
query84	279	82	85	82
query85	870	311	292	292
query86	330	293	288	288
query87	4500	4308	4275	4275
query88	3533	2383	2379	2379
query89	418	297	290	290
query90	1976	187	194	187
query91	190	153	153	153
query92	61	54	50	50
query93	1928	555	546	546
query94	743	299	299	299
query95	362	258	254	254
query96	611	280	278	278
query97	3328	3216	3166	3166
query98	214	202	195	195
query99	1583	1313	1285	1285
Total cold run time: 312677 ms
Total hot run time: 197722 ms

@doris-robot
Copy link

doris-robot commented May 9, 2025

ClickBench: Total hot run time: 31.84 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1244acfc059b849cde943bc11f788b5fdd070390, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.61	0.11	0.11
query5	0.50	0.53	0.51
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.57	0.50	0.48
query10	0.55	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.14
query13	0.61	0.59	0.60
query14	2.87	2.84	2.87
query15	0.90	0.84	0.82
query16	0.37	0.38	0.38
query17	1.02	1.04	1.05
query18	0.23	0.22	0.22
query19	1.91	1.84	1.99
query20	0.01	0.02	0.01
query21	15.35	0.60	0.58
query22	3.16	2.66	1.14
query23	17.06	0.95	0.84
query24	3.35	1.78	1.28
query25	0.33	0.07	0.15
query26	0.48	0.14	0.13
query27	0.07	0.03	0.04
query28	9.31	0.56	0.53
query29	12.56	3.21	3.20
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.23	0.47	0.45
query33	3.03	2.99	3.03
query34	17.31	4.47	4.41
query35	4.50	4.48	4.50
query36	0.67	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.02	0.03
Total cold run time: 106.97 s
Total hot run time: 31.84 s

dataroaring
dataroaring approved these changes May 13, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 13, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented May 13, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor Author

github-actions bot commented May 13, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit a510382 into branch-3.0 May 13, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-50724-branch-3.0 branch May 13, 2025 14:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @dataroaring @morningman