Skip to content

[enhancement](cloud) expose file cache capacity for each disk #42867

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Oct 30, 2024
Merged

[enhancement](cloud) expose file cache capacity for each disk #42867

dataroaring merged 1 commit into from
Oct 30, 2024

Conversation

@freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Oct 29, 2024

expose file cache capacity for better monitoring and easier to calculate file cache space usage using prometheus.

Proposed changes

Issue Number: close #xxx

@freemandealer
[enhancement](cloud) expose file cache capacity for each disk
81fb0dd 
expose file cache capacity for better monitoring and easier to calculate
file cache space usage using prometheus.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@doris-robot
Copy link

doris-robot commented Oct 29, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@freemandealer
Copy link
Contributor Author

freemandealer commented Oct 29, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

TPC-H: Total hot run time: 41547 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 81fb0dd4fec5aed841d162a18e8a9eee94f91b49, data reload: false

------ Round 1 ----------------------------------
q1	17600	7730	7348	7348
q2	2073	169	162	162
q3	10955	1148	1185	1148
q4	10577	813	901	813
q5	7757	3109	3078	3078
q6	234	152	150	150
q7	1026	609	614	609
q8	9351	1992	2020	1992
q9	6687	6457	6473	6457
q10	7070	2394	2439	2394
q11	440	251	247	247
q12	399	220	213	213
q13	17807	3036	3004	3004
q14	242	214	210	210
q15	565	508	507	507
q16	646	586	602	586
q17	981	553	568	553
q18	7445	6718	6716	6716
q19	1333	1115	984	984
q20	464	187	183	183
q21	4017	3200	3440	3200
q22	1122	993	997	993
Total cold run time: 108791 ms
Total hot run time: 41547 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7305	7286	7335	7286
q2	329	229	227	227
q3	3057	2989	3052	2989
q4	2181	1890	1863	1863
q5	6119	5809	5848	5809
q6	237	142	145	142
q7	2221	1845	1855	1845
q8	3395	3577	3557	3557
q9	9017	8963	8968	8963
q10	3583	3576	3900	3576
q11	589	493	488	488
q12	867	642	600	600
q13	6400	3207	3208	3207
q14	327	286	285	285
q15	581	523	527	523
q16	684	636	665	636
q17	1892	1661	1636	1636
q18	8302	7888	7773	7773
q19	1712	1566	1569	1566
q20	2103	1899	1893	1893
q21	5570	5450	5535	5450
q22	1121	1036	1062	1036
Total cold run time: 67592 ms
Total hot run time: 61350 ms

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

TeamCity be ut coverage result:
Function Coverage: 37.47% (9718/25938)
Line Coverage: 28.72% (80591/280619)
Region Coverage: 28.14% (41660/148038)
Branch Coverage: 24.71% (21165/85666)
Coverage Report: http://coverage.selectdb-in.cc/coverage/81fb0dd4fec5aed841d162a18e8a9eee94f91b49_81fb0dd4fec5aed841d162a18e8a9eee94f91b49/report/index.html

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

TPC-DS: Total hot run time: 197502 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 81fb0dd4fec5aed841d162a18e8a9eee94f91b49, data reload: false

query1	1226	1002	1032	1002
query2	6240	2093	2059	2059
query3	11519	4936	4791	4791
query4	33584	23591	23642	23591
query5	4694	448	437	437
query6	280	182	174	174
query7	3980	291	307	291
query8	295	237	231	231
query9	9570	2752	2740	2740
query10	502	256	251	251
query11	18275	15248	15381	15248
query12	153	103	102	102
query13	1568	431	404	404
query14	10265	7426	7753	7426
query15	259	183	191	183
query16	8171	483	480	480
query17	1580	601	575	575
query18	2161	315	326	315
query19	370	152	155	152
query20	125	116	120	116
query21	212	111	109	109
query22	4790	4638	4332	4332
query23	35048	34311	34150	34150
query24	11465	2725	2777	2725
query25	655	386	402	386
query26	1651	154	156	154
query27	2567	282	301	282
query28	7774	2462	2471	2462
query29	1001	422	414	414
query30	260	158	160	158
query31	1021	816	797	797
query32	90	55	56	55
query33	762	264	265	264
query34	988	511	511	511
query35	1045	875	881	875
query36	1114	938	946	938
query37	197	81	76	76
query38	4475	4220	4341	4220
query39	1534	1427	1428	1427
query40	255	97	97	97
query41	49	48	47	47
query42	111	103	100	100
query43	518	479	510	479
query44	1279	827	824	824
query45	184	166	169	166
query46	1134	699	698	698
query47	1927	1855	1842	1842
query48	432	323	339	323
query49	1020	404	403	403
query50	801	397	413	397
query51	7144	7040	6975	6975
query52	99	93	85	85
query53	254	181	175	175
query54	1250	412	392	392
query55	83	75	79	75
query56	251	233	240	233
query57	1330	1180	1143	1143
query58	219	200	197	197
query59	3220	2991	2882	2882
query60	273	247	250	247
query61	100	95	99	95
query62	868	715	675	675
query63	210	189	177	177
query64	5040	608	623	608
query65	3280	3236	3211	3211
query66	1337	313	297	297
query67	16450	15947	15729	15729
query68	4844	570	573	570
query69	426	250	255	250
query70	1230	1168	1151	1151
query71	382	263	257	257
query72	6295	3971	3956	3956
query73	770	363	369	363
query74	10181	9032	8979	8979
query75	3437	2658	2672	2658
query76	2712	929	937	929
query77	375	277	279	277
query78	10686	9571	9551	9551
query79	1134	613	588	588
query80	766	463	419	419
query81	528	246	237	237
query82	1328	115	113	113
query83	224	147	132	132
query84	218	70	68	68
query85	1105	287	282	282
query86	319	266	301	266
query87	4838	4737	4616	4616
query88	3489	2194	2247	2194
query89	410	285	290	285
query90	2017	183	184	183
query91	132	104	97	97
query92	57	46	49	46
query93	1190	561	547	547
query94	755	284	288	284
query95	348	250	244	244
query96	619	276	284	276
query97	2984	2721	2716	2716
query98	210	196	197	196
query99	1519	1344	1297	1297
Total cold run time: 306112 ms
Total hot run time: 197502 ms

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

ClickBench: Total hot run time: 32.36 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 81fb0dd4fec5aed841d162a18e8a9eee94f91b49, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.41	0.39	0.40
query6	1.17	0.68	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.55	0.56	0.55
query11	0.13	0.10	0.11
query12	0.13	0.11	0.11
query13	0.63	0.59	0.59
query14	2.70	2.76	2.71
query15	0.89	0.82	0.82
query16	0.38	0.40	0.38
query17	1.06	1.00	1.01
query18	0.19	0.20	0.20
query19	1.96	1.84	1.87
query20	0.02	0.01	0.01
query21	15.36	0.59	0.59
query22	2.33	3.05	1.43
query23	17.03	0.78	0.87
query24	3.25	1.27	1.48
query25	0.18	0.12	0.08
query26	0.68	0.13	0.13
query27	0.04	0.04	0.06
query28	10.12	1.09	1.08
query29	12.63	3.30	3.30
query30	0.24	0.06	0.05
query31	2.97	0.38	0.38
query32	3.31	0.46	0.46
query33	2.93	3.02	3.03
query34	17.11	4.46	4.41
query35	4.45	4.46	4.50
query36	0.66	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.58 s
Total hot run time: 32.36 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Oct 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Oct 30, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit fe4da33 into apache:master Oct 30, 2024
github-actions bot pushed a commit that referenced this pull request Oct 30, 2024
@freemandealer
[enhancement](cloud) expose file cache capacity for each disk (#42867)
60b764a 
expose file cache capacity for better monitoring and easier to calculate
file cache space usage using prometheus.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@github-actions github-actions bot mentioned this pull request Oct 30, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 7, 2024
@github-actions @freemandealer
branch-3.0: [enhancement](cloud) expose file cache capacity for each …
4b85b60 
…disk (#42896)

PR Body: expose file cache capacity for better monitoring and easier to
calculate file cache space usage using prometheus.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

 
 Cherry-picked from #42867

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
Co-authored-by: zhengyu <freeman.zhang1992@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@freemandealer @doris-robot @gavinchou @dataroaring