Skip to content

[improvement](segmentcache) limit segment cache by fd limit or memory #39658

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Aug 21, 2024
Merged

[improvement](segmentcache) limit segment cache by fd limit or memory #39658

dataroaring merged 1 commit into from
Aug 21, 2024

Conversation

@dataroaring
Copy link
Contributor

@dataroaring dataroaring commented Aug 20, 2024

remove a useless config.

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Aug 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor Author

dataroaring commented Aug 20, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Aug 20, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Aug 20, 2024

TPC-H: Total hot run time: 38843 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a17104ed585135dde65d7746c0344e1644ad97a, data reload: false

------ Round 1 ----------------------------------
q1	18202	4726	4455	4455
q2	2888	211	215	211
q3	10791	1176	1115	1115
q4	10738	763	844	763
q5	8219	2943	2909	2909
q6	271	164	165	164
q7	1027	666	653	653
q8	9403	2141	2108	2108
q9	7410	6581	6587	6581
q10	7104	2293	2371	2293
q11	507	284	266	266
q12	440	259	252	252
q13	17785	3017	2965	2965
q14	309	250	260	250
q15	560	534	523	523
q16	536	405	408	405
q17	993	623	683	623
q18	7556	6947	6912	6912
q19	5282	1088	1174	1088
q20	706	336	352	336
q21	3909	3035	2932	2932
q22	1127	1042	1039	1039
Total cold run time: 115763 ms
Total hot run time: 38843 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4550	4286	4358	4286
q2	402	310	302	302
q3	2840	2664	2623	2623
q4	1905	1628	1657	1628
q5	5405	5431	5400	5400
q6	231	147	150	147
q7	2093	1720	1715	1715
q8	3250	3403	3429	3403
q9	8488	8448	8554	8448
q10	3450	3213	3198	3198
q11	632	529	525	525
q12	805	643	627	627
q13	17678	3002	2973	2973
q14	319	298	319	298
q15	571	515	535	515
q16	521	456	456	456
q17	1806	1519	1497	1497
q18	7924	7549	7497	7497
q19	1722	1563	1630	1563
q20	2046	1803	1836	1803
q21	7438	5086	5133	5086
q22	1149	1033	1046	1033
Total cold run time: 75225 ms
Total hot run time: 55023 ms

@doris-robot
Copy link

doris-robot commented Aug 20, 2024

TPC-DS: Total hot run time: 190924 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a17104ed585135dde65d7746c0344e1644ad97a, data reload: false

query1	990	398	393	393
query2	6743	1983	1978	1978
query3	6675	222	251	222
query4	34163	23218	23301	23218
query5	4364	741	726	726
query6	323	217	211	211
query7	4615	330	327	327
query8	474	439	438	438
query9	8709	2549	2513	2513
query10	512	359	353	353
query11	17850	14974	15020	14974
query12	193	142	138	138
query13	1730	455	459	455
query14	10030	7323	7131	7131
query15	282	197	188	188
query16	8192	500	512	500
query17	1692	604	582	582
query18	2185	339	360	339
query19	364	170	172	170
query20	144	136	136	136
query21	245	143	141	141
query22	4358	4083	3948	3948
query23	34066	33272	33429	33272
query24	11344	2886	2889	2886
query25	676	424	429	424
query26	1184	185	183	183
query27	2536	299	302	299
query28	7296	2113	2086	2086
query29	868	461	447	447
query30	323	184	188	184
query31	1049	845	853	845
query32	119	81	83	81
query33	809	359	346	346
query34	924	512	516	512
query35	888	790	760	760
query36	1126	954	935	935
query37	179	108	105	105
query38	3984	3850	3791	3791
query39	1522	1465	1478	1465
query40	244	159	159	159
query41	144	138	138	138
query42	141	122	119	119
query43	538	508	512	508
query44	1238	785	805	785
query45	230	229	197	197
query46	1121	797	762	762
query47	1929	1832	1884	1832
query48	412	342	345	342
query49	1187	600	590	590
query50	871	466	460	460
query51	7243	7181	6929	6929
query52	122	111	112	111
query53	302	239	232	232
query54	969	516	504	504
query55	90	91	93	91
query56	334	326	320	320
query57	1253	1137	1111	1111
query58	321	300	335	300
query59	2940	2797	2829	2797
query60	367	339	338	338
query61	149	149	145	145
query62	909	697	694	694
query63	269	223	222	222
query64	5321	2405	1838	1838
query65	3258	3194	3193	3193
query66	1149	678	713	678
query67	15463	15167	14923	14923
query68	8703	614	611	611
query69	724	431	348	348
query70	1440	1159	1105	1105
query71	555	328	323	323
query72	6870	2295	2067	2067
query73	2216	362	366	362
query74	8958	8858	8904	8858
query75	4369	2761	2713	2713
query76	5261	988	1040	988
query77	817	451	507	451
query78	10435	9068	8978	8978
query79	7505	571	558	558
query80	1481	617	622	617
query81	633	267	263	263
query82	318	159	157	157
query83	271	217	214	214
query84	300	99	98	98
query85	1031	353	349	349
query86	366	334	321	321
query87	4318	4202	4311	4202
query88	4631	2520	2511	2511
query89	534	341	336	336
query90	1984	242	236	236
query91	155	171	129	129
query92	91	75	77	75
query93	5542	562	544	544
query94	982	296	325	296
query95	392	293	294	293
query96	630	295	287	287
query97	3192	3091	3086	3086
query98	255	247	225	225
query99	1556	1303	1325	1303
Total cold run time: 318755 ms
Total hot run time: 190924 ms

@doris-robot
Copy link

doris-robot commented Aug 20, 2024

ClickBench: Total hot run time: 31.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a17104ed585135dde65d7746c0344e1644ad97a, data reload: false

query1	0.04	0.05	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.07	0.07
query5	0.54	0.51	0.52
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.55	0.47	0.49
query10	0.56	0.53	0.54
query11	0.16	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.60	0.60
query14	0.77	0.80	0.80
query15	0.85	0.84	0.82
query16	0.36	0.39	0.38
query17	0.98	1.03	0.97
query18	0.22	0.21	0.21
query19	1.85	1.70	1.86
query20	0.02	0.01	0.01
query21	15.40	0.65	0.66
query22	5.40	5.32	2.72
query23	18.29	1.45	1.31
query24	2.07	0.25	0.22
query25	0.13	0.09	0.09
query26	0.26	0.17	0.19
query27	0.08	0.09	0.09
query28	13.25	1.04	1.02
query29	12.64	3.32	3.30
query30	0.44	0.26	0.25
query31	2.80	0.40	0.40
query32	3.25	0.49	0.49
query33	2.98	2.96	2.99
query34	17.20	4.35	4.36
query35	4.45	4.42	4.42
query36	0.66	0.48	0.50
query37	0.22	0.17	0.18
query38	0.17	0.17	0.17
query39	0.06	0.06	0.06
query40	0.19	0.15	0.16
query41	0.12	0.08	0.07
query42	0.07	0.06	0.07
query43	0.07	0.07	0.07
Total cold run time: 111.08 s
Total hot run time: 31.9 s

@github-actions
Copy link
Contributor

github-actions bot commented Aug 21, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 21, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 21, 2024

PR approved by anyone and no changes requested.

zhannngchen
zhannngchen approved these changes Aug 21, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 86d9721 into apache:master Aug 21, 2024
dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Aug 21, 2024
@dataroaring
[improvement](segmentcache) limit segment cache by fd limit or memory (…
183b1fb 
…apache#39658)

remove a useless config.
@dataroaring dataroaring mentioned this pull request Aug 21, 2024
Merged
dataroaring added a commit that referenced this pull request Aug 21, 2024
@dataroaring
[improvement](segmentcache) limit segment cache by fd limit or memory… (
610f694 
#39689)

… (#39658)

remove a useless config.
dataroaring added a commit that referenced this pull request Oct 9, 2024
@dataroaring
[improvement](segmentcache) limit segment cache by fd limit or memory (…
996388c 
…#39658)

remove a useless config.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.3-merged kind/behavior-changed reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dataroaring @doris-robot @gavinchou @zhannngchen