Skip to content

[improvement](segmentcache) limit segment cache by memory or segment num #37026

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
Jun 30, 2024
Merged

[improvement](segmentcache) limit segment cache by memory or segment num #37026

dataroaring merged 3 commits into from
Jun 30, 2024

Conversation

@dataroaring
Copy link
Contributor

@dataroaring dataroaring commented Jun 28, 2024 •
edited by gavinchou
Loading

Also enlarge columns per segment.

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor Author

dataroaring commented Jun 28, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

clang-tidy review says "All clean, LGTM! 👍"

gavinchou
gavinchou previously approved these changes Jun 28, 2024
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 28, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor Author

dataroaring commented Jun 28, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

clang-tidy review says "All clean, LGTM! 👍"

@dataroaring
Copy link
Contributor Author

dataroaring commented Jun 29, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 29, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 29, 2024

TPC-H: Total hot run time: 39929 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b8ee16a18416ede3ceeafc01af509783c039dac7, data reload: false

------ Round 1 ----------------------------------
q1	17632	4368	4257	4257
q2	2019	194	198	194
q3	10464	1164	1211	1164
q4	10191	836	755	755
q5	7504	2675	2672	2672
q6	223	137	136	136
q7	955	595	601	595
q8	9218	2116	2098	2098
q9	8841	6537	6466	6466
q10	8945	3714	3728	3714
q11	453	233	233	233
q12	426	231	220	220
q13	17768	2959	3001	2959
q14	267	220	219	219
q15	524	483	484	483
q16	515	372	374	372
q17	982	672	712	672
q18	8182	7431	7431	7431
q19	7522	1569	1477	1477
q20	657	317	339	317
q21	4964	3155	3917	3155
q22	387	340	340	340
Total cold run time: 118639 ms
Total hot run time: 39929 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4450	4206	4226	4206
q2	373	276	267	267
q3	3039	2826	2922	2826
q4	2002	1745	1756	1745
q5	5556	5536	5433	5433
q6	230	130	130	130
q7	2224	1884	1846	1846
q8	3249	3481	3429	3429
q9	8770	8784	8814	8784
q10	4279	3720	3824	3720
q11	589	479	510	479
q12	820	637	633	633
q13	17094	3145	3201	3145
q14	301	260	281	260
q15	538	480	495	480
q16	467	430	406	406
q17	1849	1548	1534	1534
q18	8094	7980	7824	7824
q19	3532	1732	1615	1615
q20	2172	1850	1848	1848
q21	5143	4882	4777	4777
q22	641	536	542	536
Total cold run time: 75412 ms
Total hot run time: 55923 ms

@doris-robot
Copy link

doris-robot commented Jun 29, 2024

TPC-DS: Total hot run time: 174542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b8ee16a18416ede3ceeafc01af509783c039dac7, data reload: false

query1	907	380	371	371
query2	6449	2517	2305	2305
query3	6645	204	218	204
query4	19387	17670	17292	17292
query5	3758	495	493	493
query6	253	155	160	155
query7	4584	291	289	289
query8	322	296	297	296
query9	8469	2416	2388	2388
query10	572	314	286	286
query11	10932	10165	10089	10089
query12	114	88	87	87
query13	1644	358	374	358
query14	9298	7775	7582	7582
query15	256	188	195	188
query16	7932	289	275	275
query17	1869	564	547	547
query18	2062	282	277	277
query19	203	154	154	154
query20	86	83	91	83
query21	211	144	136	136
query22	4542	4177	4214	4177
query23	33821	33588	33581	33581
query24	11140	2879	2974	2879
query25	663	409	404	404
query26	775	160	158	158
query27	2355	320	319	319
query28	6564	2129	2127	2127
query29	923	716	628	628
query30	249	162	160	160
query31	996	755	761	755
query32	104	60	55	55
query33	776	326	294	294
query34	965	475	472	472
query35	790	637	641	637
query36	1141	980	986	980
query37	147	82	82	82
query38	2952	2904	2843	2843
query39	898	832	833	832
query40	215	132	130	130
query41	58	55	56	55
query42	110	111	104	104
query43	607	513	532	513
query44	1227	728	742	728
query45	188	164	170	164
query46	1090	699	737	699
query47	1871	1787	1791	1787
query48	380	294	302	294
query49	912	517	419	419
query50	773	377	384	377
query51	6827	6685	6728	6685
query52	110	93	93	93
query53	357	293	287	287
query54	869	454	438	438
query55	73	76	73	73
query56	292	257	256	256
query57	1117	1049	1030	1030
query58	247	234	243	234
query59	3260	3337	3198	3198
query60	302	288	310	288
query61	96	92	94	92
query62	598	429	442	429
query63	311	282	287	282
query64	8754	2251	1732	1732
query65	3135	3079	3186	3079
query66	729	325	314	314
query67	15882	14892	15125	14892
query68	7935	532	543	532
query69	708	398	334	334
query70	1218	1131	1145	1131
query71	562	284	274	274
query72	8592	5184	5535	5184
query73	806	323	317	317
query74	5920	5557	5617	5557
query75	5043	2663	2694	2663
query76	4908	914	916	914
query77	769	298	303	298
query78	10573	9879	9772	9772
query79	8202	522	520	520
query80	1044	483	488	483
query81	545	219	224	219
query82	743	111	110	110
query83	325	175	170	170
query84	265	88	87	87
query85	1310	276	272	272
query86	443	304	311	304
query87	3279	3088	3086	3086
query88	4856	2352	2381	2352
query89	561	391	392	391
query90	2032	189	190	189
query91	132	100	101	100
query92	66	48	50	48
query93	6289	504	492	492
query94	1260	188	192	188
query95	405	323	318	318
query96	617	268	271	268
query97	3236	3036	3015	3015
query98	216	201	191	191
query99	1162	856	853	853
Total cold run time: 290166 ms
Total hot run time: 174542 ms

@doris-robot
Copy link

doris-robot commented Jun 29, 2024

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b8ee16a18416ede3ceeafc01af509783c039dac7, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.07
query5	0.51	0.50	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.50	0.51
query10	0.56	0.54	0.54
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.60	0.59
query14	0.80	0.78	0.77
query15	0.86	0.81	0.82
query16	0.36	0.38	0.33
query17	0.97	0.98	1.03
query18	0.21	0.24	0.22
query19	1.76	1.68	1.65
query20	0.02	0.01	0.01
query21	15.44	0.76	0.66
query22	4.42	6.77	2.39
query23	18.30	1.42	1.16
query24	2.14	0.22	0.22
query25	0.15	0.09	0.08
query26	0.26	0.18	0.19
query27	0.08	0.08	0.09
query28	13.16	1.01	1.00
query29	12.62	3.32	3.28
query30	0.25	0.06	0.06
query31	2.85	0.38	0.39
query32	3.29	0.48	0.49
query33	2.90	2.89	2.90
query34	16.90	4.39	4.40
query35	4.51	4.49	4.50
query36	0.65	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.37 s
Total hot run time: 30.88 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 30, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 30, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit aa43892 into apache:master Jun 30, 2024
dataroaring added a commit to dataroaring/incubator-doris that referenced this pull request Jun 30, 2024
@dataroaring
[improvement](segmentcache) limit segment cache by memory or segment …
7b04452 
…num (apache#37026)

Also enlarge columns per segment.
@dataroaring dataroaring mentioned this pull request Jun 30, 2024
Merged
dataroaring added a commit that referenced this pull request Jun 30, 2024
@dataroaring
[improvement](segmentcache) limit segment cache by memory or segment …
3df52cb 
…num (#37026)

Also enlarge columns per segment.
dataroaring added a commit that referenced this pull request Jun 30, 2024
@dataroaring
[improvement](segmentcache) limit segment cache by memory or segment … (
07278e9 
#37035)

…num (#37026)

pick ##37026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dataroaring @doris-robot @gavinchou @Yukang-Lian