Skip to content

[cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" #37602

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jul 13, 2024
Merged

[cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support dynamic delete bitmap cache (#32991)" #37602

dataroaring merged 1 commit into from
Jul 13, 2024

Conversation

@Yukang-Lian
Copy link
Collaborator

@Yukang-Lian Yukang-Lian commented Jul 10, 2024 •
edited by dataroaring
Loading

The default delete bitmap cache is set to 100MB, which can be insufficient and cause performance issues when the amount of user data is large. To mitigate the problem of an inadequate cache, we will take the larger of 5% of the total memory and 100MB as the delete bitmap cache size.

pick #32991

Proposed changes

Issue Number: close #xxx

@Yukang-Lian
[Enhancement](merge-on-write) Support dynamic delete bitmap cache (ap…
f188b0d 
…ache#32991)

* The default delete bitmap cache is set to 100MB, which can be insufficient and cause performance issues when the amount of user data is large. To mitigate the problem of an inadequate cache, we will take the larger of 5% of the total memory and 100MB as the delete bitmap cache size.
@doris-robot
Copy link

doris-robot commented Jul 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 10, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Yukang-Lian
Copy link
Collaborator Author

Yukang-Lian commented Jul 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-H: Total hot run time: 50116 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd, data reload: false

------ Round 1 ----------------------------------
q1	17822	4450	4459	4450
q2	2085	161	148	148
q3	10257	1927	1932	1927
q4	10118	1253	1328	1253
q5	8470	3962	3962	3962
q6	242	132	125	125
q7	2056	1607	1583	1583
q8	9382	2758	2738	2738
q9	11031	10493	10273	10273
q10	8671	3570	3549	3549
q11	424	237	248	237
q12	474	301	303	301
q13	18354	3950	4030	3950
q14	346	324	334	324
q15	528	456	462	456
q16	667	572	588	572
q17	1155	959	981	959
q18	7390	6906	6852	6852
q19	1839	1645	1603	1603
q20	588	312	297	297
q21	4482	4107	4127	4107
q22	541	456	450	450
Total cold run time: 116922 ms
Total hot run time: 50116 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4305	4290	4327	4290
q2	324	222	223	222
q3	4171	4136	4156	4136
q4	2763	2720	2758	2720
q5	7188	7118	7119	7118
q6	241	120	118	118
q7	3285	2871	2860	2860
q8	4418	4467	4487	4467
q9	16797	16798	16682	16682
q10	4227	4271	4258	4258
q11	716	718	691	691
q12	1041	838	842	838
q13	4412	3693	3741	3693
q14	452	438	432	432
q15	518	456	454	454
q16	729	682	673	673
q17	3814	3808	3835	3808
q18	8857	8697	8851	8697
q19	1753	1685	1674	1674
q20	2382	2182	2095	2095
q21	8479	8516	8554	8516
q22	1030	992	988	988
Total cold run time: 81902 ms
Total hot run time: 79430 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TPC-DS: Total hot run time: 203121 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd, data reload: false

query1	919	427	374	374
query2	6526	2654	2630	2630
query3	6922	203	199	199
query4	20499	18061	17956	17956
query5	19734	6508	6601	6508
query6	365	213	234	213
query7	4780	292	295	292
query8	457	430	415	415
query9	3135	2671	2608	2608
query10	429	300	300	300
query11	11449	10768	10797	10768
query12	121	80	74	74
query13	5600	730	706	706
query14	18945	13644	14054	13644
query15	369	241	258	241
query16	6451	288	263	263
query17	1618	1461	887	887
query18	2299	410	411	410
query19	221	150	154	150
query20	81	76	80	76
query21	192	96	94	94
query22	5228	5092	5023	5023
query23	32377	31965	32145	31965
query24	6987	6548	6517	6517
query25	524	436	442	436
query26	532	162	165	162
query27	1821	298	296	296
query28	6181	2360	2325	2325
query29	2883	2782	2773	2773
query30	243	163	168	163
query31	922	740	792	740
query32	70	65	61	61
query33	399	275	257	257
query34	844	469	472	469
query35	1155	934	876	876
query36	1332	1104	1330	1104
query37	90	63	60	60
query38	3054	2920	2909	2909
query39	1393	1314	1321	1314
query40	202	96	97	96
query41	49	45	44	44
query42	79	82	83	82
query43	679	685	686	685
query44	1118	724	720	720
query45	251	242	234	234
query46	1239	979	973	973
query47	1919	1751	1714	1714
query48	1020	717	724	717
query49	626	370	373	370
query50	863	605	642	605
query51	4763	4659	4696	4659
query52	97	81	79	79
query53	448	328	328	328
query54	2655	2479	2470	2470
query55	90	86	88	86
query56	247	217	221	217
query57	1272	1049	1227	1049
query58	224	223	198	198
query59	4215	3716	3735	3716
query60	231	203	197	197
query61	101	94	96	94
query62	883	454	473	454
query63	480	350	343	343
query64	2483	1514	1396	1396
query65	3618	3546	3566	3546
query66	794	381	381	381
query67	16172	16989	14873	14873
query68	8703	658	649	649
query69	582	361	350	350
query70	1627	1337	1344	1337
query71	414	311	318	311
query72	6529	3514	3487	3487
query73	733	324	321	321
query74	6480	5920	5881	5881
query75	5346	3611	3689	3611
query76	5220	1131	1121	1121
query77	863	272	264	264
query78	12788	11651	11577	11577
query79	8170	650	640	640
query80	1302	398	407	398
query81	488	236	237	236
query82	1699	95	102	95
query83	181	130	134	130
query84	256	72	70	70
query85	889	328	333	328
query86	339	291	306	291
query87	3220	3018	3000	3000
query88	4886	2303	2308	2303
query89	387	305	324	305
query90	1932	206	213	206
query91	197	139	140	139
query92	58	52	53	52
query93	5549	575	567	567
query94	713	211	204	204
query95	1129	1076	1077	1076
query96	652	323	321	321
query97	6473	6433	6259	6259
query98	205	181	172	172
query99	3023	836	864	836
Total cold run time: 314998 ms
Total hot run time: 203121 ms

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

TeamCity be ut coverage result:
Function Coverage: 37.89% (8117/21420)
Line Coverage: 29.56% (66520/225016)
Region Coverage: 29.03% (34280/118096)
Branch Coverage: 24.89% (17607/70726)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd_f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd/report/index.html

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

ClickBench: Total hot run time: 30.62 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.03
query3	0.25	0.05	0.05
query4	1.80	0.07	0.06
query5	0.53	0.52	0.52
query6	1.24	0.62	0.60
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.49	0.48
query10	0.53	0.54	0.54
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.60	0.61
query14	0.77	0.78	0.80
query15	0.76	0.75	0.76
query16	0.36	0.38	0.37
query17	0.97	1.02	1.03
query18	0.21	0.26	0.24
query19	1.95	1.87	1.89
query20	0.01	0.01	0.01
query21	15.49	0.53	0.55
query22	2.44	2.31	1.64
query23	17.31	0.90	0.95
query24	6.32	0.87	1.15
query25	0.36	0.10	0.04
query26	0.71	0.16	0.15
query27	0.05	0.04	0.05
query28	6.87	0.76	0.77
query29	12.72	2.35	2.24
query30	0.55	0.51	0.52
query31	2.84	0.38	0.37
query32	3.39	0.49	0.49
query33	3.12	3.04	3.07
query34	15.27	4.80	4.78
query35	4.87	4.85	4.84
query36	1.04	1.01	1.02
query37	0.06	0.05	0.04
query38	0.03	0.03	0.02
query39	0.02	0.01	0.02
query40	0.16	0.15	0.15
query41	0.07	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.63 s
Total hot run time: 30.62 s

@doris-robot
Copy link

doris-robot commented Jul 10, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f188b0de2cd0fe068acd43dcf5d487d4a92a6dfd with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

dataroaring
dataroaring approved these changes Jul 13, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 13, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 13, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 13, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit a805e0b into apache:branch-2.0 Jul 13, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@Yukang-Lian
[cherry-pick](branch-2.0) Pick "[Enhancement](merge-on-write) Support…
3898536 
… dynamic delete bitmap cache (apache#32991)" (apache#37602)

The default delete bitmap cache is set to 100MB, which can be
insufficient and cause performance issues when the amount of user data
is large. To mitigate the problem of an inadequate cache, we will take
the larger of 5% of the total memory and 100MB as the delete bitmap
cache size.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Yukang-Lian @doris-robot @dataroaring