Skip to content

[fix](cloud-mow)Full compaction no need to sync delete bitmap #43337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 2 commits into from
Nov 8, 2024
Merged

[fix](cloud-mow)Full compaction no need to sync delete bitmap #43337

dataroaring merged 2 commits into from
Nov 8, 2024

Conversation

@hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Nov 6, 2024

sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it.

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Nov 6, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Nov 6, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/cloud/cloud_meta_mgr.cpp
}

Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data) {
Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
Copy link
Contributor

@github-actions github-actions bot Nov 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'sync_tablet_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]

Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
                     ^
Additional context

be/src/cloud/cloud_meta_mgr.cpp:386: 181 lines including whitespace and comments (threshold 80)

Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
                     ^

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Nov 6, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TeamCity be ut coverage result:
Function Coverage: 37.89% (9847/25989)
Line Coverage: 29.04% (81868/281904)
Region Coverage: 28.27% (42177/149181)
Branch Coverage: 24.85% (21399/86106)
Coverage Report: http://coverage.selectdb-in.cc/coverage/35b58b30654b8f4a0b87668407395a983c99a68c_35b58b30654b8f4a0b87668407395a983c99a68c/report/index.html

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TPC-H: Total hot run time: 41542 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35b58b30654b8f4a0b87668407395a983c99a68c, data reload: false

------ Round 1 ----------------------------------
q1	17572	7491	7324	7324
q2	2048	176	169	169
q3	10833	1102	1154	1102
q4	10563	878	896	878
q5	7738	3097	3063	3063
q6	244	147	150	147
q7	1036	612	614	612
q8	9396	2012	2085	2012
q9	6631	6491	6508	6491
q10	7069	2443	2452	2443
q11	463	267	262	262
q12	404	219	208	208
q13	17779	3063	3009	3009
q14	243	215	212	212
q15	569	532	505	505
q16	662	611	577	577
q17	983	632	513	513
q18	7415	6708	6636	6636
q19	1329	985	1019	985
q20	473	184	188	184
q21	3988	3225	3221	3221
q22	1108	1038	989	989
Total cold run time: 108546 ms
Total hot run time: 41542 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7301	7302	7281	7281
q2	342	258	255	255
q3	3098	2965	2983	2965
q4	2119	1826	1835	1826
q5	5759	5788	5886	5788
q6	227	148	144	144
q7	2234	1812	1800	1800
q8	3507	3564	3508	3508
q9	9020	8940	8905	8905
q10	3590	3588	3555	3555
q11	612	506	519	506
q12	813	629	589	589
q13	9269	3238	3262	3238
q14	301	270	289	270
q15	610	566	559	559
q16	683	657	667	657
q17	1844	1655	1601	1601
q18	8440	7905	7679	7679
q19	1747	1686	1644	1644
q20	2131	1869	1993	1869
q21	5626	5635	5535	5535
q22	1172	1056	1045	1045
Total cold run time: 70445 ms
Total hot run time: 61219 ms

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TPC-DS: Total hot run time: 196012 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35b58b30654b8f4a0b87668407395a983c99a68c, data reload: false

query1	2272	2088	2077	2077
query2	6244	2063	2097	2063
query3	15731	11612	238	238
query4	33432	23626	23744	23626
query5	3421	468	453	453
query6	273	194	187	187
query7	3985	297	297	297
query8	290	237	234	234
query9	9287	2728	2724	2724
query10	451	263	248	248
query11	17981	15409	15313	15313
query12	153	103	101	101
query13	1611	433	417	417
query14	8584	7862	7855	7855
query15	248	180	186	180
query16	7368	486	496	486
query17	1599	600	625	600
query18	1630	620	635	620
query19	394	188	186	186
query20	125	125	118	118
query21	209	105	103	103
query22	4748	4537	4408	4408
query23	34799	34223	34228	34223
query24	11265	3343	3389	3343
query25	582	395	402	395
query26	748	175	174	174
query27	1946	279	284	279
query28	6400	2426	2438	2426
query29	680	434	432	432
query30	397	309	309	309
query31	1032	777	811	777
query32	95	56	58	56
query33	710	288	276	276
query34	896	507	515	507
query35	887	764	731	731
query36	1085	958	956	956
query37	120	76	69	69
query38	4501	4331	4469	4331
query39	1490	1407	1449	1407
query40	204	99	100	99
query41	49	47	48	47
query42	109	95	99	95
query43	552	502	501	501
query44	1196	828	815	815
query45	183	169	167	167
query46	1156	701	677	677
query47	1923	1848	1851	1848
query48	414	321	322	321
query49	902	396	417	396
query50	813	395	407	395
query51	7328	7207	7172	7172
query52	102	90	86	86
query53	249	176	172	172
query54	1281	403	410	403
query55	79	73	76	73
query56	255	227	246	227
query57	1238	1187	1131	1131
query58	224	205	196	196
query59	3316	2982	2915	2915
query60	275	241	243	241
query61	111	111	108	108
query62	875	669	765	669
query63	215	193	185	185
query64	3952	659	645	645
query65	3300	3206	3236	3206
query66	835	300	304	300
query67	16016	15604	15580	15580
query68	4458	583	578	578
query69	423	255	259	255
query70	1248	1168	1056	1056
query71	332	255	255	255
query72	6293	4100	3918	3918
query73	756	356	359	356
query74	10266	9025	8986	8986
query75	3403	2683	2747	2683
query76	2903	1143	1168	1143
query77	373	272	269	269
query78	10225	9385	9394	9385
query79	1769	602	590	590
query80	1232	426	429	426
query81	570	242	244	242
query82	990	116	115	115
query83	247	161	158	158
query84	235	81	71	71
query85	1333	320	302	302
query86	426	307	308	307
query87	4909	4681	4659	4659
query88	3354	2199	2160	2160
query89	418	294	292	292
query90	1951	186	187	186
query91	139	105	105	105
query92	59	50	50	50
query93	2142	538	539	538
query94	928	295	300	295
query95	351	247	243	243
query96	611	275	282	275
query97	2911	2710	2718	2710
query98	218	203	203	203
query99	1597	1340	1306	1306
Total cold run time: 301891 ms
Total hot run time: 196012 ms

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

ClickBench: Total hot run time: 32.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35b58b30654b8f4a0b87668407395a983c99a68c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.64	0.10	0.11
query5	0.40	0.39	0.42
query6	1.18	0.65	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.51	0.48
query10	0.57	0.56	0.55
query11	0.15	0.11	0.10
query12	0.14	0.10	0.11
query13	0.61	0.60	0.60
query14	2.71	2.75	2.81
query15	0.91	0.84	0.83
query16	0.40	0.38	0.39
query17	0.98	0.99	1.07
query18	0.20	0.19	0.20
query19	1.84	1.87	1.87
query20	0.01	0.01	0.03
query21	15.37	0.63	0.62
query22	2.84	2.09	1.80
query23	16.89	1.05	0.95
query24	2.63	1.00	0.71
query25	0.22	0.16	0.11
query26	0.37	0.15	0.14
query27	0.06	0.05	0.04
query28	11.34	1.10	1.08
query29	12.54	3.25	3.21
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.47	0.46
query33	3.01	3.00	3.01
query34	17.14	4.57	4.50
query35	4.51	4.57	4.53
query36	0.65	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.14 s
Total hot run time: 32.55 s

dataroaring
dataroaring approved these changes Nov 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 1216490 into apache:master Nov 8, 2024
@hust-hhb hust-hhb mentioned this pull request Nov 11, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 11, 2024
@hust-hhb
[fix](cloud-mow)Full compaction no need to sync delete bitmap (#43613)
c7b707f 
sync delete bitmap may cost too much time,which may lead to full
compaction fail, but full compaction no need to sync it.
pick pr:#43337

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hust-hhb @doris-robot @gavinchou @dataroaring