Skip to content

[bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock backoff sleep time #52044

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhannngchen merged 1 commit into from
Jun 25, 2025
Merged

[bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock backoff sleep time #52044

zhannngchen merged 1 commit into from
Jun 25, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Jun 20, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

dataroaring
dataroaring previously approved these changes Jun 20, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 20, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 20, 2025

PR approved by anyone and no changes requested.

zhannngchen
zhannngchen previously approved these changes Jun 20, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bobhan1 bobhan1 dismissed stale reviews from zhannngchen and dataroaring via 03d405a June 23, 2025 01:58
@bobhan1 bobhan1 force-pushed the recored-be-get-mow-lock-backoff-sleep-time branch from 626efeb to 03d405a Compare June 23, 2025 01:58
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 23, 2025
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jun 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

TPC-H: Total hot run time: 33678 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 03d405a2422860d3cba05d9673e61e7aba7153c8, data reload: false

------ Round 1 ----------------------------------
q1	17601	5183	5068	5068
q2	1954	302	190	190
q3	10349	1258	748	748
q4	10220	1006	522	522
q5	7535	2311	2336	2311
q6	176	165	135	135
q7	899	734	610	610
q8	9325	1210	1041	1041
q9	6748	5056	5028	5028
q10	6889	2359	1956	1956
q11	474	284	269	269
q12	347	348	214	214
q13	17774	3636	3028	3028
q14	237	230	223	223
q15	569	490	491	490
q16	429	434	373	373
q17	588	867	354	354
q18	7609	7097	7147	7097
q19	2342	953	530	530
q20	320	323	222	222
q21	3652	2501	2294	2294
q22	1053	1019	975	975
Total cold run time: 107090 ms
Total hot run time: 33678 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5212	5088	5083	5083
q2	243	320	229	229
q3	2182	2636	2272	2272
q4	1385	1830	1353	1353
q5	4327	4088	4290	4088
q6	212	168	129	129
q7	2040	1918	1781	1781
q8	2534	2659	2499	2499
q9	7104	7144	7129	7129
q10	3050	3317	2828	2828
q11	569	501	491	491
q12	691	768	585	585
q13	3478	3909	3181	3181
q14	279	289	275	275
q15	525	473	480	473
q16	441	483	438	438
q17	1136	1472	1375	1375
q18	7485	7192	7037	7037
q19	758	769	807	769
q20	1932	1985	1842	1842
q21	4669	4294	4326	4294
q22	1047	1020	975	975
Total cold run time: 51299 ms
Total hot run time: 49126 ms

This was referenced Jun 23, 2025
Merged
Merged
@doris-robot
Copy link

doris-robot commented Jun 23, 2025

TPC-DS: Total hot run time: 185525 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 03d405a2422860d3cba05d9673e61e7aba7153c8, data reload: false

query1	986	386	396	386
query2	6520	1811	1830	1811
query3	6745	220	220	220
query4	26406	23453	23202	23202
query5	4416	627	483	483
query6	313	218	200	200
query7	4631	513	289	289
query8	271	229	214	214
query9	8611	2637	2649	2637
query10	472	321	282	282
query11	15277	15245	14711	14711
query12	173	116	108	108
query13	1659	539	421	421
query14	9496	6130	6113	6113
query15	207	202	170	170
query16	7674	625	471	471
query17	1200	737	590	590
query18	2035	408	301	301
query19	205	196	168	168
query20	122	118	115	115
query21	209	137	109	109
query22	4058	4090	4038	4038
query23	33826	33173	33257	33173
query24	8465	2339	2352	2339
query25	547	456	387	387
query26	1211	265	144	144
query27	2721	496	345	345
query28	4268	2108	2085	2085
query29	737	545	435	435
query30	283	218	191	191
query31	934	834	737	737
query32	72	66	64	64
query33	572	373	302	302
query34	781	851	537	537
query35	794	817	717	717
query36	932	987	874	874
query37	130	96	84	84
query38	4031	4114	4003	4003
query39	1469	1420	1478	1420
query40	207	120	108	108
query41	63	58	59	58
query42	124	111	115	111
query43	495	509	476	476
query44	1302	836	841	836
query45	182	168	163	163
query46	840	1016	625	625
query47	1749	1833	1741	1741
query48	385	425	308	308
query49	724	490	382	382
query50	622	687	397	397
query51	4131	4143	4087	4087
query52	108	109	97	97
query53	221	268	189	189
query54	566	565	511	511
query55	88	84	84	84
query56	319	290	282	282
query57	1189	1208	1109	1109
query58	270	262	262	262
query59	2527	2580	2522	2522
query60	345	327	319	319
query61	125	121	122	121
query62	805	730	653	653
query63	226	185	184	184
query64	4280	992	665	665
query65	4311	4164	4146	4146
query66	1064	409	322	322
query67	15682	15643	15638	15638
query68	8370	884	533	533
query69	475	301	272	272
query70	1209	1105	1067	1067
query71	460	328	319	319
query72	5211	4858	4637	4637
query73	688	577	356	356
query74	9137	8791	8917	8791
query75	3784	3181	2700	2700
query76	3610	1181	741	741
query77	786	376	292	292
query78	10022	10256	9335	9335
query79	1557	824	570	570
query80	643	519	434	434
query81	487	255	229	229
query82	433	127	95	95
query83	272	254	245	245
query84	273	107	85	85
query85	779	353	311	311
query86	338	296	289	289
query87	4338	4468	4330	4330
query88	2917	2301	2300	2300
query89	380	308	294	294
query90	1915	211	216	211
query91	142	140	113	113
query92	77	62	62	62
query93	1053	931	585	585
query94	672	425	308	308
query95	386	300	279	279
query96	488	565	279	279
query97	2727	2752	2699	2699
query98	230	206	206	206
query99	1460	1388	1286	1286
Total cold run time: 271728 ms
Total hot run time: 185525 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2025

ClickBench: Total hot run time: 29.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 03d405a2422860d3cba05d9673e61e7aba7153c8, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.08	0.06
query4	1.59	0.10	0.10
query5	0.41	0.43	0.40
query6	1.16	0.64	0.67
query7	0.03	0.01	0.02
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.55	0.58	0.56
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.63	0.61	0.62
query14	0.80	0.81	0.81
query15	0.91	0.87	0.90
query16	0.40	0.38	0.39
query17	1.04	1.07	1.10
query18	0.23	0.21	0.21
query19	2.02	1.87	1.84
query20	0.01	0.01	0.01
query21	15.42	0.91	0.55
query22	0.76	1.13	0.84
query23	14.74	1.42	0.64
query24	7.24	1.11	0.34
query25	0.41	0.20	0.14
query26	0.69	0.17	0.15
query27	0.07	0.05	0.05
query28	8.97	0.94	0.44
query29	12.56	4.05	3.35
query30	0.26	0.10	0.06
query31	2.82	0.60	0.40
query32	3.24	0.55	0.47
query33	3.09	3.02	3.14
query34	16.15	5.40	4.82
query35	4.81	4.83	4.79
query36	0.69	0.52	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.18	0.14	0.15
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.48 s
Total hot run time: 29.27 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.55% (15099/26702)
Line Coverage 45.32% (135253/298427)
Region Coverage 44.42% (67980/153039)
Branch Coverage 38.99% (34898/89502)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 60.06% (15778/26269)
Line Coverage 49.25% (146888/298226)
Region Coverage 46.45% (83625/180013)
Branch Coverage 39.83% (40737/102270)

1 similar comment
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 60.06% (15778/26269)
Line Coverage 49.25% (146888/298226)
Region Coverage 46.45% (83625/180013)
Branch Coverage 39.83% (40737/102270)

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jun 23, 2025

run p0

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jun 23, 2025

run cloud_p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 23, 2025

BE Regression && UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 62.68% (16465/26269)
Line Coverage 52.11% (155398/298226)
Region Coverage 49.62% (89331/180013)
Branch Coverage 42.99% (43967/102270)

dataroaring
dataroaring approved these changes Jun 23, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jun 23, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 23, 2025
koarz
koarz approved these changes Jun 25, 2025
View reviewed changes
Copy link
Contributor

@koarz koarz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

zhannngchen
zhannngchen approved these changes Jun 25, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen zhannngchen merged commit 2bae60a into apache:master Jun 25, 2025
30 of 33 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 25, 2025
@bobhan1
[bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock …
4dd91d4 
…backoff sleep time (#52044)
@github-actions github-actions bot mentioned this pull request Jun 25, 2025
Merged
bobhan1 added a commit to bobhan1/doris that referenced this pull request Jun 25, 2025
@bobhan1
[bvar](cloud-mow) Add bvar for mow compaction get delete bitmap lock …
742713d 
…backoff sleep time (apache#52044)
@bobhan1 bobhan1 mentioned this pull request Jun 25, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 26, 2025
@github-actions @bobhan1
branch-3.1: [bvar](cloud-mow) Add bvar for mow compaction get delete …
3a1b09e 
…bitmap lock backoff sleep time #52044 (#52289)

Cherry-picked from #52044

Co-authored-by: bobhan1 <baohan@selectdb.com>
dataroaring pushed a commit that referenced this pull request Jun 27, 2025
@bobhan1
branch-3.0: [bvar](cloud-mow) Add bvar for mow compaction get delete …
a425e42 
…bitmap lock backoff sleep time (#52044) (#52297)

pick #52044
koarz pushed a commit to koarz/doris that referenced this pull request Jul 3, 2025
@bobhan1 @koarz
branch-3.0: [bvar](cloud-mow) Add bvar for mow compaction get delete …
767394f 
…bitmap lock backoff sleep time (apache#52044) (apache#52297)

pick apache#52044
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

+1 more reviewer

@koarz koarz koarz approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@bobhan1 @Thearas @doris-robot @hello-stephen @zhannngchen @koarz @dataroaring @morrySnow