Skip to content

[Opt](log) Add more delete bitmap log for merge-on-write table #48930

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zhannngchen merged 2 commits into from
Mar 12, 2025
Merged

[Opt](log) Add more delete bitmap log for merge-on-write table #48930

zhannngchen merged 2 commits into from
Mar 12, 2025

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 11, 2025 •
edited by zhannngchen
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 11, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1 bobhan1 force-pushed the mow-log-delete-bitmap-cardinality branch from 7b97007 to 7c0282c Compare March 11, 2025 10:43
@bobhan1 bobhan1 force-pushed the mow-log-delete-bitmap-cardinality branch from 7c0282c to 3d6f837 Compare March 11, 2025 10:52
@bobhan1 bobhan1 changed the title [Opt](log) Add delete bitmap cardinality log for merge-on-write table [Opt](log) Add more delete bitmap log for merge-on-write table Mar 11, 2025
@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 11, 2025

run buildall

@bobhan1 bobhan1 force-pushed the mow-log-delete-bitmap-cardinality branch from 14f857a to 1a9b460 Compare March 11, 2025 11:57
@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 11, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 11, 2025

TPC-H: Total hot run time: 32888 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1a9b46077563b3faff4e2e0973a1912b1cbc188b, data reload: false

------ Round 1 ----------------------------------
q1	18499	5302	5135	5135
q2	2047	308	167	167
q3	10752	1275	729	729
q4	10487	1013	544	544
q5	7490	2362	2386	2362
q6	187	163	130	130
q7	903	775	638	638
q8	9353	1315	1134	1134
q9	5496	5007	4921	4921
q10	6967	2338	1947	1947
q11	489	283	264	264
q12	355	366	210	210
q13	20663	3801	3134	3134
q14	227	225	209	209
q15	540	480	471	471
q16	648	609	596	596
q17	582	841	366	366
q18	6993	6507	6293	6293
q19	1482	971	577	577
q20	320	337	200	200
q21	2998	2153	1913	1913
q22	1008	992	948	948
Total cold run time: 108486 ms
Total hot run time: 32888 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5229	5129	5304	5129
q2	242	325	228	228
q3	2122	2674	2296	2296
q4	1378	1794	1392	1392
q5	4245	4132	4361	4132
q6	214	166	127	127
q7	2024	1876	1759	1759
q8	2650	2591	2645	2591
q9	7237	7227	7216	7216
q10	2965	3172	2675	2675
q11	576	521	487	487
q12	723	752	631	631
q13	3547	3905	3311	3311
q14	303	307	271	271
q15	545	499	509	499
q16	649	689	668	668
q17	1122	1566	1359	1359
q18	7694	7667	7534	7534
q19	868	900	939	900
q20	1981	2050	1918	1918
q21	5402	4881	4999	4881
q22	1145	1075	997	997
Total cold run time: 52861 ms
Total hot run time: 51001 ms

@doris-robot
Copy link

doris-robot commented Mar 11, 2025

TPC-DS: Total hot run time: 192008 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1a9b46077563b3faff4e2e0973a1912b1cbc188b, data reload: false

query1	1467	1000	988	988
query2	6317	1923	1905	1905
query3	10985	4382	4421	4382
query4	55352	25059	23389	23389
query5	5121	530	458	458
query6	322	181	174	174
query7	4883	509	284	284
query8	307	249	233	233
query9	5652	2625	2629	2625
query10	421	318	255	255
query11	15227	15066	14888	14888
query12	158	109	104	104
query13	1045	506	406	406
query14	10119	6462	6281	6281
query15	199	208	177	177
query16	7042	661	549	549
query17	1080	745	571	571
query18	1534	429	335	335
query19	201	188	173	173
query20	136	138	122	122
query21	212	125	113	113
query22	4490	4593	4557	4557
query23	34030	33329	33459	33329
query24	6362	2424	2426	2424
query25	459	452	416	416
query26	849	279	160	160
query27	2405	499	333	333
query28	2953	2488	2440	2440
query29	571	571	432	432
query30	270	217	189	189
query31	868	832	792	792
query32	71	64	97	64
query33	453	380	306	306
query34	778	869	504	504
query35	825	878	734	734
query36	937	1001	890	890
query37	119	100	78	78
query38	4080	4194	4275	4194
query39	1504	1447	1480	1447
query40	201	115	107	107
query41	61	51	49	49
query42	122	105	102	102
query43	500	528	478	478
query44	1333	802	789	789
query45	182	169	178	169
query46	857	1039	658	658
query47	1864	1884	1842	1842
query48	395	426	297	297
query49	709	504	440	440
query50	693	750	422	422
query51	4337	4374	4274	4274
query52	108	104	95	95
query53	236	279	201	201
query54	499	502	403	403
query55	78	79	81	79
query56	264	280	260	260
query57	1174	1178	1126	1126
query58	255	245	247	245
query59	2824	2850	2754	2754
query60	280	269	256	256
query61	122	119	117	117
query62	748	729	672	672
query63	229	194	194	194
query64	1966	1068	757	757
query65	4579	4468	4446	4446
query66	759	413	299	299
query67	16110	15556	15165	15165
query68	8220	870	498	498
query69	532	302	260	260
query70	1270	1107	1143	1107
query71	497	292	269	269
query72	5828	3596	3679	3596
query73	1186	728	360	360
query74	9047	8964	8915	8915
query75	3844	3161	2674	2674
query76	4251	1180	760	760
query77	731	374	278	278
query78	10020	10146	9287	9287
query79	2332	821	581	581
query80	632	512	432	432
query81	480	255	225	225
query82	454	126	100	100
query83	327	174	147	147
query84	280	100	74	74
query85	787	365	311	311
query86	382	306	276	276
query87	4506	4367	4374	4367
query88	3337	2236	2230	2230
query89	409	324	284	284
query90	1939	208	208	208
query91	148	141	113	113
query92	71	62	59	59
query93	1833	1012	573	573
query94	671	418	300	300
query95	344	269	265	265
query96	482	574	277	277
query97	3296	3387	3325	3325
query98	227	203	201	201
query99	1445	1369	1259	1259
Total cold run time: 301097 ms
Total hot run time: 192008 ms

@doris-robot
Copy link

doris-robot commented Mar 11, 2025

ClickBench: Total hot run time: 31.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1a9b46077563b3faff4e2e0973a1912b1cbc188b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.25	0.07	0.06
query4	1.62	0.10	0.10
query5	0.56	0.55	0.55
query6	1.19	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.59	0.52	0.52
query10	0.58	0.60	0.58
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.61	0.59
query14	2.67	2.68	2.67
query15	0.93	0.86	0.83
query16	0.38	0.39	0.40
query17	1.00	1.05	1.03
query18	0.21	0.19	0.20
query19	1.91	1.86	1.96
query20	0.01	0.02	0.01
query21	15.35	0.90	0.53
query22	0.75	1.14	0.82
query23	14.80	1.36	0.64
query24	6.80	2.07	0.95
query25	0.49	0.32	0.17
query26	0.67	0.15	0.14
query27	0.05	0.05	0.04
query28	9.22	0.81	0.43
query29	12.53	3.98	3.31
query30	0.24	0.10	0.07
query31	2.84	0.59	0.40
query32	3.23	0.55	0.47
query33	3.01	3.02	3.04
query34	15.82	5.12	4.50
query35	4.49	4.48	4.52
query36	0.65	0.49	0.50
query37	0.09	0.07	0.06
query38	0.05	0.04	0.05
query39	0.04	0.03	0.02
query40	0.16	0.14	0.13
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.45 s
Total hot run time: 31.21 s

@doris-robot
Copy link

doris-robot commented Mar 11, 2025

BE UT Coverage Report

Increment line coverage 56.36% (31/55) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 46.89% (12540/26745)
Line Coverage 36.61% (107216/292869)
Region Coverage 35.65% (54789/153687)
Branch Coverage 31.06% (27611/88896)

dataroaring
dataroaring approved these changes Mar 12, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 12, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 12, 2025

PR approved by anyone and no changes requested.

zhannngchen
zhannngchen approved these changes Mar 12, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen zhannngchen merged commit 293b809 into apache:master Mar 12, 2025
27 of 28 checks passed
bobhan1 added a commit to bobhan1/doris that referenced this pull request Mar 12, 2025
@bobhan1
[Opt](log) Add more delete bitmap log for merge-on-write table (apach…
af68770 
…e#48930)

Add more delete bitmap log for merge-on-write table
This was referenced Mar 12, 2025
Merged
Merged
dataroaring pushed a commit that referenced this pull request Mar 13, 2025
@bobhan1
branch-3.0-pick: [Opt](log) Add more delete bitmap log for merge-on-w…
0cbdf18 
…rite table (#48930) (#48966)

pick #48930
zhannngchen pushed a commit that referenced this pull request Mar 17, 2025
@bobhan1
[Fix](log) Fix some delete bitmap log (#49009)
da4294b 
### What problem does this PR solve?

1. fix wrong delete bitmap logs in
#41636 and
#48930
2. add count and cardinality log when calculating between segments
github-actions bot pushed a commit that referenced this pull request Mar 17, 2025
@bobhan1
[Fix](log) Fix some delete bitmap log (#49009)
58ba909 
### What problem does this PR solve?

1. fix wrong delete bitmap logs in
#41636 and
#48930
2. add count and cardinality log when calculating between segments
bobhan1 added a commit to bobhan1/doris that referenced this pull request Apr 15, 2025
@bobhan1
[Opt](log) Add more delete bitmap log for merge-on-write table (apach…
21c0c5f 
…e#48930)

Add more delete bitmap log for merge-on-write table
@bobhan1 bobhan1 mentioned this pull request Apr 15, 2025
Merged
yiguolei pushed a commit that referenced this pull request Apr 16, 2025
@bobhan1
branch-2.1-pick: [Opt](log) Add and fix logs for delete bitmaps (#50060)
afaab7f 
pick #48523,
#48930
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@bobhan1 @koarz
[Opt](log) Add more delete bitmap log for merge-on-write table (apach…
1785cbb 
…e#48930)

Add more delete bitmap log for merge-on-write table
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@bobhan1 @koarz
[Fix](log) Fix some delete bitmap log (apache#49009)
07f20fc 
### What problem does this PR solve?

1. fix wrong delete bitmap logs in
apache#41636 and
apache#48930
2. add count and cardinality log when calculating between segments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.5-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@bobhan1 @hello-stephen @doris-robot @zhannngchen @dataroaring @yiguolei