[Fix](cloud) Should not skip to calculate delete bitmaps in publish phase if cache miss when get delete bitmap #48867

bobhan1 · 2025-03-10T09:28:10Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-03-10T09:28:15Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

bobhan1 · 2025-03-10T10:25:50Z

run buildall

doris-robot · 2025-03-10T10:55:30Z

TPC-H: Total hot run time: 32666 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 419f3234a3e8865daf35b81350ac4855aabca137, data reload: false

------ Round 1 ----------------------------------
q1	17631	5229	5096	5096
q2	2034	300	165	165
q3	10405	1331	708	708
q4	10222	1050	564	564
q5	7540	2433	2315	2315
q6	188	167	132	132
q7	914	746	612	612
q8	9299	1291	1143	1143
q9	5036	4692	4956	4692
q10	6818	2299	1899	1899
q11	482	289	258	258
q12	345	353	217	217
q13	17765	3688	3077	3077
q14	234	237	211	211
q15	541	486	465	465
q16	638	627	579	579
q17	572	871	349	349
q18	6910	6506	6480	6480
q19	1205	944	549	549
q20	313	328	194	194
q21	2881	2167	1952	1952
q22	1031	1018	1009	1009
Total cold run time: 103004 ms
Total hot run time: 32666 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5207	5129	5149	5129
q2	231	326	233	233
q3	2185	2659	2287	2287
q4	1385	1812	1356	1356
q5	4250	4148	4167	4148
q6	206	164	125	125
q7	1883	1972	1753	1753
q8	2613	2695	2540	2540
q9	7297	7181	7101	7101
q10	3019	3195	2731	2731
q11	575	501	484	484
q12	700	765	601	601
q13	3505	3845	3246	3246
q14	276	296	293	293
q15	539	475	463	463
q16	657	683	641	641
q17	1173	1586	1377	1377
q18	7812	7675	7502	7502
q19	824	852	852	852
q20	1993	2035	1871	1871
q21	5498	5004	4919	4919
q22	1091	1039	1000	1000
Total cold run time: 52919 ms
Total hot run time: 50652 ms

doris-robot · 2025-03-10T11:07:03Z

TPC-DS: Total hot run time: 185683 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 419f3234a3e8865daf35b81350ac4855aabca137, data reload: false

query1	997	404	368	368
query2	6533	1907	1906	1906
query3	6795	214	215	214
query4	26910	23312	22947	22947
query5	4303	676	496	496
query6	318	217	179	179
query7	4599	494	294	294
query8	294	237	231	231
query9	8618	2643	2639	2639
query10	478	317	273	273
query11	15861	15223	14882	14882
query12	152	111	114	111
query13	1672	511	404	404
query14	9383	6393	6688	6393
query15	209	188	169	169
query16	7421	653	492	492
query17	1214	732	579	579
query18	1980	397	325	325
query19	189	179	157	157
query20	126	119	118	118
query21	211	122	107	107
query22	4380	4552	4439	4439
query23	34097	33058	33067	33058
query24	7699	2355	2355	2355
query25	542	445	394	394
query26	1224	275	154	154
query27	2302	478	327	327
query28	4082	2486	2404	2404
query29	734	576	420	420
query30	287	213	190	190
query31	939	860	774	774
query32	74	63	63	63
query33	547	346	299	299
query34	790	841	496	496
query35	792	827	744	744
query36	955	1007	897	897
query37	113	100	74	74
query38	4116	4155	4058	4058
query39	1484	1399	1391	1391
query40	206	112	100	100
query41	60	55	51	51
query42	124	104	100	100
query43	496	501	472	472
query44	1287	800	810	800
query45	171	161	157	157
query46	826	995	626	626
query47	1766	1838	1759	1759
query48	383	421	296	296
query49	771	514	426	426
query50	674	718	405	405
query51	4252	4151	4124	4124
query52	107	103	94	94
query53	230	252	186	186
query54	499	504	418	418
query55	83	81	83	81
query56	280	261	250	250
query57	1139	1167	1073	1073
query58	253	234	238	234
query59	2504	2673	2576	2576
query60	304	285	251	251
query61	148	122	122	122
query62	769	744	658	658
query63	225	191	182	182
query64	4317	982	656	656
query65	4460	4327	4324	4324
query66	1147	416	294	294
query67	15910	15646	15465	15465
query68	8110	881	509	509
query69	464	296	253	253
query70	1246	1072	1103	1072
query71	461	297	262	262
query72	5721	3557	3855	3557
query73	786	737	348	348
query74	8977	9254	8922	8922
query75	3840	3146	2676	2676
query76	3725	1196	770	770
query77	787	363	280	280
query78	10125	10095	9446	9446
query79	2639	824	586	586
query80	642	520	430	430
query81	490	256	219	219
query82	711	134	96	96
query83	221	169	151	151
query84	286	194	69	69
query85	766	346	309	309
query86	391	321	280	280
query87	4363	4521	4347	4347
query88	3637	2264	2266	2264
query89	395	310	288	288
query90	1813	211	214	211
query91	140	137	110	110
query92	81	62	60	60
query93	1725	1061	579	579
query94	656	405	314	314
query95	350	267	261	261
query96	488	567	280	280
query97	3332	3436	3282	3282
query98	238	205	204	204
query99	1405	1376	1247	1247
Total cold run time: 275822 ms
Total hot run time: 185683 ms

doris-robot · 2025-03-10T11:12:22Z

ClickBench: Total hot run time: 30.85 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 419f3234a3e8865daf35b81350ac4855aabca137, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.56	0.56	0.54
query6	1.19	0.73	0.71
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.58	0.53	0.50
query10	0.56	0.62	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.86	2.72	2.71
query15	0.92	0.86	0.85
query16	0.37	0.37	0.39
query17	1.06	1.02	1.00
query18	0.22	0.20	0.19
query19	1.87	1.76	2.03
query20	0.02	0.01	0.01
query21	15.37	0.90	0.53
query22	0.75	1.15	0.62
query23	15.02	1.40	0.62
query24	6.76	1.45	0.92
query25	0.51	0.32	0.15
query26	0.57	0.17	0.15
query27	0.04	0.05	0.06
query28	10.17	0.85	0.43
query29	12.68	4.01	3.35
query30	0.25	0.08	0.06
query31	2.83	0.59	0.38
query32	3.23	0.54	0.46
query33	3.01	3.01	3.05
query34	15.87	5.16	4.51
query35	4.56	4.60	4.53
query36	0.66	0.50	0.47
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 105.9 s
Total hot run time: 30.85 s

bobhan1 · 2025-03-10T11:23:56Z

run beut

hello-stephen · 2025-03-10T12:57:20Z

BE UT Coverage Report

Increment line coverage 0.00% (0/17) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	46.83% (12518/26732)
Line Coverage	36.44% (106688/292750)
Region Coverage	35.47% (54505/153645)
Branch Coverage	30.86% (27427/88870)

dataroaring

LGTM

github-actions · 2025-03-11T08:38:30Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-03-11T08:38:32Z

PR approved by anyone and no changes requested.

bobhan1 · 2025-03-11T10:24:31Z

run buildall

doris-robot · 2025-03-11T10:53:17Z

TPC-H: Total hot run time: 32697 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 569dbebc7ebfc9821e59a482071bc5bd2c568cf8, data reload: false

------ Round 1 ----------------------------------
q1	17629	5270	5142	5142
q2	2045	297	168	168
q3	10403	1324	733	733
q4	10222	1022	542	542
q5	7546	2380	2349	2349
q6	191	167	132	132
q7	909	745	613	613
q8	9296	1328	1076	1076
q9	4939	4749	4747	4747
q10	6816	2343	1928	1928
q11	471	285	255	255
q12	350	354	219	219
q13	17771	3728	3122	3122
q14	221	236	208	208
q15	520	485	493	485
q16	623	611	610	610
q17	577	876	344	344
q18	6880	6558	6394	6394
q19	1209	963	560	560
q20	317	367	210	210
q21	2849	2240	1906	1906
q22	1017	1029	954	954
Total cold run time: 102801 ms
Total hot run time: 32697 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5167	5153	5163	5153
q2	237	330	233	233
q3	2142	2648	2319	2319
q4	1475	1874	1407	1407
q5	4298	4123	4145	4123
q6	204	167	125	125
q7	1907	1930	1794	1794
q8	2640	2666	2605	2605
q9	7194	7099	7040	7040
q10	3021	3232	2769	2769
q11	577	505	492	492
q12	669	794	626	626
q13	3577	3957	3246	3246
q14	285	285	261	261
q15	533	468	480	468
q16	641	678	694	678
q17	1140	1521	1424	1424
q18	7745	7617	7605	7605
q19	835	852	972	852
q20	1965	2040	1884	1884
q21	5638	4941	4851	4851
q22	1123	1043	1039	1039
Total cold run time: 53013 ms
Total hot run time: 50994 ms

doris-robot · 2025-03-11T11:04:54Z

TPC-DS: Total hot run time: 186543 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 569dbebc7ebfc9821e59a482071bc5bd2c568cf8, data reload: false

query1	1007	390	389	389
query2	6540	1919	1891	1891
query3	6806	227	223	223
query4	26913	23343	23811	23343
query5	4760	654	476	476
query6	300	184	175	175
query7	4616	504	304	304
query8	283	244	224	224
query9	8585	2649	2644	2644
query10	485	324	240	240
query11	15895	15161	14993	14993
query12	157	106	106	106
query13	1642	513	379	379
query14	10076	6418	7026	6418
query15	209	204	173	173
query16	7491	597	467	467
query17	1159	734	577	577
query18	2011	405	335	335
query19	188	188	176	176
query20	119	113	124	113
query21	209	123	109	109
query22	4124	4204	4242	4204
query23	33966	33191	33142	33142
query24	7643	2429	2394	2394
query25	557	480	410	410
query26	1243	277	161	161
query27	2155	489	333	333
query28	3937	2426	2466	2426
query29	707	547	465	465
query30	286	222	189	189
query31	953	873	802	802
query32	74	65	64	64
query33	559	361	297	297
query34	782	849	505	505
query35	785	823	760	760
query36	953	993	923	923
query37	114	96	73	73
query38	4269	4187	4049	4049
query39	1544	1580	1409	1409
query40	205	124	107	107
query41	54	55	52	52
query42	122	102	106	102
query43	492	504	477	477
query44	1259	784	782	782
query45	176	170	166	166
query46	822	1014	630	630
query47	1744	1806	1722	1722
query48	376	412	301	301
query49	795	523	427	427
query50	677	726	401	401
query51	4183	4180	4101	4101
query52	116	103	101	101
query53	229	260	194	194
query54	496	482	402	402
query55	79	79	80	79
query56	304	272	246	246
query57	1133	1145	1071	1071
query58	252	240	248	240
query59	2490	2639	2481	2481
query60	283	293	271	271
query61	134	133	141	133
query62	815	738	678	678
query63	228	191	188	188
query64	4235	1005	673	673
query65	4567	4337	4370	4337
query66	1103	408	300	300
query67	15980	15453	15390	15390
query68	8464	868	549	549
query69	463	312	264	264
query70	1224	1143	1122	1122
query71	467	295	259	259
query72	5712	3577	3808	3577
query73	809	733	351	351
query74	9051	9232	9002	9002
query75	3938	3178	2716	2716
query76	3697	1178	750	750
query77	775	360	284	284
query78	10113	10217	9285	9285
query79	2086	880	595	595
query80	708	524	451	451
query81	476	266	227	227
query82	663	132	106	106
query83	205	171	156	156
query84	292	105	75	75
query85	775	366	317	317
query86	340	310	292	292
query87	4578	4649	4512	4512
query88	2944	2288	2323	2288
query89	399	378	295	295
query90	1943	231	228	228
query91	143	137	108	108
query92	74	64	61	61
query93	1186	1060	590	590
query94	670	414	312	312
query95	363	278	269	269
query96	480	563	284	284
query97	3329	3398	3280	3280
query98	226	213	210	210
query99	1498	1397	1315	1315
Total cold run time: 275542 ms
Total hot run time: 186543 ms

doris-robot · 2025-03-11T11:10:14Z

ClickBench: Total hot run time: 31.05 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 569dbebc7ebfc9821e59a482071bc5bd2c568cf8, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.11
query5	0.55	0.57	0.57
query6	1.18	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.53	0.53
query10	0.59	0.63	0.59
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.60	0.59
query14	2.66	2.70	2.67
query15	0.93	0.85	0.84
query16	0.36	0.39	0.39
query17	1.06	1.08	0.99
query18	0.22	0.20	0.20
query19	1.92	1.97	1.86
query20	0.01	0.01	0.02
query21	15.35	0.90	0.54
query22	0.76	1.11	0.73
query23	14.90	1.40	0.61
query24	7.09	0.90	1.43
query25	0.52	0.30	0.11
query26	0.53	0.16	0.13
query27	0.05	0.06	0.05
query28	9.98	0.91	0.44
query29	12.54	4.05	3.39
query30	0.25	0.09	0.06
query31	2.83	0.59	0.39
query32	3.23	0.56	0.46
query33	3.00	3.00	3.12
query34	15.62	5.15	4.49
query35	4.50	4.50	4.55
query36	0.66	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.32 s
Total hot run time: 31.05 s

hello-stephen · 2025-03-11T12:12:59Z

BE UT Coverage Report

Increment line coverage 0.00% (0/5) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	46.88% (12537/26744)
Line Coverage	36.60% (107174/292828)
Region Coverage	35.65% (54790/153682)
Branch Coverage	31.05% (27599/88894)

dataroaring

LGTM

github-actions · 2025-03-12T02:05:18Z

PR approved by at least one committer and no changes requested.

zhannngchen

LGTM

…hase if cache miss when get delete bitmap (#48867) fix a duplicate key issue for mow table should not skip to calculate delete bitmaps in publish phase if cache miss when get delete bitmap

…in publish phase if cache miss when get delete bitmap #48867 (#48962) Cherry-picked from #48867 Co-authored-by: bobhan1 <baohan@selectdb.com>

…hase if cache miss when get delete bitmap (apache#48867) fix a duplicate key issue for mow table should not skip to calculate delete bitmaps in publish phase if cache miss when get delete bitmap

bobhan1 marked this pull request as ready for review March 10, 2025 09:35

bobhan1 requested review from dataroaring, gavinchou and w41ter as code owners March 10, 2025 09:35

dataroaring added dev/3.0.x p0_w labels Mar 11, 2025

dataroaring previously approved these changes Mar 11, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 11, 2025

github-actions bot added the reviewed label Mar 11, 2025

bobhan1 added 3 commits March 11, 2025 17:08

add re-produce case

3090812

fix and correct case

cb5a559

fix

569dbeb

bobhan1 dismissed dataroaring’s stale review via 569dbeb March 11, 2025 09:21

bobhan1 force-pushed the fix-cache-miss-when-skip-calc-in-publish branch from 419f323 to 569dbeb Compare March 11, 2025 09:21

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 11, 2025

dataroaring approved these changes Mar 12, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2025

zhannngchen approved these changes Mar 12, 2025

View reviewed changes

zhannngchen merged commit 67db264 into apache:master Mar 12, 2025
25 of 27 checks passed

github-actions bot mentioned this pull request Mar 12, 2025

branch-3.0: [Fix](cloud) Should not skip to calculate delete bitmaps in publish phase if cache miss when get delete bitmap #48867 #48962

Merged

dataroaring added dev/3.0.5-merged and removed dev/3.0.x labels Mar 12, 2025

gavinchou mentioned this pull request Apr 23, 2025

Release Notes 3.0.5 #50333

Closed

[Fix](cloud) Should not skip to calculate delete bitmaps in publish phase if cache miss when get delete bitmap #48867

[Fix](cloud) Should not skip to calculate delete bitmaps in publish phase if cache miss when get delete bitmap #48867

Uh oh!

Conversation

bobhan1 commented Mar 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Mar 10, 2025

Uh oh!

bobhan1 commented Mar 10, 2025

Uh oh!

doris-robot commented Mar 10, 2025

Uh oh!

doris-robot commented Mar 10, 2025

Uh oh!

doris-robot commented Mar 10, 2025

Uh oh!

bobhan1 commented Mar 10, 2025

Uh oh!

hello-stephen commented Mar 10, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 11, 2025

Uh oh!

github-actions bot commented Mar 11, 2025

Uh oh!

bobhan1 commented Mar 11, 2025

Uh oh!

doris-robot commented Mar 11, 2025

Uh oh!

doris-robot commented Mar 11, 2025

Uh oh!

doris-robot commented Mar 11, 2025

Uh oh!

hello-stephen commented Mar 11, 2025

BE UT Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 12, 2025

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

bobhan1 commented Mar 10, 2025 •

edited

Loading