Skip to content

[test](cloud-mow)Add update big delete bitmap case #47097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
Feb 11, 2025
Merged

[test](cloud-mow)Add update big delete bitmap case #47097

dataroaring merged 3 commits into from
Feb 11, 2025

Conversation

@hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Jan 16, 2025 •
edited
Loading

This case is used to test the scenario of updating a large number of delete bitmap when doing compaction, if delete bitmap size over the fdb limit(10MB), updating delete bitmap will fail, pr #46969 fix this problem, before the fix, this case will fail with error msg "Transaction exceeds byte limit" on metaservice.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 16, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 16, 2025

run buildall

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Jan 16, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

TPC-H: Total hot run time: 32158 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 36b9d92153350a528845a5ef4cf0d2e27f49adc4, data reload: false

------ Round 1 ----------------------------------
q1	17593	5516	5332	5332
q2	2046	297	190	190
q3	10453	1236	722	722
q4	10213	965	530	530
q5	7526	2367	2163	2163
q6	192	165	132	132
q7	903	747	600	600
q8	9241	1377	1182	1182
q9	5253	5072	4845	4845
q10	6900	2338	1889	1889
q11	515	279	256	256
q12	342	363	218	218
q13	18071	3731	3062	3062
q14	226	240	210	210
q15	512	483	484	483
q16	626	634	587	587
q17	559	856	323	323
q18	6880	6592	6413	6413
q19	2332	950	535	535
q20	310	315	195	195
q21	2980	2181	1987	1987
q22	370	341	304	304
Total cold run time: 104043 ms
Total hot run time: 32158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5631	5487	5703	5487
q2	240	332	240	240
q3	2281	2646	2322	2322
q4	1425	1805	1365	1365
q5	4315	4738	4705	4705
q6	175	184	128	128
q7	2024	1947	1935	1935
q8	2672	2831	2720	2720
q9	7297	7244	7371	7244
q10	2962	3327	2742	2742
q11	565	503	480	480
q12	637	762	608	608
q13	3552	3924	3252	3252
q14	300	324	291	291
q15	516	489	473	473
q16	644	700	662	662
q17	1239	1743	1263	1263
q18	7676	7470	7499	7470
q19	777	1133	1094	1094
q20	2004	2052	1897	1897
q21	5739	5156	5077	5077
q22	611	637	566	566
Total cold run time: 53282 ms
Total hot run time: 52021 ms

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

TeamCity be ut coverage result:
Function Coverage: 41.53% (10822/26059)
Line Coverage: 31.92% (91431/286462)
Region Coverage: 31.08% (46861/150794)
Branch Coverage: 27.16% (23728/87354)
Coverage Report: http://coverage.selectdb-in.cc/coverage/36b9d92153350a528845a5ef4cf0d2e27f49adc4_36b9d92153350a528845a5ef4cf0d2e27f49adc4/report/index.html

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

TPC-DS: Total hot run time: 194344 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 36b9d92153350a528845a5ef4cf0d2e27f49adc4, data reload: false

query1	1294	962	971	962
query2	6132	2053	2014	2014
query3	10984	4476	4350	4350
query4	60948	32059	23458	23458
query5	5190	617	477	477
query6	428	203	191	191
query7	5524	523	299	299
query8	329	251	234	234
query9	8242	2676	2668	2668
query10	454	313	244	244
query11	15813	15167	15510	15167
query12	152	112	107	107
query13	1422	575	431	431
query14	11739	6645	7421	6645
query15	251	200	186	186
query16	7229	628	464	464
query17	1144	716	587	587
query18	1832	423	313	313
query19	202	183	155	155
query20	114	110	113	110
query21	211	134	103	103
query22	4482	4915	4549	4549
query23	33883	33198	33453	33198
query24	5664	2393	2330	2330
query25	465	463	406	406
query26	655	268	167	167
query27	1754	476	341	341
query28	4019	2482	2468	2468
query29	550	573	437	437
query30	210	196	160	160
query31	1009	891	839	839
query32	65	57	54	54
query33	443	359	294	294
query34	741	867	526	526
query35	825	850	778	778
query36	1025	1064	936	936
query37	115	103	73	73
query38	4272	4321	4268	4268
query39	1510	1450	1507	1450
query40	212	111	105	105
query41	54	56	82	56
query42	128	108	107	107
query43	527	533	501	501
query44	1365	858	843	843
query45	186	179	174	174
query46	886	1069	671	671
query47	1933	1888	1885	1885
query48	404	428	335	335
query49	720	504	408	408
query50	667	668	403	403
query51	7055	7034	6899	6899
query52	111	99	94	94
query53	227	255	185	185
query54	486	517	420	420
query55	89	83	85	83
query56	267	265	269	265
query57	1250	1203	1075	1075
query58	234	231	224	224
query59	2909	3060	2876	2876
query60	269	272	254	254
query61	112	116	115	115
query62	716	707	638	638
query63	219	186	190	186
query64	1306	995	710	710
query65	3276	3159	3170	3159
query66	695	411	292	292
query67	16320	15525	15370	15370
query68	4119	829	535	535
query69	491	295	259	259
query70	1173	1151	1154	1151
query71	411	278	264	264
query72	6413	3840	3830	3830
query73	707	756	367	367
query74	10056	8977	8876	8876
query75	3236	3151	2694	2694
query76	3415	1174	761	761
query77	485	388	279	279
query78	10042	9946	9354	9354
query79	3137	833	591	591
query80	1296	535	444	444
query81	547	278	236	236
query82	570	148	122	122
query83	261	168	149	149
query84	294	99	71	71
query85	763	337	367	337
query86	408	324	306	306
query87	4537	4558	4432	4432
query88	4805	2198	2171	2171
query89	399	332	293	293
query90	1594	189	194	189
query91	137	138	116	116
query92	68	56	51	51
query93	2834	898	526	526
query94	770	410	303	303
query95	328	264	256	256
query96	493	617	357	357
query97	2875	2909	2783	2783
query98	223	195	192	192
query99	1337	1366	1277	1277
Total cold run time: 312168 ms
Total hot run time: 194344 ms

@doris-robot
Copy link

doris-robot commented Jan 16, 2025

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 36b9d92153350a528845a5ef4cf0d2e27f49adc4, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.06
query4	1.60	0.10	0.10
query5	0.41	0.41	0.40
query6	1.16	0.66	0.65
query7	0.03	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.50	0.49
query10	0.55	0.57	0.56
query11	0.14	0.10	0.11
query12	0.14	0.12	0.11
query13	0.61	0.60	0.59
query14	2.73	2.86	2.74
query15	0.91	0.83	0.82
query16	0.38	0.37	0.39
query17	1.01	1.07	1.05
query18	0.23	0.21	0.21
query19	1.96	1.87	1.98
query20	0.01	0.01	0.01
query21	15.36	0.93	0.60
query22	0.75	0.73	0.57
query23	15.45	1.45	0.53
query24	2.60	2.09	1.29
query25	0.19	0.06	0.16
query26	0.27	0.15	0.15
query27	0.07	0.04	0.05
query28	14.35	0.94	0.43
query29	12.57	3.96	3.33
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.22	0.55	0.46
query33	3.00	2.98	3.08
query34	16.46	5.18	4.48
query35	4.54	4.47	4.50
query36	0.66	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.9 s
Total hot run time: 30.86 s

@dataroaring dataroaring reopened this Jan 27, 2025
zhannngchen
zhannngchen approved these changes Feb 11, 2025
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 11, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Feb 11, 2025

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Feb 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring changed the title [improve](cloud-mow)Add update big delete bitmap case [test](cloud-mow)Add update big delete bitmap case Feb 11, 2025
@dataroaring dataroaring merged commit cbcebb1 into apache:master Feb 11, 2025
34 of 38 checks passed
github-actions bot pushed a commit that referenced this pull request Feb 11, 2025
@hust-hhb
[test](cloud-mow)Add update big delete bitmap case (#47097)
7933e5f 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr #46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
@github-actions github-actions bot mentioned this pull request Feb 11, 2025
Merged
dataroaring pushed a commit that referenced this pull request Feb 12, 2025
@github-actions @hust-hhb
branch-3.0: [test](cloud-mow)Add update big delete bitmap case #47097 (…
ef7a2c1 
…#47767)

Cherry-picked from #47097

Co-authored-by: huanghaibin <huanghaibin@selectdb.com>
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@hust-hhb @lzyy2024
[test](cloud-mow)Add update big delete bitmap case (apache#47097)
4ede535 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@hust-hhb @koarz
[test](cloud-mow)Add update big delete bitmap case (apache#47097)
1b065e2 
This case is used to test the scenario of updating a large number of
delete bitmap when doing compaction, if delete bitmap size over the fdb
limit(10MB), updating delete bitmap will fail, pr apache#46969 fix this
problem, before the fix, this case will fail with error msg "Transaction
exceeds byte limit" on metaservice.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhannngchen zhannngchen zhannngchen approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hust-hhb @hello-stephen @doris-robot @zhannngchen @dataroaring