Skip to content

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Feb 19, 2025

What problem does this PR solve?

Problem Summary:

considering the following situation:

  1. heavy sc start on BE and change tablet's state to NOT_READY
  2. load1 skip to calculate delete bitmaps in flush phase because the tablet's state is NOT_READY
  3. heavy sc finished and change tablet's state to RUNNING
  4. load1 calculate delete bitmaps in publish phase for converted historical rowsets

If load1 write multi segments, this will cause duplicate key because it forget to calculate delete bitmaps between segments in any phase.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Feb 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Feb 19, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31769 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 19f6ceb9d63959e9c70654b6f278222c71d36511, data reload: false

------ Round 1 ----------------------------------
q1	17616	5487	5087	5087
q2	2061	308	167	167
q3	10412	1261	719	719
q4	10222	1057	581	581
q5	7788	2491	2328	2328
q6	212	170	134	134
q7	921	1404	600	600
q8	9294	1323	1159	1159
q9	4929	4785	4925	4785
q10	6816	2314	1902	1902
q11	488	277	256	256
q12	344	355	230	230
q13	17760	3665	3043	3043
q14	224	219	218	218
q15	509	475	459	459
q16	635	609	580	580
q17	578	878	340	340
q18	6677	6189	6120	6120
q19	1210	961	534	534
q20	310	330	193	193
q21	2735	2279	2035	2035
q22	368	326	299	299
Total cold run time: 102109 ms
Total hot run time: 31769 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5114	5155	5169	5155
q2	230	342	234	234
q3	2164	2677	2313	2313
q4	1398	1851	1389	1389
q5	4246	4198	4183	4183
q6	216	174	132	132
q7	1874	1805	1802	1802
q8	2620	2685	2561	2561
q9	7290	7154	7216	7154
q10	3066	3230	2743	2743
q11	635	506	503	503
q12	722	765	616	616
q13	3457	3835	3206	3206
q14	287	302	300	300
q15	531	479	462	462
q16	633	671	622	622
q17	1129	1570	1357	1357
q18	7581	7305	7340	7305
q19	759	782	930	782
q20	1949	2037	1884	1884
q21	5385	5028	4831	4831
q22	637	573	569	569
Total cold run time: 51923 ms
Total hot run time: 50103 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184031 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 19f6ceb9d63959e9c70654b6f278222c71d36511, data reload: false

query1	985	398	381	381
query2	6540	1875	1842	1842
query3	6797	212	209	209
query4	26370	23572	23331	23331
query5	4357	698	524	524
query6	290	200	197	197
query7	4606	501	310	310
query8	298	240	243	240
query9	8593	2554	2539	2539
query10	470	319	257	257
query11	15552	14995	14899	14899
query12	162	105	107	105
query13	1646	513	396	396
query14	8619	6154	6159	6154
query15	202	194	179	179
query16	7133	655	483	483
query17	1183	706	541	541
query18	1933	387	303	303
query19	192	200	156	156
query20	118	122	112	112
query21	211	120	106	106
query22	4084	4268	4338	4268
query23	34376	33363	32930	32930
query24	8378	2391	2377	2377
query25	558	474	411	411
query26	1237	276	158	158
query27	2729	493	334	334
query28	4365	2421	2440	2421
query29	748	544	404	404
query30	231	188	156	156
query31	918	834	805	805
query32	72	61	62	61
query33	548	353	297	297
query34	766	851	489	489
query35	775	815	716	716
query36	967	975	905	905
query37	115	97	71	71
query38	4209	4112	4179	4112
query39	1453	1405	1418	1405
query40	209	114	102	102
query41	52	57	50	50
query42	125	105	113	105
query43	490	519	467	467
query44	1265	803	792	792
query45	174	166	158	158
query46	875	1034	644	644
query47	1793	1781	1698	1698
query48	378	427	305	305
query49	784	493	416	416
query50	671	729	413	413
query51	4155	4250	4108	4108
query52	103	105	92	92
query53	238	256	185	185
query54	488	475	418	418
query55	82	79	87	79
query56	268	265	256	256
query57	1137	1140	1074	1074
query58	249	229	240	229
query59	2487	2593	2493	2493
query60	295	282	260	260
query61	119	157	119	119
query62	772	737	694	694
query63	246	191	195	191
query64	4382	970	653	653
query65	3256	3118	3123	3118
query66	1127	396	304	304
query67	15762	15407	15560	15407
query68	4378	793	550	550
query69	482	304	272	272
query70	1190	1158	1134	1134
query71	382	308	318	308
query72	5743	3629	3915	3629
query73	778	763	365	365
query74	9165	9107	9157	9107
query75	3161	3160	2699	2699
query76	3173	1154	710	710
query77	473	372	353	353
query78	10017	9939	9264	9264
query79	2795	816	633	633
query80	1094	542	457	457
query81	540	268	237	237
query82	553	123	93	93
query83	166	174	168	168
query84	244	89	78	78
query85	786	346	300	300
query86	433	317	296	296
query87	4488	4443	4560	4443
query88	4336	2214	2208	2208
query89	388	317	281	281
query90	1887	193	193	193
query91	134	144	114	114
query92	72	60	59	59
query93	2581	1003	594	594
query94	731	406	304	304
query95	345	270	263	263
query96	528	557	280	280
query97	2842	2851	2736	2736
query98	224	204	214	204
query99	1314	1418	1285	1285
Total cold run time: 270191 ms
Total hot run time: 184031 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.25 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 19f6ceb9d63959e9c70654b6f278222c71d36511, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.41	0.41	0.41
query6	1.20	0.65	0.67
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.60	0.51	0.52
query10	0.58	0.59	0.57
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.63	0.60	0.59
query14	2.73	2.71	2.71
query15	0.93	0.85	0.84
query16	0.37	0.36	0.37
query17	1.02	1.05	1.04
query18	0.21	0.19	0.19
query19	1.92	1.86	1.95
query20	0.01	0.02	0.01
query21	15.35	0.88	0.53
query22	0.76	1.20	0.71
query23	14.83	1.40	0.65
query24	11.34	1.21	0.37
query25	0.33	0.08	0.11
query26	0.83	0.19	0.15
query27	0.05	0.04	0.05
query28	8.09	0.78	0.43
query29	12.54	3.92	3.26
query30	0.25	0.09	0.06
query31	2.82	0.59	0.38
query32	3.23	0.55	0.46
query33	3.01	2.99	3.05
query34	15.82	5.20	4.57
query35	4.55	4.56	4.59
query36	0.66	0.48	0.48
query37	0.10	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.18	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 107.93 s
Total hot run time: 30.25 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 43.83% (11635/26545)
Line Coverage: 33.75% (97546/289019)
Region Coverage: 32.83% (49916/152023)
Branch Coverage: 28.55% (25092/87882)
Coverage Report: http://coverage.selectdb-in.cc/coverage/19f6ceb9d63959e9c70654b6f278222c71d36511_19f6ceb9d63959e9c70654b6f278222c71d36511/report/index.html

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 19, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zhannngchen zhannngchen added p0_c and removed approved Indicates a PR has been approved by one committer. reviewed labels Feb 19, 2025
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 19, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zhannngchen zhannngchen merged commit 33375c4 into apache:master Feb 19, 2025
32 of 35 checks passed
bobhan1 added a commit to bobhan1/doris that referenced this pull request Feb 19, 2025
…s before skip if tablet is in `NOT_READY` state in flush phase (apache#48056)

Problem Summary:

considering the following situation:
1. heavy sc start on BE and change tablet's state to `NOT_READY`
2. load1 skip to calculate delete bitmaps in flush phase because the
tablet's state is `NOT_READY`
3. heavy sc finished and change tablet's state to `RUNNING`
4. load1 calculate delete bitmaps in publish phase for converted
historical rowsets

If load1 write multi segments, this will cause duplicate key because it
forget to calculate delete bitmaps between segments in any phase.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Feb 19, 2025
…s before skip if tablet is in `NOT_READY` state in flush phase (apache#48056)

Problem Summary:

considering the following situation:
1. heavy sc start on BE and change tablet's state to `NOT_READY`
2. load1 skip to calculate delete bitmaps in flush phase because the
tablet's state is `NOT_READY`
3. heavy sc finished and change tablet's state to `RUNNING`
4. load1 calculate delete bitmaps in publish phase for converted
historical rowsets

If load1 write multi segments, this will cause duplicate key because it
forget to calculate delete bitmaps between segments in any phase.
bobhan1 added a commit to bobhan1/doris that referenced this pull request Feb 19, 2025
…s before skip if tablet is in `NOT_READY` state in flush phase (apache#48056)

Problem Summary:

considering the following situation:
1. heavy sc start on BE and change tablet's state to `NOT_READY`
2. load1 skip to calculate delete bitmaps in flush phase because the
tablet's state is `NOT_READY`
3. heavy sc finished and change tablet's state to `RUNNING`
4. load1 calculate delete bitmaps in publish phase for converted
historical rowsets

If load1 write multi segments, this will cause duplicate key because it
forget to calculate delete bitmaps between segments in any phase.
dataroaring pushed a commit that referenced this pull request Feb 19, 2025
…s between segments before skip if tablet is in NOT_READY state in flush phase #48056 (#48085)

pick #48056
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
…s before skip if tablet is in `NOT_READY` state in flush phase (apache#48056)

### What problem does this PR solve?
Problem Summary:

considering the following situation:
1. heavy sc start on BE and change tablet's state to `NOT_READY`
2. load1 skip to calculate delete bitmaps in flush phase because the
tablet's state is `NOT_READY`
3. heavy sc finished and change tablet's state to `RUNNING`
4. load1 calculate delete bitmaps in publish phase for converted
historical rowsets

If load1 write multi segments, this will cause duplicate key because it
forget to calculate delete bitmaps between segments in any phase.
dataroaring pushed a commit that referenced this pull request Feb 21, 2025
…s between segmens before skip if tablet is in `NOT_READY` state in flush phase (#48056) (#48089)

pick #48056
@yiguolei yiguolei mentioned this pull request Mar 25, 2025
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…s before skip if tablet is in `NOT_READY` state in flush phase (apache#48056)

### What problem does this PR solve?
Problem Summary:

considering the following situation:
1. heavy sc start on BE and change tablet's state to `NOT_READY`
2. load1 skip to calculate delete bitmaps in flush phase because the
tablet's state is `NOT_READY`
3. heavy sc finished and change tablet's state to `RUNNING`
4. load1 calculate delete bitmaps in publish phase for converted
historical rowsets

If load1 write multi segments, this will cause duplicate key because it
forget to calculate delete bitmaps between segments in any phase.
deardeng pushed a commit to deardeng/incubator-doris that referenced this pull request Dec 19, 2025
…s between segmens before skip if tablet is in `NOT_READY` state in flush phase (apache#48056) (apache#48089) (apache#334)

pick apache#48056
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged p0_c reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants