Skip to content

Conversation

@bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Mar 21, 2025

What problem does this PR solve?

When calculating delete bitmaps for incremental rowsets during schema change double write phase, sync_rowsets on tmp_tablet may override local converted historical rowsets with rowsets on new tablet. But these rowsets will be removed when commit sc job.
This PR re-add converted rowsets after sync_rowsets to avoid the problem

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 21, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34174 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae2df8107d2c18343328eeefdeb57ec4bbeb0a4e, data reload: false

------ Round 1 ----------------------------------
q1	24420	5126	5088	5088
q2	2047	315	181	181
q3	10379	1274	690	690
q4	10234	987	547	547
q5	7544	2388	2329	2329
q6	186	159	132	132
q7	920	732	602	602
q8	9319	1282	1149	1149
q9	6842	5125	5102	5102
q10	6813	2311	1892	1892
q11	482	268	258	258
q12	345	355	218	218
q13	17770	3698	3120	3120
q14	224	228	209	209
q15	545	491	487	487
q16	639	614	585	585
q17	575	854	337	337
q18	7539	7198	7081	7081
q19	1210	1083	545	545
q20	322	322	199	199
q21	3988	3492	2463	2463
q22	1005	998	960	960
Total cold run time: 113348 ms
Total hot run time: 34174 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5190	5166	5153	5153
q2	233	323	234	234
q3	2139	2654	2324	2324
q4	1416	1843	1415	1415
q5	4485	4425	4440	4425
q6	222	167	128	128
q7	1985	1905	1793	1793
q8	2616	2638	2601	2601
q9	7128	7194	7079	7079
q10	2963	3186	2738	2738
q11	562	522	529	522
q12	701	807	650	650
q13	3602	3969	3363	3363
q14	295	291	279	279
q15	518	482	472	472
q16	677	671	640	640
q17	1125	1626	1311	1311
q18	7777	7702	7427	7427
q19	836	835	912	835
q20	2003	2023	1810	1810
q21	5402	5028	4885	4885
q22	1059	1051	1032	1032
Total cold run time: 52934 ms
Total hot run time: 51116 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193767 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae2df8107d2c18343328eeefdeb57ec4bbeb0a4e, data reload: false

query1	1466	1057	1035	1035
query2	6140	1909	1878	1878
query3	10975	4511	4464	4464
query4	57669	25494	23555	23555
query5	5224	496	497	496
query6	426	204	201	201
query7	5332	492	273	273
query8	332	258	241	241
query9	7379	2638	2592	2592
query10	415	306	259	259
query11	15501	15183	14786	14786
query12	157	108	103	103
query13	1394	552	414	414
query14	10110	6391	7967	6391
query15	193	195	178	178
query16	7031	667	486	486
query17	1049	714	592	592
query18	1505	398	305	305
query19	220	193	160	160
query20	124	119	124	119
query21	207	122	103	103
query22	4336	4441	4288	4288
query23	34114	33312	33282	33282
query24	5672	2486	2447	2447
query25	453	477	402	402
query26	691	275	155	155
query27	1686	500	334	334
query28	2770	2449	2431	2431
query29	586	556	438	438
query30	290	230	206	206
query31	895	876	810	810
query32	74	62	60	60
query33	453	375	308	308
query34	770	848	524	524
query35	814	845	789	789
query36	955	1003	916	916
query37	136	105	81	81
query38	4171	4230	4331	4230
query39	1498	1443	1460	1443
query40	229	119	106	106
query41	57	52	53	52
query42	129	112	116	112
query43	505	509	484	484
query44	1345	833	816	816
query45	200	176	173	173
query46	873	1062	708	708
query47	1820	1860	1776	1776
query48	376	438	312	312
query49	700	506	448	448
query50	729	750	425	425
query51	4210	4350	4173	4173
query52	106	106	100	100
query53	229	258	189	189
query54	503	528	428	428
query55	79	80	84	80
query56	275	261	263	261
query57	1165	1166	1116	1116
query58	245	244	243	243
query59	2708	2873	2753	2753
query60	290	285	272	272
query61	126	141	119	119
query62	755	730	677	677
query63	224	185	188	185
query64	1501	1119	790	790
query65	4553	4500	4384	4384
query66	727	407	314	314
query67	15780	15450	15110	15110
query68	5611	841	522	522
query69	545	323	281	281
query70	1194	1137	1125	1125
query71	448	301	274	274
query72	6103	5141	5211	5141
query73	1011	680	339	339
query74	9163	8914	9162	8914
query75	3578	3236	2698	2698
query76	3843	1194	783	783
query77	534	370	289	289
query78	10292	10109	9283	9283
query79	2675	814	585	585
query80	741	508	438	438
query81	494	256	228	228
query82	674	126	95	95
query83	177	170	153	153
query84	288	91	76	76
query85	814	367	303	303
query86	449	314	296	296
query87	4432	4436	4433	4433
query88	3584	2269	2257	2257
query89	425	377	272	272
query90	1882	211	218	211
query91	143	139	109	109
query92	75	60	51	51
query93	2236	1064	578	578
query94	672	423	314	314
query95	352	274	264	264
query96	500	573	271	271
query97	3385	3511	3314	3314
query98	228	222	196	196
query99	1421	1410	1293	1293
Total cold run time: 301621 ms
Total hot run time: 193767 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae2df8107d2c18343328eeefdeb57ec4bbeb0a4e, data reload: false

query1	0.04	0.04	0.04
query2	0.13	0.10	0.11
query3	0.25	0.19	0.19
query4	1.59	0.20	0.19
query5	0.60	0.57	0.61
query6	1.19	0.71	0.72
query7	0.03	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.54	0.55
query10	0.56	0.58	0.58
query11	0.15	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	2.79	2.71	2.81
query15	0.92	0.86	0.86
query16	0.39	0.40	0.38
query17	1.02	1.04	1.01
query18	0.21	0.20	0.19
query19	1.96	1.97	1.83
query20	0.01	0.01	0.02
query21	15.38	0.90	0.54
query22	0.79	1.24	0.80
query23	14.71	1.36	0.62
query24	6.68	2.27	0.71
query25	0.49	0.08	0.26
query26	0.61	0.16	0.14
query27	0.06	0.05	0.05
query28	9.69	0.92	0.43
query29	12.59	4.08	3.40
query30	0.25	0.10	0.07
query31	2.82	0.60	0.38
query32	3.22	0.58	0.45
query33	3.02	3.05	3.07
query34	15.68	5.12	4.53
query35	4.50	4.54	4.48
query36	0.67	0.52	0.49
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 104.86 s
Total hot run time: 31.31 s

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 21, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 21, 2025

run p0

@bobhan1
Copy link
Contributor Author

bobhan1 commented Mar 21, 2025

run cloud_p0

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit f7b77e5 into apache:master Mar 22, 2025
33 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Mar 22, 2025
…s for incremental rowsets based on wrong historical rowsets during SC (#49350)

### What problem does this PR solve?

When calculating delete bitmaps for incremental rowsets during schema
change double write phase, `sync_rowsets` on tmp_tablet may override
local converted historical rowsets with rowsets on new tablet. But these
rowsets will be removed when commit sc job.
This PR re-add converted rowsets after `sync_rowsets` to avoid the
problem
dataroaring pushed a commit that referenced this pull request Mar 24, 2025
…elete bitmaps for incremental rowsets based on wrong historical rowsets during SC #49350 (#49364)

Cherry-picked from #49350

Co-authored-by: bobhan1 <baohan@selectdb.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…s for incremental rowsets based on wrong historical rowsets during SC (apache#49350)

### What problem does this PR solve?

When calculating delete bitmaps for incremental rowsets during schema
change double write phase, `sync_rowsets` on tmp_tablet may override
local converted historical rowsets with rowsets on new tablet. But these
rowsets will be removed when commit sc job.
This PR re-add converted rowsets after `sync_rowsets` to avoid the
problem
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged p0_w reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants