Skip to content

[fix](backup) Save snapshot meta during replay #49550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
w41ter merged 2 commits into from
Mar 28, 2025
Merged

[fix](backup) Save snapshot meta during replay #49550

w41ter merged 2 commits into from
Mar 28, 2025

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Mar 26, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 26, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@w41ter
Copy link
Contributor Author

w41ter commented Mar 26, 2025

run buildall

dataroaring
dataroaring previously approved these changes Mar 26, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 26, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 26, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 26, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-H: Total hot run time: 34171 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8974b054a03905919f2cdf0697e47b7955bb8a9, data reload: false

------ Round 1 ----------------------------------
q1	25772	5189	5029	5029
q2	2083	279	157	157
q3	10413	1238	679	679
q4	10231	997	531	531
q5	7513	2387	2253	2253
q6	193	166	131	131
q7	917	732	609	609
q8	9323	1201	1147	1147
q9	6729	5121	5198	5121
q10	6798	2269	1893	1893
q11	471	267	249	249
q12	354	350	220	220
q13	17775	3690	3083	3083
q14	221	243	212	212
q15	533	473	485	473
q16	632	614	599	599
q17	570	843	340	340
q18	7712	7368	7545	7368
q19	1232	932	541	541
q20	318	339	192	192
q21	3884	2608	2398	2398
q22	1036	1014	946	946
Total cold run time: 114710 ms
Total hot run time: 34171 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5148	5094	5112	5094
q2	245	321	225	225
q3	2194	2683	2265	2265
q4	1415	1807	1361	1361
q5	4472	4398	4373	4373
q6	211	171	129	129
q7	1980	1898	1753	1753
q8	2609	2641	2563	2563
q9	7281	7263	7190	7190
q10	2990	3131	2746	2746
q11	571	512	502	502
q12	696	758	628	628
q13	3518	3895	3272	3272
q14	311	287	260	260
q15	528	479	488	479
q16	643	683	648	648
q17	1146	1510	1380	1380
q18	7767	7627	7512	7512
q19	834	847	919	847
q20	1915	1964	1814	1814
q21	5371	4905	4861	4861
q22	1118	1045	1027	1027
Total cold run time: 52963 ms
Total hot run time: 50929 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

TPC-DS: Total hot run time: 192562 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8974b054a03905919f2cdf0697e47b7955bb8a9, data reload: false

query1	1420	1074	1020	1020
query2	6238	1899	1913	1899
query3	11095	4729	4548	4548
query4	25741	23829	22957	22957
query5	4416	672	468	468
query6	298	192	186	186
query7	3986	488	266	266
query8	294	229	228	228
query9	8503	2525	2523	2523
query10	469	314	261	261
query11	15628	15025	14818	14818
query12	169	113	107	107
query13	1580	527	410	410
query14	8943	6181	6261	6181
query15	215	181	168	168
query16	7613	629	515	515
query17	1212	686	582	582
query18	2022	419	334	334
query19	192	195	153	153
query20	128	116	114	114
query21	206	125	107	107
query22	4722	4701	4303	4303
query23	34302	33861	33531	33531
query24	8550	2410	2395	2395
query25	520	486	411	411
query26	1201	270	140	140
query27	2939	509	326	326
query28	4796	2413	2418	2413
query29	689	566	457	457
query30	282	228	189	189
query31	920	856	810	810
query32	78	59	63	59
query33	563	386	324	324
query34	807	871	520	520
query35	858	915	749	749
query36	968	1006	909	909
query37	116	104	81	81
query38	4237	4238	4238	4238
query39	1503	1507	1411	1411
query40	212	122	109	109
query41	55	87	52	52
query42	120	114	106	106
query43	495	503	470	470
query44	1329	814	815	814
query45	184	174	165	165
query46	859	1043	650	650
query47	1867	1912	1829	1829
query48	397	414	315	315
query49	784	533	432	432
query50	731	758	425	425
query51	4318	4255	4214	4214
query52	114	105	101	101
query53	224	260	174	174
query54	496	490	427	427
query55	80	79	80	79
query56	280	271	268	268
query57	1184	1211	1140	1140
query58	261	237	240	237
query59	2834	2942	2780	2780
query60	301	298	274	274
query61	133	134	126	126
query62	785	751	665	665
query63	226	187	184	184
query64	4039	1044	720	720
query65	4491	4383	4389	4383
query66	994	395	299	299
query67	16276	15645	15394	15394
query68	8201	874	515	515
query69	480	301	259	259
query70	1145	1138	1114	1114
query71	456	295	265	265
query72	5128	4628	4613	4613
query73	649	576	346	346
query74	9039	9163	8937	8937
query75	3971	3233	2708	2708
query76	3764	1180	757	757
query77	804	362	278	278
query78	9795	10077	9285	9285
query79	1222	817	567	567
query80	602	517	447	447
query81	474	251	223	223
query82	191	124	93	93
query83	177	175	155	155
query84	232	93	73	73
query85	735	380	306	306
query86	327	301	282	282
query87	4374	4541	4368	4368
query88	2804	2245	2264	2245
query89	385	316	285	285
query90	2042	211	208	208
query91	155	155	210	155
query92	72	62	55	55
query93	1144	1067	578	578
query94	653	413	308	308
query95	357	264	268	264
query96	489	560	272	272
query97	3116	3268	3199	3199
query98	238	209	198	198
query99	1439	1409	1261	1261
Total cold run time: 277148 ms
Total hot run time: 192562 ms

@doris-robot
Copy link

doris-robot commented Mar 26, 2025

ClickBench: Total hot run time: 31.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8974b054a03905919f2cdf0697e47b7955bb8a9, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.11	0.11
query3	0.24	0.20	0.19
query4	1.59	0.19	0.19
query5	0.62	0.58	0.60
query6	1.21	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.57	0.53	0.51
query10	0.59	0.58	0.56
query11	0.16	0.10	0.11
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	2.81	2.71	2.80
query15	0.93	0.87	0.84
query16	0.38	0.40	0.38
query17	1.04	1.05	1.03
query18	0.22	0.20	0.19
query19	1.88	1.96	1.89
query20	0.02	0.02	0.01
query21	15.35	0.88	0.54
query22	0.77	1.17	0.63
query23	15.01	1.41	0.57
query24	7.71	1.74	0.90
query25	0.49	0.18	0.13
query26	0.69	0.16	0.13
query27	0.05	0.06	0.06
query28	9.68	0.84	0.46
query29	12.54	3.98	3.33
query30	0.25	0.10	0.06
query31	2.81	0.61	0.37
query32	3.23	0.54	0.49
query33	3.02	3.10	3.15
query34	15.80	5.11	4.55
query35	4.56	4.54	4.58
query36	0.66	0.51	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.18	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.37 s
Total hot run time: 31.44 s

@w41ter
Copy link
Contributor Author

w41ter commented Mar 27, 2025

run buildall

@w41ter
Copy link
Contributor Author

w41ter commented Mar 27, 2025

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 27, 2025
@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-H: Total hot run time: 34119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 00a6ba27eb7c220559b204ea9f86b08a8934fcf1, data reload: false

------ Round 1 ----------------------------------
q1	25727	5133	5076	5076
q2	2070	287	163	163
q3	10403	1289	673	673
q4	10216	1046	562	562
q5	7550	2371	2377	2371
q6	189	167	134	134
q7	943	760	619	619
q8	9325	1199	1110	1110
q9	6783	5159	5116	5116
q10	6859	2325	1908	1908
q11	489	272	253	253
q12	350	365	220	220
q13	17764	3686	3117	3117
q14	222	227	213	213
q15	536	474	488	474
q16	640	605	584	584
q17	577	858	347	347
q18	7540	7219	7003	7003
q19	1462	972	584	584
q20	322	334	203	203
q21	3990	2659	2419	2419
q22	1021	1035	970	970
Total cold run time: 114978 ms
Total hot run time: 34119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5230	5174	5345	5174
q2	233	319	231	231
q3	2189	2674	2293	2293
q4	1412	1878	1487	1487
q5	4500	4458	4387	4387
q6	220	175	133	133
q7	2010	1936	1745	1745
q8	2619	2628	2559	2559
q9	7254	7223	6986	6986
q10	3045	3196	2746	2746
q11	559	503	498	498
q12	702	780	621	621
q13	3517	3895	3360	3360
q14	290	296	291	291
q15	537	495	487	487
q16	648	697	648	648
q17	1138	1552	1449	1449
q18	7835	7748	7461	7461
q19	814	847	1002	847
q20	1979	2024	1839	1839
q21	5476	4840	4869	4840
q22	1084	1072	1053	1053
Total cold run time: 53291 ms
Total hot run time: 51135 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

TPC-DS: Total hot run time: 191308 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 00a6ba27eb7c220559b204ea9f86b08a8934fcf1, data reload: false

query1	1437	1086	1048	1048
query2	6345	1887	1861	1861
query3	11036	4465	4542	4465
query4	53283	25420	23695	23695
query5	5110	644	495	495
query6	337	188	190	188
query7	4877	501	283	283
query8	339	254	244	244
query9	5379	2543	2546	2543
query10	442	305	264	264
query11	15398	15089	15049	15049
query12	159	120	108	108
query13	1035	520	398	398
query14	10069	6464	6741	6464
query15	208	203	195	195
query16	7166	665	489	489
query17	1086	742	628	628
query18	1533	428	330	330
query19	204	222	184	184
query20	124	124	119	119
query21	214	129	118	118
query22	4474	4496	4377	4377
query23	34124	33428	33479	33428
query24	6621	2434	2475	2434
query25	468	465	397	397
query26	741	270	148	148
query27	2489	522	329	329
query28	3156	2475	2475	2475
query29	591	565	467	467
query30	274	220	194	194
query31	847	894	815	815
query32	78	66	66	66
query33	447	364	307	307
query34	764	852	532	532
query35	828	825	773	773
query36	951	1025	896	896
query37	121	105	80	80
query38	4140	4099	4216	4099
query39	1529	1450	1443	1443
query40	204	117	111	111
query41	54	52	50	50
query42	127	112	106	106
query43	499	481	487	481
query44	1347	824	815	815
query45	173	188	171	171
query46	843	1032	647	647
query47	1883	1877	1839	1839
query48	395	433	302	302
query49	673	540	447	447
query50	697	746	435	435
query51	4336	4266	4229	4229
query52	108	112	106	106
query53	227	263	179	179
query54	496	511	449	449
query55	87	81	83	81
query56	275	284	302	284
query57	1190	1221	1130	1130
query58	256	265	257	257
query59	2712	2785	2657	2657
query60	300	294	266	266
query61	147	132	128	128
query62	711	755	674	674
query63	227	189	184	184
query64	1811	1078	695	695
query65	4437	4309	4363	4309
query66	717	410	309	309
query67	15939	15679	15311	15311
query68	6889	881	515	515
query69	529	294	266	266
query70	1204	1111	1133	1111
query71	508	291	276	276
query72	5558	2385	5402	2385
query73	1521	762	360	360
query74	8877	8994	8695	8695
query75	4117	3238	2756	2756
query76	4233	1187	769	769
query77	737	381	282	282
query78	9937	10046	9333	9333
query79	3349	812	555	555
query80	648	516	461	461
query81	478	254	219	219
query82	475	132	96	96
query83	232	184	163	163
query84	282	99	78	78
query85	776	350	332	332
query86	331	321	284	284
query87	4701	4459	4372	4372
query88	3087	2227	2196	2196
query89	419	313	282	282
query90	1980	211	207	207
query91	141	143	109	109
query92	73	57	56	56
query93	2033	1069	577	577
query94	680	423	293	293
query95	354	285	261	261
query96	476	563	269	269
query97	3190	3189	3169	3169
query98	225	215	225	215
query99	1471	1383	1286	1286
Total cold run time: 298806 ms
Total hot run time: 191308 ms

@doris-robot
Copy link

doris-robot commented Mar 27, 2025

ClickBench: Total hot run time: 31.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 00a6ba27eb7c220559b204ea9f86b08a8934fcf1, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.10
query3	0.24	0.20	0.19
query4	1.59	0.18	0.11
query5	0.56	0.54	0.54
query6	1.17	0.72	0.71
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.53
query10	0.59	0.58	0.61
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.62	0.59	0.60
query14	2.70	2.81	2.81
query15	0.92	0.85	0.84
query16	0.38	0.37	0.38
query17	1.00	1.04	1.02
query18	0.22	0.20	0.20
query19	2.02	1.88	1.86
query20	0.01	0.01	0.01
query21	15.35	0.94	0.54
query22	0.76	1.17	0.65
query23	14.93	1.40	0.65
query24	7.09	1.16	1.05
query25	0.50	0.20	0.12
query26	0.69	0.16	0.12
query27	0.06	0.05	0.05
query28	9.19	0.90	0.42
query29	12.54	3.98	3.30
query30	0.25	0.09	0.06
query31	2.83	0.57	0.38
query32	3.26	0.55	0.48
query33	3.09	3.12	3.14
query34	15.86	5.14	4.52
query35	4.53	4.50	4.50
query36	0.65	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.04
query39	0.02	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.18 s
Total hot run time: 31.53 s

dataroaring
dataroaring approved these changes Mar 27, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 27, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 27, 2025

PR approved by at least one committer and no changes requested.

xy720
xy720 approved these changes Mar 27, 2025
View reviewed changes
Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@w41ter w41ter merged commit 9056b62 into apache:master Mar 28, 2025
29 of 31 checks passed
@w41ter w41ter deleted the fix_backup_follower_replay branch March 28, 2025 01:58
github-actions bot pushed a commit that referenced this pull request Mar 28, 2025
@w41ter
[fix](backup) Save snapshot meta during replay (#49550)
23cdf0f
@github-actions github-actions bot mentioned this pull request Mar 28, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Mar 28, 2025
@w41ter
[fix](backup) Save snapshot meta during replay (#49550)
9f3a734
@github-actions github-actions bot mentioned this pull request Mar 28, 2025
Merged
dataroaring pushed a commit that referenced this pull request Mar 28, 2025
@github-actions @w41ter
branch-3.0: [fix](backup) Save snapshot meta during replay #49550 (#4…
5e42583 
…9605)

Cherry-picked from #49550

Co-authored-by: walter <maochuan@selectdb.com>
yiguolei pushed a commit that referenced this pull request Apr 9, 2025
@github-actions @w41ter
branch-2.1: [fix](backup) Save snapshot meta during replay #49550 (#4…
81ea088 
…9606)

Cherry-picked from #49550

Co-authored-by: walter <maochuan@selectdb.com>
@yiguolei yiguolei mentioned this pull request May 13, 2025
Closed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@w41ter @koarz
[fix](backup) Save snapshot meta during replay (apache#49550)
c1d328e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xy720 xy720 xy720 approved these changes

@dataroaring dataroaring dataroaring approved these changes

+2 more reviewers

@koarz koarz koarz approved these changes

@wyxxxcat wyxxxcat wyxxxcat approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.5-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@w41ter @Thearas @doris-robot @xy720 @koarz @dataroaring @wyxxxcat @yiguolei