[fix](mv) refresh failed while open enable_single_replica_insert #50986

cambyzju · 2025-05-16T12:19:40Z

What problem does this PR solve?

pick #32845 to nereids

Problem Summary:

errCode = 2, detailMessage = Failed to commit txn 3059, cause tablet 13661 succ replica num 1 < load required replica num 2. table 13263, partition: [ id=13660, commit version 1, visible version 1 ], this tablet detail: 1 replicas final succ: { [replicaId=13662, backendId=10002, backendAlive=true, version=1, state=NORMAL] }; 2 replicas write data failed: { [replicaId=13663, backendId=10003, backendAlive=true, version=1, state=NORMAL], [replicaId=13664, backendId=10004, backendAlive=true, version=1, state=NORMAL] }, please check BE log for details; . Please try again later.

Reproduce Steps:

make sure enable_memtable_on_sink_node is opened

CREATE TABLE `test_a` (
  `k` int NULL,
  `v1` text NULL,
  `v2` text NULL
) ENGINE=OLAP
UNIQUE KEY(`k`)
DISTRIBUTED BY HASH(`k`) BUCKETS 10;

insert into test_a values(1,2,3),(4,5,6),(7,8,9);

set global experimental_enable_single_replica_insert=true;

CREATE MATERIALIZED VIEW mv_test
REFRESH COMPLETE ON MANUAL
DISTRIBUTED BY HASH(`k`) BUCKETS 10
AS
SELECT * FROM test_a;

REFRESH MATERIALIZED VIEW mv_test AUTO;

select * from tasks("type"="mv");

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2025-05-16T12:19:47Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

dataroaring

LGTM

github-actions · 2025-05-16T12:22:39Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-05-16T12:22:41Z

PR approved by anyone and no changes requested.

lide-reed

LGTM

cambyzju · 2025-05-19T02:18:56Z

run buildall

doris-robot · 2025-05-19T02:57:53Z

TPC-H: Total hot run time: 33812 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11da4ba5781575f42c5ea2d0cf3e55cb03ff13d4, data reload: false

------ Round 1 ----------------------------------
q1	26074	5024	5024	5024
q2	2061	273	183	183
q3	10402	1221	705	705
q4	10218	999	505	505
q5	7555	2256	2369	2256
q6	180	160	131	131
q7	921	749	627	627
q8	9306	1230	1097	1097
q9	6826	5246	5168	5168
q10	6910	2353	1908	1908
q11	506	291	266	266
q12	365	354	207	207
q13	18415	3645	3078	3078
q14	225	234	215	215
q15	518	498	483	483
q16	415	437	374	374
q17	586	849	353	353
q18	7481	7168	7071	7071
q19	1365	969	547	547
q20	340	344	228	228
q21	3738	2554	2422	2422
q22	1029	1027	964	964
Total cold run time: 115436 ms
Total hot run time: 33812 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5090	5031	5034	5031
q2	232	335	231	231
q3	2124	2654	2275	2275
q4	1326	1728	1365	1365
q5	4360	4387	4380	4380
q6	223	168	125	125
q7	1968	1929	1737	1737
q8	2606	2674	2561	2561
q9	7191	7097	6959	6959
q10	3036	3181	2767	2767
q11	581	538	490	490
q12	664	793	619	619
q13	3518	3876	3290	3290
q14	274	296	277	277
q15	539	519	515	515
q16	449	474	440	440
q17	1119	1553	1380	1380
q18	7922	7531	7450	7450
q19	783	779	881	779
q20	1979	2113	1865	1865
q21	4685	4500	4366	4366
q22	1096	1062	1056	1056
Total cold run time: 51765 ms
Total hot run time: 49958 ms

doris-robot · 2025-05-19T03:10:07Z

TPC-DS: Total hot run time: 193688 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11da4ba5781575f42c5ea2d0cf3e55cb03ff13d4, data reload: false

query1	1429	1100	1054	1054
query2	6382	1874	1809	1809
query3	11018	4438	4610	4438
query4	53329	24286	23089	23089
query5	5035	563	481	481
query6	361	222	202	202
query7	4921	516	311	311
query8	321	258	251	251
query9	5596	2678	2698	2678
query10	457	344	290	290
query11	14994	15014	14989	14989
query12	160	111	103	103
query13	1023	537	408	408
query14	10085	6407	6257	6257
query15	196	197	167	167
query16	6991	656	480	480
query17	1075	729	575	575
query18	1552	413	317	317
query19	198	203	177	177
query20	128	125	121	121
query21	211	133	110	110
query22	4385	4532	4346	4346
query23	34537	33590	33653	33590
query24	6617	2459	2479	2459
query25	523	517	450	450
query26	689	285	169	169
query27	2208	519	360	360
query28	3249	2182	2170	2170
query29	617	605	503	503
query30	277	231	197	197
query31	867	917	802	802
query32	76	63	62	62
query33	446	382	317	317
query34	792	864	538	538
query35	796	826	777	777
query36	936	999	897	897
query37	113	95	77	77
query38	4266	4261	4290	4261
query39	1532	1443	1507	1443
query40	212	134	108	108
query41	61	56	54	54
query42	126	114	114	114
query43	500	515	486	486
query44	1328	862	866	862
query45	187	175	170	170
query46	912	1036	665	665
query47	1871	1870	1829	1829
query48	410	433	351	351
query49	740	525	434	434
query50	673	719	409	409
query51	4251	4275	4305	4275
query52	114	111	104	104
query53	236	264	193	193
query54	587	597	533	533
query55	90	92	90	90
query56	333	323	308	308
query57	1183	1209	1088	1088
query58	271	283	270	270
query59	2756	2748	2699	2699
query60	335	343	335	335
query61	132	126	142	126
query62	712	722	671	671
query63	220	186	182	182
query64	1894	1036	738	738
query65	4285	4256	4208	4208
query66	797	402	294	294
query67	16209	15681	15532	15532
query68	8313	900	546	546
query69	548	360	280	280
query70	1186	1084	1067	1067
query71	533	326	294	294
query72	5527	4829	4832	4829
query73	1428	651	364	364
query74	9017	8859	8947	8859
query75	3939	3207	2705	2705
query76	4246	1203	772	772
query77	682	368	290	290
query78	9945	10233	9498	9498
query79	2448	823	590	590
query80	639	528	463	463
query81	484	256	224	224
query82	421	129	93	93
query83	332	255	234	234
query84	293	102	90	90
query85	785	356	392	356
query86	365	302	269	269
query87	4331	4507	4396	4396
query88	3491	2370	2353	2353
query89	405	316	286	286
query90	1925	224	221	221
query91	147	142	117	117
query92	72	62	61	61
query93	1392	948	597	597
query94	684	408	296	296
query95	379	296	290	290
query96	502	584	292	292
query97	2736	2799	2620	2620
query98	242	203	205	203
query99	1457	1419	1280	1280
Total cold run time: 298707 ms
Total hot run time: 193688 ms

xy720

LGTM

doris-robot · 2025-05-19T03:15:18Z

ClickBench: Total hot run time: 28.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11da4ba5781575f42c5ea2d0cf3e55cb03ff13d4, data reload: false

query1	0.04	0.03	0.04
query2	0.13	0.10	0.12
query3	0.26	0.19	0.20
query4	1.59	0.19	0.11
query5	0.44	0.42	0.42
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.52	0.52
query10	0.58	0.57	0.56
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	0.78	0.81	0.81
query15	0.88	0.86	0.84
query16	0.38	0.38	0.38
query17	1.02	1.07	1.08
query18	0.22	0.20	0.21
query19	1.91	1.86	1.82
query20	0.02	0.00	0.01
query21	15.41	0.87	0.55
query22	0.77	1.22	0.64
query23	14.92	1.41	0.62
query24	7.32	1.49	0.77
query25	0.48	0.24	0.08
query26	0.66	0.16	0.13
query27	0.05	0.06	0.05
query28	9.17	0.92	0.44
query29	12.55	3.97	3.26
query30	0.25	0.10	0.07
query31	2.83	0.61	0.38
query32	3.24	0.56	0.46
query33	3.13	3.05	3.19
query34	15.79	5.10	4.52
query35	4.51	4.52	4.54
query36	0.69	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 103.25 s
Total hot run time: 28.92 s

)

…_insert #50986 (#51021) Cherry-picked from #50986 Co-authored-by: camby <cambyzhu@tencent.com>

…_insert #50986 (#51020) Cherry-picked from #50986 Co-authored-by: camby <cambyzhu@tencent.com>

…che#50986)

fix mtmv refresh failed

11da4ba

cambyzju added dev/2.1.x dev/3.0.x dev/3.1.x labels May 16, 2025

dataroaring approved these changes May 16, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label May 16, 2025

github-actions bot added the reviewed label May 16, 2025

cambyzju changed the title ~~[fix](mv) refresh failed while open enable_memtable_on_sink_node~~ [fix](mv) refresh failed while open enable_single_replica_insert May 16, 2025

lide-reed approved these changes May 17, 2025

View reviewed changes

liutang123 approved these changes May 19, 2025

View reviewed changes

xy720 approved these changes May 19, 2025

View reviewed changes

cambyzju merged commit bba3d35 into apache:master May 19, 2025
34 of 36 checks passed

github-actions bot pushed a commit that referenced this pull request May 19, 2025

[fix](mv) refresh failed while open enable_single_replica_insert (#50986

682754a

)

github-actions bot mentioned this pull request May 19, 2025

branch-3.0: [fix](mv) refresh failed while open enable_single_replica_insert #50986 #51020

Merged

github-actions bot pushed a commit that referenced this pull request May 19, 2025

[fix](mv) refresh failed while open enable_single_replica_insert (#50986

87d779b

)

github-actions bot mentioned this pull request May 19, 2025

branch-2.1: [fix](mv) refresh failed while open enable_single_replica_insert #50986 #51021

Merged

yiguolei added dev/2.1.11-merged and removed dev/2.1.x labels May 21, 2025

yiguolei pushed a commit that referenced this pull request May 21, 2025

branch-2.1: [fix](mv) refresh failed while open enable_single_replica…

d63815f

…_insert #50986 (#51021) Cherry-picked from #50986 Co-authored-by: camby <cambyzhu@tencent.com>

dataroaring pushed a commit that referenced this pull request May 24, 2025

branch-3.0: [fix](mv) refresh failed while open enable_single_replica…

016168d

…_insert #50986 (#51020) Cherry-picked from #50986 Co-authored-by: camby <cambyzhu@tencent.com>

dataroaring added dev/3.0.6-merged and removed dev/3.0.x labels May 24, 2025

koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025

[fix](mv) refresh failed while open enable_single_replica_insert (apa…

2deaa0e

…che#50986)

morrySnow removed the dev/3.1.x label Aug 1, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](mv) refresh failed while open enable_single_replica_insert #50986

[fix](mv) refresh failed while open enable_single_replica_insert #50986

Uh oh!

cambyzju commented May 16, 2025 •

edited

Loading

Uh oh!

Thearas commented May 16, 2025

Uh oh!

dataroaring left a comment

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

lide-reed left a comment

Uh oh!

cambyzju commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

xy720 left a comment

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

[fix](mv) refresh failed while open enable_single_replica_insert #50986

[fix](mv) refresh failed while open enable_single_replica_insert #50986

Uh oh!

Conversation

cambyzju commented May 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Problem Summary:

Reproduce Steps:

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented May 16, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

github-actions bot commented May 16, 2025

Uh oh!

lide-reed left a comment

Choose a reason for hiding this comment

Uh oh!

cambyzju commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

xy720 left a comment

Choose a reason for hiding this comment

Uh oh!

doris-robot commented May 19, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

cambyzju commented May 16, 2025 •

edited

Loading