Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #47968

…ble_on_sink_node (#47968)

Problem Summary:

if `set global enable_memtable_on_sink_node = true`, replay wal will
failed:
```
failed to replay wal=/data3/storage/wal/118291/1739470368974/1_40199330_47385148733034496_group_commit_104d6db2bc9875fe_f7d1b359986d9c83, st=[INTERNAL_ERROR][INTERNAL_ERROR]close wait failed coz rpc error. VNodeChannel[1739470368975-40199330], load_id=ca40e6a07491aacf-a1aa8bb9dc1c31a5, txn_id=47401455637994497, node=[192.168.35.80:8060](http://192.168.35.80:8060/), add batch req success but status isn't ok, err: [INVALID_ARGUMENT]PStatus: ([192.168.35.80](http://192.168.35.80/))[INVALID_ARGUMENT]illegal partial update block columns: 61, num key columns: 2, total schema columns: 61, host: [192.168.35.80](http://192.168.35.80/)
```
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 18, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40791 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 36fdc318e43c5e37d1074024d88d2e273babb3b0, data reload: false

------ Round 1 ----------------------------------
q1	17567	7345	7163	7163
q2	2061	174	163	163
q3	10634	1096	1199	1096
q4	10545	705	762	705
q5	7712	2850	2824	2824
q6	234	141	142	141
q7	964	614	605	605
q8	9338	1924	2020	1924
q9	6605	6391	6399	6391
q10	7048	2291	2298	2291
q11	486	260	263	260
q12	412	214	208	208
q13	17799	3002	3029	3002
q14	236	206	211	206
q15	579	528	529	528
q16	661	595	594	594
q17	963	568	578	568
q18	7192	6684	6676	6676
q19	1405	1075	1036	1036
q20	473	210	205	205
q21	3974	3218	3319	3218
q22	1117	1007	987	987
Total cold run time: 108005 ms
Total hot run time: 40791 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7252	7164	7160	7160
q2	336	239	229	229
q3	2890	2870	2926	2870
q4	2026	1834	1875	1834
q5	5677	5657	5726	5657
q6	223	136	137	136
q7	2259	1806	1858	1806
q8	3340	3531	3539	3531
q9	8791	8864	8854	8854
q10	3620	3566	3540	3540
q11	590	508	504	504
q12	802	618	609	609
q13	8892	3226	3158	3158
q14	298	276	279	276
q15	590	523	517	517
q16	693	660	674	660
q17	1837	1625	1599	1599
q18	8280	7687	7644	7644
q19	1651	1514	1548	1514
q20	2096	1840	1864	1840
q21	5512	5235	5315	5235
q22	1163	1093	1030	1030
Total cold run time: 68818 ms
Total hot run time: 60203 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 198160 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 36fdc318e43c5e37d1074024d88d2e273babb3b0, data reload: false

query1	1276	918	896	896
query2	6246	2096	2085	2085
query3	10966	4442	4461	4442
query4	66546	28790	23424	23424
query5	4982	443	436	436
query6	416	179	179	179
query7	5670	310	305	305
query8	332	222	230	222
query9	9327	2698	2692	2692
query10	474	283	254	254
query11	17558	15258	16049	15258
query12	158	108	104	104
query13	1543	453	424	424
query14	10957	7682	7380	7380
query15	200	182	177	177
query16	7124	492	516	492
query17	1114	589	589	589
query18	1808	325	315	315
query19	229	156	153	153
query20	120	112	109	109
query21	213	106	104	104
query22	4631	4245	4405	4245
query23	34677	34975	33988	33988
query24	6175	2975	2940	2940
query25	524	428	426	426
query26	670	171	165	165
query27	1979	366	357	357
query28	4281	2435	2438	2435
query29	671	475	430	430
query30	250	168	166	166
query31	1017	830	848	830
query32	74	55	56	55
query33	399	271	299	271
query34	900	506	526	506
query35	836	756	722	722
query36	1108	954	963	954
query37	125	73	73	73
query38	4177	4087	4079	4079
query39	1516	1472	1467	1467
query40	196	106	104	104
query41	52	49	50	49
query42	118	102	99	99
query43	550	512	507	507
query44	1155	830	816	816
query45	185	170	165	165
query46	1140	726	680	680
query47	2039	1953	1976	1953
query48	477	375	394	375
query49	745	417	396	396
query50	851	437	425	425
query51	7369	7156	7097	7097
query52	102	87	89	87
query53	251	177	179	177
query54	557	452	455	452
query55	78	73	83	73
query56	263	248	273	248
query57	1225	1113	1112	1112
query58	207	204	204	204
query59	3264	3162	3204	3162
query60	273	243	253	243
query61	103	104	105	104
query62	848	749	713	713
query63	218	183	186	183
query64	1400	678	678	678
query65	3282	3195	3189	3189
query66	700	294	304	294
query67	15733	15590	15645	15590
query68	3938	578	564	564
query69	429	268	270	268
query70	1211	1140	1134	1134
query71	354	270	270	270
query72	6104	4032	4048	4032
query73	747	346	345	345
query74	10366	9071	9206	9071
query75	3333	2688	2676	2676
query76	1867	955	1114	955
query77	493	272	281	272
query78	10512	9562	9592	9562
query79	1419	576	578	576
query80	856	442	426	426
query81	503	237	231	231
query82	1266	121	117	117
query83	164	143	144	143
query84	284	76	79	76
query85	827	322	311	311
query86	342	307	301	301
query87	4524	4403	4308	4308
query88	3718	2426	2401	2401
query89	412	290	298	290
query90	2036	193	188	188
query91	183	147	147	147
query92	64	51	50	50
query93	1875	544	542	542
query94	795	295	301	295
query95	367	259	256	256
query96	609	284	278	278
query97	3346	3145	3191	3145
query98	227	192	200	192
query99	1722	1449	1414	1414
Total cold run time: 319543 ms
Total hot run time: 198160 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.81 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 36fdc318e43c5e37d1074024d88d2e273babb3b0, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.10	0.10
query5	0.52	0.53	0.51
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.06	0.03	0.03
query9	0.56	0.49	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.58	0.59
query14	2.77	2.82	2.84
query15	0.90	0.83	0.82
query16	0.38	0.37	0.38
query17	1.05	1.06	1.07
query18	0.22	0.22	0.22
query19	1.95	1.86	1.97
query20	0.01	0.01	0.02
query21	15.36	0.59	0.55
query22	2.79	2.08	1.63
query23	16.92	1.02	0.80
query24	3.06	1.92	0.99
query25	0.22	0.10	0.13
query26	0.46	0.14	0.13
query27	0.05	0.04	0.05
query28	10.20	1.08	1.06
query29	12.60	3.29	3.25
query30	0.24	0.07	0.06
query31	2.87	0.39	0.39
query32	3.23	0.46	0.45
query33	2.99	3.02	3.01
query34	17.10	4.53	4.50
query35	4.54	4.56	4.55
query36	0.65	0.49	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.78 s
Total hot run time: 32.81 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 1bdffba into branch-3.0 Feb 24, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-47968-branch-3.0 branch February 24, 2025 02:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants