Skip to content

branch-3.0: [fix](group commit) group commit failed if enable global enable_unique_key_partial_update #48120 #48221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 26, 2025
Merged

branch-3.0: [fix](group commit) group commit failed if enable global enable_unique_key_partial_update #48120 #48221

dataroaring merged 1 commit into from
Feb 26, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Feb 24, 2025

Cherry-picked from #48120

@mymeiyi
[fix](group commit) group commit failed if enable global enable_uniqu…
0ec23cd 
…e_key_partial_update (#48120)

if `set global enable_unique_key_partial_update = true`, group commit
will failed:
```
curl --location-trusted -u root: -T 1.csv -H "group_commit:sync_mode"  -H "column_separator:,"  http://127.0.0.1:9082/api/db/t/_stream_load
{
    "Label": "group_commit_b642d0ae0fb0c318_78d44d3b051cfc84",
    "GroupCommit": true,
    "Status": "Fail",
    "Message": "[CANCELLED]cancel group_commit, label=group_commit_b642d0ae0fb0c318_78d44d3b051cfc84, status=[INTERNAL_ERROR]tablet error: [INVALID_ARGUMENT]illegal partial update block columns: 3, num key columns: 1, total schema columns: 3, host: 172.21.16.12",
```
@dataroaring dataroaring reopened this Feb 25, 2025
@dataroaring
Copy link
Contributor

dataroaring commented Feb 25, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 25, 2025

TPC-H: Total hot run time: 40107 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ec23cdff55d026cb46eb1f82267159e9d1a2c22, data reload: false

------ Round 1 ----------------------------------
q1	17581	6705	6606	6606
q2	2045	180	174	174
q3	10619	1063	1196	1063
q4	10572	773	737	737
q5	7749	2838	2791	2791
q6	219	139	130	130
q7	980	608	614	608
q8	9348	1902	2036	1902
q9	6594	6432	6460	6432
q10	7076	2288	2317	2288
q11	475	269	276	269
q12	405	216	220	216
q13	18215	3073	3103	3073
q14	251	219	213	213
q15	513	464	458	458
q16	677	582	585	582
q17	984	578	560	560
q18	7463	6604	6623	6604
q19	1382	1059	1048	1048
q20	489	214	199	199
q21	3967	3154	3229	3154
q22	1075	1000	1019	1000
Total cold run time: 108679 ms
Total hot run time: 40107 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6566	6561	6576	6561
q2	333	243	231	231
q3	2899	2785	2919	2785
q4	2046	1773	1808	1773
q5	5731	5747	5733	5733
q6	213	125	126	125
q7	2190	1772	1833	1772
q8	3390	3546	3466	3466
q9	8846	8853	8908	8853
q10	3557	3541	3511	3511
q11	588	510	490	490
q12	804	607	615	607
q13	9710	3223	3173	3173
q14	300	274	295	274
q15	515	470	462	462
q16	677	665	646	646
q17	1864	1632	1594	1594
q18	8311	7865	7799	7799
q19	1661	1447	1571	1447
q20	2045	1868	1860	1860
q21	5548	5296	5327	5296
q22	1160	1070	1028	1028
Total cold run time: 68954 ms
Total hot run time: 59486 ms

@doris-robot
Copy link

doris-robot commented Feb 25, 2025

TPC-DS: Total hot run time: 198255 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0ec23cdff55d026cb46eb1f82267159e9d1a2c22, data reload: false

query1	1287	899	904	899
query2	6246	2037	2060	2037
query3	10823	4466	4322	4322
query4	60710	28505	23723	23723
query5	5135	486	445	445
query6	403	182	183	182
query7	5496	322	313	313
query8	308	233	222	222
query9	8718	2602	2570	2570
query10	447	288	257	257
query11	17741	15279	15634	15279
query12	161	105	104	104
query13	1437	441	429	429
query14	10873	7503	7188	7188
query15	202	183	176	176
query16	7109	496	489	489
query17	1146	596	593	593
query18	2002	315	312	312
query19	211	166	163	163
query20	120	113	115	113
query21	205	107	103	103
query22	4849	4535	4729	4535
query23	34554	34252	33971	33971
query24	6264	2942	3005	2942
query25	557	454	423	423
query26	657	170	170	170
query27	1778	364	373	364
query28	4063	2460	2460	2460
query29	670	447	463	447
query30	251	161	156	156
query31	1002	822	840	822
query32	64	54	56	54
query33	418	292	293	292
query34	934	498	502	498
query35	852	715	724	715
query36	1095	981	974	974
query37	112	65	70	65
query38	4039	4023	4053	4023
query39	1528	1618	1457	1457
query40	202	97	101	97
query41	49	49	51	49
query42	126	111	104	104
query43	533	501	480	480
query44	1206	852	812	812
query45	185	170	174	170
query46	1141	752	724	724
query47	2034	1944	1971	1944
query48	483	407	381	381
query49	721	404	398	398
query50	875	429	422	422
query51	7378	7140	7269	7140
query52	101	88	98	88
query53	258	186	195	186
query54	593	469	459	459
query55	86	76	80	76
query56	264	248	252	248
query57	1311	1186	1143	1143
query58	216	211	211	211
query59	3185	3031	2908	2908
query60	277	267	263	263
query61	117	114	117	114
query62	768	664	687	664
query63	218	189	193	189
query64	1371	657	631	631
query65	3278	3202	3216	3202
query66	662	296	308	296
query67	16101	15747	15635	15635
query68	4328	573	561	561
query69	429	271	271	271
query70	1082	1110	1111	1110
query71	332	262	257	257
query72	6406	4027	4089	4027
query73	747	344	353	344
query74	10149	9336	9297	9297
query75	3324	2652	2656	2652
query76	1909	1088	1072	1072
query77	484	270	274	270
query78	10563	9646	9577	9577
query79	1709	604	590	590
query80	1009	425	413	413
query81	523	239	235	235
query82	181	92	88	88
query83	163	142	153	142
query84	285	81	77	77
query85	955	301	298	298
query86	384	301	300	300
query87	4400	4247	4319	4247
query88	4159	2401	2394	2394
query89	408	288	290	288
query90	1945	189	196	189
query91	188	149	147	147
query92	62	48	51	48
query93	2386	545	546	545
query94	760	297	289	289
query95	368	267	261	261
query96	630	280	285	280
query97	3302	3179	3195	3179
query98	215	206	200	200
query99	1565	1325	1286	1286
Total cold run time: 313556 ms
Total hot run time: 198255 ms

@doris-robot
Copy link

doris-robot commented Feb 25, 2025

ClickBench: Total hot run time: 31.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0ec23cdff55d026cb46eb1f82267159e9d1a2c22, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.54	0.51	0.52
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.05	0.04
query9	0.56	0.51	0.49
query10	0.55	0.54	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.60	0.58
query14	2.73	2.73	2.83
query15	0.89	0.83	0.82
query16	0.38	0.39	0.38
query17	1.02	1.06	1.05
query18	0.24	0.22	0.22
query19	1.83	1.79	1.99
query20	0.01	0.02	0.01
query21	15.35	0.59	0.58
query22	2.71	3.10	1.52
query23	16.92	1.19	0.84
query24	3.03	0.58	1.47
query25	0.21	0.18	0.21
query26	0.35	0.13	0.14
query27	0.04	0.05	0.05
query28	10.39	0.51	0.44
query29	12.59	3.26	3.28
query30	0.24	0.06	0.06
query31	2.85	0.39	0.37
query32	3.28	0.46	0.46
query33	2.91	3.01	2.99
query34	17.00	4.50	4.48
query35	4.54	4.50	4.54
query36	0.66	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.34 s
Total hot run time: 31.54 s

dataroaring
dataroaring approved these changes Feb 26, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 2679359 into branch-3.0 Feb 26, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48120-branch-3.0 branch February 26, 2025 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@dataroaring @doris-robot @mymeiyi