Skip to content

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Feb 20, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

if set global enable_unique_key_partial_update = true, group commit will failed:

curl --location-trusted -u root: -T 1.csv -H "group_commit:sync_mode"  -H "column_separator:,"  http://127.0.0.1:9082/api/db/t/_stream_load
{
    "Label": "group_commit_b642d0ae0fb0c318_78d44d3b051cfc84",
    "GroupCommit": true,
    "Status": "Fail",
    "Message": "[CANCELLED]cancel group_commit, label=group_commit_b642d0ae0fb0c318_78d44d3b051cfc84, status=[INTERNAL_ERROR]tablet error: [INVALID_ARGUMENT]illegal partial update block columns: 3, num key columns: 1, total schema columns: 3, host: 172.21.16.12",

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Feb 20, 2025

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 20, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 31222 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b217de6dbd3da21a358ae39e8ef72d364924dc52, data reload: false

------ Round 1 ----------------------------------
q1	17595	5430	5106	5106
q2	2048	296	170	170
q3	10420	1343	702	702
q4	10212	1001	537	537
q5	7496	2472	2242	2242
q6	188	169	136	136
q7	894	765	610	610
q8	9308	1323	1182	1182
q9	4930	4554	4606	4554
q10	6821	2314	1888	1888
q11	506	280	251	251
q12	347	373	225	225
q13	17790	3643	3035	3035
q14	224	226	203	203
q15	522	456	451	451
q16	606	593	561	561
q17	564	879	333	333
q18	6757	6154	6120	6120
q19	1212	951	531	531
q20	299	326	193	193
q21	2733	2118	1881	1881
q22	363	334	311	311
Total cold run time: 101835 ms
Total hot run time: 31222 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5080	5089	5071	5071
q2	231	324	228	228
q3	2138	2671	2272	2272
q4	1409	1817	1365	1365
q5	4201	4126	4144	4126
q6	207	178	124	124
q7	1861	1817	1663	1663
q8	2556	2583	2535	2535
q9	7377	7151	7041	7041
q10	3004	3214	2724	2724
q11	566	508	487	487
q12	718	770	644	644
q13	3477	3911	3282	3282
q14	283	291	261	261
q15	500	456	465	456
q16	632	675	630	630
q17	1141	1580	1326	1326
q18	7711	7303	7336	7303
q19	793	810	902	810
q20	2001	2022	1843	1843
q21	5316	4862	4737	4737
q22	606	593	557	557
Total cold run time: 51808 ms
Total hot run time: 49485 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184342 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b217de6dbd3da21a358ae39e8ef72d364924dc52, data reload: false

query1	976	383	377	377
query2	6534	1918	1874	1874
query3	6787	212	218	212
query4	26632	23604	23694	23604
query5	4358	650	528	528
query6	307	201	190	190
query7	4600	496	300	300
query8	310	254	233	233
query9	8619	2528	2530	2528
query10	469	305	256	256
query11	15717	15153	15036	15036
query12	158	106	103	103
query13	1670	511	405	405
query14	9691	6075	6057	6057
query15	215	195	169	169
query16	7329	633	470	470
query17	1212	703	566	566
query18	1953	406	313	313
query19	216	208	156	156
query20	123	119	116	116
query21	209	121	106	106
query22	4057	4461	4201	4201
query23	33898	32885	33037	32885
query24	7698	2386	2379	2379
query25	518	453	374	374
query26	1229	275	152	152
query27	2139	491	330	330
query28	3900	2413	2391	2391
query29	718	534	439	439
query30	229	184	156	156
query31	927	861	781	781
query32	72	63	62	62
query33	568	392	300	300
query34	756	837	523	523
query35	782	810	729	729
query36	949	979	899	899
query37	130	92	74	74
query38	4123	4155	4129	4129
query39	1444	1385	1405	1385
query40	207	116	106	106
query41	56	56	52	52
query42	132	101	98	98
query43	503	511	487	487
query44	1262	792	799	792
query45	175	167	167	167
query46	853	1013	653	653
query47	1759	1824	1704	1704
query48	387	408	308	308
query49	785	516	412	412
query50	669	737	408	408
query51	4192	4161	4102	4102
query52	107	118	97	97
query53	229	258	184	184
query54	504	476	421	421
query55	82	83	79	79
query56	263	256	267	256
query57	1110	1148	1086	1086
query58	251	225	240	225
query59	2660	2770	2819	2770
query60	294	264	255	255
query61	124	120	116	116
query62	794	706	669	669
query63	243	193	188	188
query64	4338	1038	652	652
query65	3211	3116	3110	3110
query66	1160	405	312	312
query67	15857	15551	15326	15326
query68	6339	795	530	530
query69	471	305	259	259
query70	1166	1188	1146	1146
query71	387	292	255	255
query72	5721	3578	3632	3578
query73	703	735	355	355
query74	9116	9352	9052	9052
query75	3146	3166	2689	2689
query76	3157	1152	721	721
query77	482	433	284	284
query78	10074	9931	9351	9351
query79	3194	812	600	600
query80	704	525	456	456
query81	523	277	243	243
query82	1228	124	94	94
query83	178	173	152	152
query84	235	95	71	71
query85	850	348	316	316
query86	395	307	310	307
query87	4449	4538	4535	4535
query88	4313	2200	2211	2200
query89	398	327	283	283
query90	1846	197	196	196
query91	139	141	109	109
query92	85	66	56	56
query93	2653	1003	579	579
query94	716	379	302	302
query95	355	271	258	258
query96	477	550	275	275
query97	2749	2820	2768	2768
query98	228	217	208	208
query99	1312	1386	1291	1291
Total cold run time: 272218 ms
Total hot run time: 184342 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b217de6dbd3da21a358ae39e8ef72d364924dc52, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.41	0.40	0.39
query6	1.17	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.58	0.58	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	2.70	2.72	2.72
query15	0.92	0.85	0.84
query16	0.39	0.38	0.38
query17	1.02	1.01	1.01
query18	0.21	0.20	0.19
query19	1.90	1.82	1.92
query20	0.02	0.01	0.01
query21	15.38	0.92	0.54
query22	0.76	1.19	0.63
query23	14.93	1.38	0.60
query24	9.46	2.03	0.40
query25	0.39	0.24	0.14
query26	0.83	0.17	0.15
query27	0.05	0.04	0.04
query28	7.93	0.87	0.44
query29	12.55	3.95	3.25
query30	0.26	0.10	0.06
query31	2.81	0.58	0.38
query32	3.23	0.55	0.46
query33	2.93	3.00	3.04
query34	15.66	5.12	4.50
query35	4.53	4.49	4.54
query36	0.67	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 105.66 s
Total hot run time: 30.05 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 43.83% (11639/26556)
Line Coverage: 33.76% (97600/289093)
Region Coverage: 32.85% (49957/152064)
Branch Coverage: 28.57% (25113/87902)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b217de6dbd3da21a358ae39e8ef72d364924dc52_b217de6dbd3da21a358ae39e8ef72d364924dc52/report/index.html

@zhannngchen
Copy link
Contributor

run buildall

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-H: Total hot run time: 31588 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b217de6dbd3da21a358ae39e8ef72d364924dc52, data reload: false

------ Round 1 ----------------------------------
q1	17583	5311	5097	5097
q2	2044	317	191	191
q3	10413	1286	758	758
q4	10274	1008	562	562
q5	7909	2434	2269	2269
q6	191	170	137	137
q7	901	740	592	592
q8	9294	1289	1151	1151
q9	4818	4614	4642	4614
q10	6838	2332	1953	1953
q11	518	292	262	262
q12	348	357	224	224
q13	17901	3694	3117	3117
q14	229	235	208	208
q15	505	460	451	451
q16	631	603	583	583
q17	562	862	336	336
q18	6569	6230	6102	6102
q19	1431	963	547	547
q20	324	319	187	187
q21	2738	2202	1947	1947
q22	366	339	300	300
Total cold run time: 102387 ms
Total hot run time: 31588 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5166	5153	5152	5152
q2	249	333	233	233
q3	2143	2691	2299	2299
q4	1434	1831	1376	1376
q5	4219	4130	4137	4130
q6	219	165	127	127
q7	1846	1854	1690	1690
q8	2598	2571	2677	2571
q9	7208	7150	7158	7150
q10	3014	3237	2821	2821
q11	608	519	493	493
q12	674	830	627	627
q13	3383	3928	3251	3251
q14	280	300	274	274
q15	507	464	448	448
q16	650	694	645	645
q17	1131	1534	1413	1413
q18	7608	7337	7303	7303
q19	793	799	984	799
q20	1980	2053	1871	1871
q21	5451	5108	4651	4651
q22	604	567	545	545
Total cold run time: 51765 ms
Total hot run time: 49869 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189637 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b217de6dbd3da21a358ae39e8ef72d364924dc52, data reload: false

query1	1302	953	937	937
query2	6236	1873	1829	1829
query3	10984	4371	4574	4371
query4	56563	25043	23341	23341
query5	5041	532	498	498
query6	414	211	186	186
query7	5271	516	295	295
query8	317	248	249	248
query9	6879	2517	2539	2517
query10	422	295	249	249
query11	15193	15142	14923	14923
query12	158	109	107	107
query13	1250	505	379	379
query14	10087	6179	7016	6179
query15	205	191	182	182
query16	7069	652	502	502
query17	1078	753	592	592
query18	1533	426	322	322
query19	200	205	187	187
query20	128	121	135	121
query21	211	127	108	108
query22	4265	4534	4317	4317
query23	33868	33340	33279	33279
query24	5733	2429	2455	2429
query25	451	480	435	435
query26	684	288	153	153
query27	1984	532	345	345
query28	2799	2499	2473	2473
query29	565	576	432	432
query30	215	197	158	158
query31	902	907	842	842
query32	75	64	57	57
query33	437	361	322	322
query34	739	879	520	520
query35	814	840	749	749
query36	949	1022	919	919
query37	125	99	75	75
query38	4297	4377	4341	4341
query39	1522	1437	1424	1424
query40	201	118	109	109
query41	55	52	51	51
query42	124	105	103	103
query43	507	509	477	477
query44	1308	808	799	799
query45	179	172	168	168
query46	865	1052	678	678
query47	1826	1885	1740	1740
query48	385	409	326	326
query49	698	512	417	417
query50	728	748	432	432
query51	4223	4277	4235	4235
query52	109	106	99	99
query53	248	261	191	191
query54	483	493	411	411
query55	90	90	83	83
query56	249	264	264	264
query57	1143	1193	1149	1149
query58	254	249	244	244
query59	2611	2843	2727	2727
query60	318	297	283	283
query61	147	142	171	142
query62	723	747	659	659
query63	230	190	194	190
query64	1458	1064	683	683
query65	3202	3183	3149	3149
query66	696	395	310	310
query67	15580	15387	15480	15387
query68	5378	797	514	514
query69	529	351	262	262
query70	1231	1131	1078	1078
query71	432	304	262	262
query72	6321	3582	3805	3582
query73	964	743	353	353
query74	8998	9188	8852	8852
query75	3490	3148	2717	2717
query76	3841	1163	723	723
query77	533	378	275	275
query78	9977	10342	9242	9242
query79	2976	826	595	595
query80	761	521	446	446
query81	529	276	233	233
query82	638	132	91	91
query83	306	165	153	153
query84	293	93	72	72
query85	779	347	292	292
query86	428	301	286	286
query87	4542	4399	4348	4348
query88	3695	2216	2184	2184
query89	400	307	289	289
query90	1882	189	191	189
query91	132	147	106	106
query92	69	62	55	55
query93	2332	990	573	573
query94	648	415	297	297
query95	344	266	257	257
query96	507	547	277	277
query97	2782	2849	2740	2740
query98	242	209	205	205
query99	1414	1397	1240	1240
Total cold run time: 296998 ms
Total hot run time: 189637 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 43.83% (11641/26557)
Line Coverage: 33.74% (97555/289098)
Region Coverage: 32.85% (49949/152069)
Branch Coverage: 28.56% (25102/87904)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b217de6dbd3da21a358ae39e8ef72d364924dc52_b217de6dbd3da21a358ae39e8ef72d364924dc52/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 30.1 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b217de6dbd3da21a358ae39e8ef72d364924dc52, data reload: false

query1	0.04	0.03	0.04
query2	0.09	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.40	0.42	0.41
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.53	0.53
query10	0.57	0.57	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.12
query13	0.61	0.60	0.59
query14	2.73	2.69	2.71
query15	0.93	0.86	0.85
query16	0.38	0.38	0.36
query17	1.04	1.02	1.03
query18	0.21	0.19	0.19
query19	1.94	1.85	2.00
query20	0.01	0.02	0.01
query21	15.37	0.86	0.53
query22	0.77	1.09	0.68
query23	15.04	1.39	0.60
query24	6.99	1.83	0.30
query25	0.30	0.10	0.10
query26	0.65	0.16	0.14
query27	0.06	0.05	0.05
query28	9.20	0.93	0.44
query29	12.57	4.02	3.29
query30	0.25	0.08	0.06
query31	2.83	0.59	0.40
query32	3.22	0.55	0.47
query33	2.95	3.03	3.03
query34	15.66	5.12	4.47
query35	4.57	4.52	4.55
query36	0.66	0.48	0.48
query37	0.10	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.13
query41	0.09	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.51 s
Total hot run time: 30.1 s

@dataroaring dataroaring merged commit fc404d7 into apache:master Feb 24, 2025
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Feb 24, 2025
…e_key_partial_update (#48120)

if `set global enable_unique_key_partial_update = true`, group commit
will failed:
```
curl --location-trusted -u root: -T 1.csv -H "group_commit:sync_mode"  -H "column_separator:,"  http://127.0.0.1:9082/api/db/t/_stream_load
{
    "Label": "group_commit_b642d0ae0fb0c318_78d44d3b051cfc84",
    "GroupCommit": true,
    "Status": "Fail",
    "Message": "[CANCELLED]cancel group_commit, label=group_commit_b642d0ae0fb0c318_78d44d3b051cfc84, status=[INTERNAL_ERROR]tablet error: [INVALID_ARGUMENT]illegal partial update block columns: 3, num key columns: 1, total schema columns: 3, host: 172.21.16.12",
```
dataroaring pushed a commit that referenced this pull request Feb 25, 2025
dataroaring pushed a commit that referenced this pull request Feb 26, 2025
…enable_unique_key_partial_update #48120 (#48221)

Cherry-picked from #48120

Co-authored-by: meiyi <meiyi@selectdb.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 28, 2025
…e_key_partial_update (apache#48120)

if `set global enable_unique_key_partial_update = true`, group commit
will failed:
```
curl --location-trusted -u root: -T 1.csv -H "group_commit:sync_mode"  -H "column_separator:,"  http://127.0.0.1:9082/api/db/t/_stream_load
{
    "Label": "group_commit_b642d0ae0fb0c318_78d44d3b051cfc84",
    "GroupCommit": true,
    "Status": "Fail",
    "Message": "[CANCELLED]cancel group_commit, label=group_commit_b642d0ae0fb0c318_78d44d3b051cfc84, status=[INTERNAL_ERROR]tablet error: [INVALID_ARGUMENT]illegal partial update block columns: 3, num key columns: 1, total schema columns: 3, host: 172.21.16.12",
```
@yiguolei yiguolei mentioned this pull request Mar 25, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…e_key_partial_update (apache#48120)

if `set global enable_unique_key_partial_update = true`, group commit
will failed:
```
curl --location-trusted -u root: -T 1.csv -H "group_commit:sync_mode"  -H "column_separator:,"  http://127.0.0.1:9082/api/db/t/_stream_load
{
    "Label": "group_commit_b642d0ae0fb0c318_78d44d3b051cfc84",
    "GroupCommit": true,
    "Status": "Fail",
    "Message": "[CANCELLED]cancel group_commit, label=group_commit_b642d0ae0fb0c318_78d44d3b051cfc84, status=[INTERNAL_ERROR]tablet error: [INVALID_ARGUMENT]illegal partial update block columns: 3, num key columns: 1, total schema columns: 3, host: 172.21.16.12",
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants