[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322

bobhan1 · 2024-09-03T08:53:17Z

Proposed changes

branch-2.1-pick: #40383
branch-2.0-pick: #40382

doris-robot · 2024-09-03T08:53:21Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

bobhan1 · 2024-09-03T08:53:43Z

run buildall

github-actions · 2024-09-03T09:00:05Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-09-03T09:12:28Z

TPC-H: Total hot run time: 38361 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3fabfb004adb6f87e8618fc49e60d5c47e3001b1, data reload: false

------ Round 1 ----------------------------------
q1	17973	4687	4419	4419
q2	2651	187	175	175
q3	12279	1107	1120	1107
q4	10519	740	768	740
q5	8081	2930	2861	2861
q6	236	137	142	137
q7	971	612	606	606
q8	9346	2067	2069	2067
q9	7017	6575	6557	6557
q10	7012	2263	2276	2263
q11	446	242	245	242
q12	402	225	221	221
q13	19209	3037	3055	3037
q14	275	241	247	241
q15	530	487	490	487
q16	588	505	500	500
q17	998	706	666	666
q18	7335	6822	6883	6822
q19	1380	1032	1023	1023
q20	681	348	341	341
q21	3877	2929	2835	2835
q22	1134	1020	1014	1014
Total cold run time: 112940 ms
Total hot run time: 38361 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4447	4341	4351	4341
q2	392	288	276	276
q3	2923	2685	2669	2669
q4	1996	1676	1694	1676
q5	5402	5408	5424	5408
q6	231	134	131	131
q7	2106	1723	1767	1723
q8	3198	3354	3365	3354
q9	8413	8406	8443	8406
q10	3478	3211	3190	3190
q11	613	508	501	501
q12	792	625	604	604
q13	12449	3095	3130	3095
q14	303	276	275	275
q15	526	474	483	474
q16	623	562	572	562
q17	1787	1507	1475	1475
q18	7765	7385	7283	7283
q19	1652	1626	1547	1547
q20	2113	1816	1815	1815
q21	5557	5138	5209	5138
q22	1122	1046	1001	1001
Total cold run time: 67888 ms
Total hot run time: 54944 ms

zhannngchen · 2024-09-03T09:12:50Z

regression-test/suites/unique_with_mow_p0/partial_update/test_partial_update_delete.groovy

+            sql "set enable_insert_strict=false;"
+            sql "set enable_unique_key_partial_update=true;"
+            sql "sync;"
+            sql "insert into ${tableName3}(k1, __DORIS_DELETE_SIGN__) values(8,1),(4,1),(9,1);"


should consider sequence col

doris-robot · 2024-09-03T09:24:07Z

TPC-DS: Total hot run time: 187988 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3fabfb004adb6f87e8618fc49e60d5c47e3001b1, data reload: false

query1	914	382	367	367
query2	6459	1936	1877	1877
query3	6650	207	218	207
query4	29159	23166	23555	23166
query5	4239	499	497	497
query6	259	174	174	174
query7	4578	287	283	283
query8	242	203	194	194
query9	8507	2448	2431	2431
query10	454	286	272	272
query11	17813	14991	15062	14991
query12	159	105	99	99
query13	1615	364	372	364
query14	9643	7244	6805	6805
query15	248	172	169	169
query16	7970	475	455	455
query17	1590	583	566	566
query18	1913	284	278	278
query19	216	140	142	140
query20	117	107	110	107
query21	212	106	101	101
query22	4597	4389	4241	4241
query23	34058	33505	33353	33353
query24	11055	2933	2838	2838
query25	637	383	394	383
query26	1162	152	154	152
query27	2804	290	279	279
query28	7528	2034	2024	2024
query29	804	417	433	417
query30	304	159	149	149
query31	975	769	778	769
query32	99	53	59	53
query33	746	287	289	287
query34	984	470	485	470
query35	876	736	705	705
query36	1082	915	954	915
query37	161	104	96	96
query38	3994	3900	3901	3900
query39	1441	1398	1380	1380
query40	200	117	112	112
query41	51	48	45	45
query42	114	95	95	95
query43	512	448	470	448
query44	1193	737	740	737
query45	188	166	169	166
query46	1091	753	743	743
query47	1878	1759	1820	1759
query48	378	296	297	296
query49	1114	474	419	419
query50	814	418	406	406
query51	6973	6809	6891	6809
query52	95	84	85	84
query53	254	184	181	181
query54	850	442	443	442
query55	76	76	78	76
query56	269	262	246	246
query57	1166	1090	1064	1064
query58	240	224	229	224
query59	2854	2808	2929	2808
query60	285	267	259	259
query61	105	99	132	99
query62	837	644	658	644
query63	221	184	181	181
query64	4238	677	663	663
query65	3235	3147	3156	3147
query66	1410	333	340	333
query67	15603	15422	15534	15422
query68	3310	594	583	583
query69	410	283	275	275
query70	1199	1151	1120	1120
query71	330	283	271	271
query72	6497	4192	4042	4042
query73	765	339	332	332
query74	9102	8920	8821	8821
query75	3376	2677	2750	2677
query76	1810	1053	1116	1053
query77	492	339	327	327
query78	9684	9138	9740	9138
query79	1550	528	529	528
query80	802	528	530	528
query81	564	238	240	238
query82	1267	149	142	142
query83	242	159	163	159
query84	229	84	85	84
query85	1122	352	334	334
query86	308	303	307	303
query87	4363	4352	4424	4352
query88	2941	2345	2316	2316
query89	384	294	284	284
query90	1952	195	196	195
query91	140	117	115	115
query92	70	54	52	52
query93	1061	523	526	523
query94	808	308	312	308
query95	359	268	268	268
query96	595	275	268	268
query97	3210	3048	3049	3048
query98	219	345	195	195
query99	1472	1341	1247	1247
Total cold run time: 283141 ms
Total hot run time: 187988 ms

doris-robot · 2024-09-03T09:29:42Z

ClickBench: Total hot run time: 33.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3fabfb004adb6f87e8618fc49e60d5c47e3001b1, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.09	0.08
query5	0.49	0.49	0.48
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.53	0.54	0.55
query11	0.15	0.12	0.12
query12	0.16	0.12	0.13
query13	0.63	0.59	0.59
query14	2.10	2.03	2.13
query15	0.82	0.82	0.83
query16	0.35	0.38	0.36
query17	0.99	0.95	0.96
query18	0.21	0.20	0.22
query19	1.82	1.82	1.74
query20	0.01	0.01	0.01
query21	15.41	0.68	0.68
query22	4.09	6.52	3.42
query23	18.25	1.39	1.22
query24	2.15	0.21	0.21
query25	0.15	0.08	0.08
query26	0.26	0.18	0.18
query27	0.07	0.07	0.07
query28	13.27	1.01	1.01
query29	12.65	3.30	3.30
query30	0.24	0.05	0.05
query31	2.87	0.43	0.39
query32	3.27	0.50	0.47
query33	2.98	2.96	2.98
query34	17.12	4.36	4.38
query35	4.48	4.44	4.42
query36	0.66	0.48	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.05
Total cold run time: 110.69 s
Total hot run time: 33.31 s

github-actions · 2024-09-03T09:45:07Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-09-03T11:25:18Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-09-03T12:35:32Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-09-03T13:39:53Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-09-03T13:42:34Z

clang-tidy review says "All clean, LGTM! 👍"

bobhan1 · 2024-09-03T13:44:47Z

run buildall

github-actions · 2024-09-03T13:50:51Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-09-03T14:19:00Z

TeamCity be ut coverage result:
Function Coverage: 36.82% (9383/25485)
Line Coverage: 28.26% (77391/273888)
Region Coverage: 27.67% (39954/144416)
Branch Coverage: 24.31% (20326/83628)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b0d7d0dbe0f9e14eaa0f336afcd1ec974897b01d_b0d7d0dbe0f9e14eaa0f336afcd1ec974897b01d/report/index.html

doris-robot · 2024-09-03T14:32:39Z

TPC-H: Total hot run time: 38033 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0d7d0dbe0f9e14eaa0f336afcd1ec974897b01d, data reload: false

------ Round 1 ----------------------------------
q1	17605	4351	4341	4341
q2	2011	179	176	176
q3	11729	930	1170	930
q4	10517	696	689	689
q5	7755	2819	2827	2819
q6	223	138	137	137
q7	947	613	602	602
q8	9323	2053	2074	2053
q9	7102	6499	6539	6499
q10	7003	2221	2168	2168
q11	447	236	242	236
q12	391	226	232	226
q13	17761	3047	3039	3039
q14	284	232	238	232
q15	533	494	495	494
q16	572	510	510	510
q17	966	669	640	640
q18	7347	7009	7174	7009
q19	1403	1136	1043	1043
q20	691	335	337	335
q21	4056	3034	2864	2864
q22	1107	1035	991	991
Total cold run time: 109773 ms
Total hot run time: 38033 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4346	4319	4260	4260
q2	387	264	282	264
q3	2885	2703	2693	2693
q4	1937	1680	1669	1669
q5	5627	5663	5735	5663
q6	234	138	134	134
q7	2220	1874	1875	1874
q8	3293	3445	3507	3445
q9	8792	8850	8876	8850
q10	3577	3438	3370	3370
q11	640	515	505	505
q12	844	699	664	664
q13	14015	3237	3284	3237
q14	309	296	312	296
q15	553	519	501	501
q16	635	591	591	591
q17	1873	1608	1541	1541
q18	8132	7936	7746	7746
q19	1743	1633	1548	1548
q20	2154	1984	1926	1926
q21	5724	5402	5370	5370
q22	1135	1004	1059	1004
Total cold run time: 71055 ms
Total hot run time: 57151 ms

doris-robot · 2024-09-03T14:44:49Z

TPC-DS: Total hot run time: 192298 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0d7d0dbe0f9e14eaa0f336afcd1ec974897b01d, data reload: false

query1	1261	905	875	875
query2	6451	1962	1882	1882
query3	10603	3879	4102	3879
query4	59344	23930	23256	23256
query5	5447	523	519	519
query6	413	167	171	167
query7	5881	297	292	292
query8	279	196	199	196
query9	8857	2453	2424	2424
query10	486	273	265	265
query11	18094	15003	15407	15003
query12	159	100	104	100
query13	1601	395	376	376
query14	11326	7358	7581	7358
query15	245	174	188	174
query16	7578	482	511	482
query17	1169	560	551	551
query18	2070	297	290	290
query19	298	168	160	160
query20	120	107	105	105
query21	215	110	107	107
query22	4812	4398	4651	4398
query23	34406	33320	33416	33320
query24	5906	2870	2856	2856
query25	526	384	376	376
query26	696	152	152	152
query27	1793	272	276	272
query28	3773	2029	2006	2006
query29	726	411	408	408
query30	232	158	164	158
query31	938	758	783	758
query32	86	53	57	53
query33	434	288	285	285
query34	861	468	468	468
query35	832	735	710	710
query36	1063	906	926	906
query37	150	91	84	84
query38	3962	3866	3872	3866
query39	1426	1378	1382	1378
query40	205	116	116	116
query41	46	46	43	43
query42	113	96	95	95
query43	501	455	456	455
query44	1080	726	735	726
query45	192	163	160	160
query46	1089	732	730	730
query47	1891	1771	1778	1771
query48	386	304	289	289
query49	762	441	464	441
query50	828	418	416	416
query51	6978	6905	6823	6823
query52	100	87	88	87
query53	260	183	182	182
query54	590	466	498	466
query55	72	79	78	78
query56	280	253	257	253
query57	1196	1058	1052	1052
query58	215	236	231	231
query59	2891	3004	2713	2713
query60	294	276	299	276
query61	108	101	97	97
query62	756	660	631	631
query63	217	186	186	186
query64	2843	747	665	665
query65	3226	3163	3171	3163
query66	688	332	332	332
query67	15309	15087	15076	15076
query68	3081	579	575	575
query69	394	278	274	274
query70	1092	1140	1077	1077
query71	349	282	276	276
query72	4940	4027	3957	3957
query73	753	331	329	329
query74	9083	8753	8909	8753
query75	3387	2671	2691	2671
query76	1525	1009	950	950
query77	586	334	331	331
query78	9632	10439	9324	9324
query79	1038	537	534	534
query80	764	504	531	504
query81	466	242	230	230
query82	243	146	147	146
query83	169	148	153	148
query84	261	73	74	73
query85	686	289	278	278
query86	308	300	305	300
query87	4501	4257	4246	4246
query88	3151	2327	2332	2327
query89	384	282	282	282
query90	1980	193	192	192
query91	128	103	101	101
query92	62	51	49	49
query93	1045	525	527	525
query94	713	300	289	289
query95	321	321	255	255
query96	599	269	260	260
query97	3217	3132	3110	3110
query98	225	201	205	201
query99	1529	1263	1267	1263
Total cold run time: 304247 ms
Total hot run time: 192298 ms

doris-robot · 2024-09-03T14:50:26Z

ClickBench: Total hot run time: 32.67 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0d7d0dbe0f9e14eaa0f336afcd1ec974897b01d, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.08	0.08
query5	0.51	0.49	0.49
query6	1.12	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.47
query10	0.54	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.60	0.60	0.59
query14	2.07	2.05	2.06
query15	0.90	0.82	0.83
query16	0.37	0.39	0.38
query17	0.97	1.03	1.02
query18	0.20	0.20	0.21
query19	1.96	1.71	1.76
query20	0.02	0.01	0.01
query21	15.39	0.68	0.67
query22	4.10	6.80	2.51
query23	18.31	1.46	1.36
query24	2.14	0.23	0.21
query25	0.15	0.08	0.09
query26	0.27	0.18	0.18
query27	0.07	0.07	0.08
query28	13.25	1.02	1.02
query29	12.61	3.28	3.30
query30	0.24	0.06	0.05
query31	2.88	0.40	0.40
query32	3.26	0.49	0.47
query33	3.07	2.93	3.03
query34	17.00	4.47	4.39
query35	4.57	4.46	4.46
query36	0.66	0.46	0.48
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.05
Total cold run time: 110.95 s
Total hot run time: 32.67 s

zhannngchen

LGTM

github-actions · 2024-09-04T09:30:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-04T09:30:46Z

PR approved by anyone and no changes requested.

…al update if the row's delete sign is marked #40322 (#40382)

…on-strict mode partial update if the row's delete sign is marked #40322" (#40383) picks #40322

…al update if the row's delete sign is marked (#40322)

wm1581066 added the usercase Important user case type label label Sep 3, 2024

zhannngchen reviewed Sep 3, 2024

View reviewed changes

bobhan1 force-pushed the delete-sign-partial-update-opt branch from 3fabfb0 to 79b2493 Compare September 3, 2024 09:38

bobhan1 added 5 commits September 3, 2024 21:43

opt

63b09ef

fix

48355b3

update

b05329e

fix

4ac4122

fix

b0d7d0d

bobhan1 force-pushed the delete-sign-partial-update-opt branch from 2dfefa0 to b0d7d0d Compare September 3, 2024 13:44

bobhan1 requested a review from zhannngchen September 3, 2024 13:44

zhannngchen approved these changes Sep 4, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 4, 2024

github-actions bot added the reviewed label Sep 4, 2024

Yukang-Lian approved these changes Sep 4, 2024

View reviewed changes

bobhan1 mentioned this pull request Sep 4, 2024

[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322 #40382

Merged

zhannngchen merged commit 3856aca into apache:master Sep 4, 2024

bobhan1 mentioned this pull request Sep 4, 2024

[branch-2.1] Picks "[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322" #40383

Merged

xiaokang pushed a commit that referenced this pull request Sep 5, 2024

[Opt](delete) Skip newly inserted rows check in non-strict mode parti…

5a3beac

…al update if the row's delete sign is marked #40322 (#40382)

xiaokang added the dev/2.0.15-merged label Sep 5, 2024

yiguolei pushed a commit that referenced this pull request Sep 9, 2024

[branch-2.1] Picks "[Opt](delete) Skip newly inserted rows check in n…

44a7eff

…on-strict mode partial update if the row's delete sign is marked #40322" (#40383) picks #40322

yiguolei added the dev/2.1.7-merged label Sep 9, 2024

zhannngchen mentioned this pull request Sep 11, 2024

[opt](partial update) Remove unnecessary lock and refactor some code for partial update #40062

Merged

xiaokang mentioned this pull request Sep 14, 2024

Release Note 2.0.15 #40882

Closed

dataroaring pushed a commit that referenced this pull request Oct 9, 2024

[Opt](delete) Skip newly inserted rows check in non-strict mode parti…

ab96b7d

…al update if the row's delete sign is marked (#40322)

dataroaring added the dev/3.0.3-merged label Oct 12, 2024

[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322

[Opt](delete) Skip newly inserted rows check in non-strict mode partial update if the row's delete sign is marked #40322

Uh oh!

Conversation

bobhan1 commented Sep 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

bobhan1 commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

zhannngchen Sep 3, 2024

Choose a reason for hiding this comment

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

bobhan1 commented Sep 3, 2024

Uh oh!

github-actions bot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

doris-robot commented Sep 3, 2024

Uh oh!

zhannngchen left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 4, 2024

Uh oh!

github-actions bot commented Sep 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

bobhan1 commented Sep 3, 2024 •

edited

Loading