[branch-2.0] Picks "[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958" #40015

bobhan1 · 2024-08-28T02:35:50Z

Proposed changes

picks #39958

doris-robot · 2024-08-28T02:35:55Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

bobhan1 · 2024-08-28T02:35:58Z

run buildall

github-actions · 2024-08-28T02:44:05Z

clang-tidy review says "All clean, LGTM! 👍"

… column for merge-on-write table (#39958) ## Proposed changes Currently, `BaseTablet::lookup_row_key()` use tablet_meta's schema to decide whether a tablet has sequence column. But users can use `ALTER TABLE tbl ENABLE FEATURE "SEQUENCE_LOAD" WITH ...` to add hidden sequence column on MOW table. This is a light schema change which will not change the BE's tablet meta, thus causing wrong behavior in `BaseTablet::lookup_row_key()`. This PR use the schema of the current load, which is the latest schema, to decide whether a tablet has sequence column and correct the lookup procedure in `BaseTablet::lookup_row_key()` and `Segment::lookup_row_key()`. branch-2.1-pick: #40010 branch-2.0-pick: #40015

doris-robot · 2024-08-28T03:20:33Z

TPC-H: Total hot run time: 49920 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 26d178a4f69000840a97be5c31d7abfb56f75f86, data reload: false

------ Round 1 ----------------------------------
q1	17671	4420	4336	4336
q2	2070	154	142	142
q3	10444	1913	1974	1913
q4	10340	1248	1332	1248
q5	8643	3893	3936	3893
q6	234	124	124	124
q7	2044	1606	1623	1606
q8	9279	2753	2711	2711
q9	11066	10608	10490	10490
q10	8648	3519	3478	3478
q11	429	248	250	248
q12	470	295	294	294
q13	18325	4022	4011	4011
q14	344	325	328	325
q15	508	451	449	449
q16	676	568	585	568
q17	1144	986	971	971
q18	7242	6785	6927	6785
q19	1698	1576	1542	1542
q20	527	305	291	291
q21	4455	4115	4101	4101
q22	486	399	394	394
Total cold run time: 116743 ms
Total hot run time: 49920 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4344	4330	4330
q2	325	228	220	220
q3	4175	4145	4127	4127
q4	2766	2752	2730	2730
q5	7151	7116	7108	7108
q6	238	119	119	119
q7	3244	2786	2819	2786
q8	4354	4465	4500	4465
q9	17388	17130	16971	16971
q10	4250	4279	4229	4229
q11	774	672	674	672
q12	1030	859	848	848
q13	6741	3770	3735	3735
q14	460	432	417	417
q15	490	465	458	458
q16	724	682	675	675
q17	3784	3818	3856	3818
q18	8807	8729	8875	8729
q19	1710	1705	1702	1702
q20	2350	2125	2101	2101
q21	8488	8403	8351	8351
q22	1051	941	924	924
Total cold run time: 84608 ms
Total hot run time: 79515 ms

doris-robot · 2024-08-28T03:21:24Z

TeamCity be ut coverage result:
Function Coverage: 37.85% (8138/21499)
Line Coverage: 29.59% (66972/226336)
Region Coverage: 29.08% (34553/118827)
Branch Coverage: 24.98% (17801/71262)
Coverage Report: http://coverage.selectdb-in.cc/coverage/26d178a4f69000840a97be5c31d7abfb56f75f86_26d178a4f69000840a97be5c31d7abfb56f75f86/report/index.html

doris-robot · 2024-08-28T03:33:15Z

TPC-DS: Total hot run time: 202423 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 26d178a4f69000840a97be5c31d7abfb56f75f86, data reload: false

query1	929	386	415	386
query2	6553	2077	2128	2077
query3	6918	212	199	199
query4	20884	18035	17994	17994
query5	19731	6567	6528	6528
query6	295	222	237	222
query7	4153	307	308	307
query8	257	300	242	242
query9	3184	2705	2659	2659
query10	429	308	292	292
query11	11433	10820	10821	10820
query12	115	83	76	76
query13	5611	659	665	659
query14	17830	13498	13396	13396
query15	360	238	228	228
query16	6454	288	265	265
query17	1741	1453	891	891
query18	2306	432	414	414
query19	213	150	151	150
query20	76	82	80	80
query21	193	108	97	97
query22	5312	5171	5086	5086
query23	32636	31767	31971	31767
query24	6939	6507	6505	6505
query25	519	422	431	422
query26	528	169	164	164
query27	1896	296	304	296
query28	6089	2307	2286	2286
query29	2765	2831	2663	2663
query30	243	171	175	171
query31	930	764	733	733
query32	69	65	59	59
query33	405	281	253	253
query34	848	467	476	467
query35	1113	916	939	916
query36	1189	1136	1205	1136
query37	94	63	62	62
query38	3100	2968	3005	2968
query39	1363	1331	1324	1324
query40	210	97	109	97
query41	39	38	37	37
query42	88	84	88	84
query43	664	616	632	616
query44	1143	713	712	712
query45	248	235	241	235
query46	1232	974	986	974
query47	2007	1712	1712	1712
query48	960	696	683	683
query49	631	389	374	374
query50	848	617	595	595
query51	4796	4725	4638	4638
query52	88	87	78	78
query53	452	321	329	321
query54	2660	2476	2457	2457
query55	89	74	86	74
query56	249	207	224	207
query57	1242	1047	1088	1047
query58	224	206	188	188
query59	3420	3436	3260	3260
query60	218	206	212	206
query61	101	99	119	99
query62	853	503	458	458
query63	486	349	347	347
query64	2510	1575	1416	1416
query65	3663	3554	3553	3553
query66	844	376	386	376
query67	15471	15546	14949	14949
query68	9683	654	643	643
query69	572	355	357	355
query70	1612	1539	1289	1289
query71	414	305	331	305
query72	6569	3535	3522	3522
query73	751	318	323	318
query74	6275	5873	5822	5822
query75	5410	3855	3811	3811
query76	6079	1170	1214	1170
query77	1034	253	259	253
query78	12880	12811	12032	12032
query79	11127	632	643	632
query80	734	416	406	406
query81	490	235	240	235
query82	1087	97	98	97
query83	162	134	132	132
query84	258	75	70	70
query85	846	325	329	325
query86	335	295	294	294
query87	3236	2985	3079	2985
query88	4873	2328	2329	2328
query89	448	294	320	294
query90	1975	216	205	205
query91	160	128	139	128
query92	64	52	54	52
query93	5811	588	618	588
query94	735	212	213	212
query95	1124	1081	1071	1071
query96	647	331	329	329
query97	6615	6478	6562	6478
query98	191	172	166	166
query99	2791	823	846	823
Total cold run time: 316127 ms
Total hot run time: 202423 ms

doris-robot · 2024-08-28T03:38:42Z

ClickBench: Total hot run time: 31.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 26d178a4f69000840a97be5c31d7abfb56f75f86, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.80	0.08	0.07
query5	0.53	0.53	0.52
query6	1.27	0.62	0.62
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.52	0.50	0.49
query10	0.53	0.53	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.63	0.62	0.61
query14	0.76	0.80	0.78
query15	0.77	0.76	0.76
query16	0.36	0.36	0.36
query17	0.94	1.00	1.03
query18	0.22	0.25	0.23
query19	1.96	1.83	1.85
query20	0.01	0.01	0.01
query21	15.50	0.55	0.55
query22	2.03	2.22	1.87
query23	16.99	1.00	0.94
query24	6.80	0.95	1.48
query25	0.38	0.11	0.06
query26	0.70	0.16	0.17
query27	0.04	0.03	0.03
query28	6.04	0.73	0.76
query29	12.64	2.31	2.30
query30	0.62	0.56	0.52
query31	2.80	0.39	0.37
query32	3.41	0.49	0.51
query33	3.06	3.09	3.05
query34	15.25	4.84	4.80
query35	4.88	4.86	4.87
query36	1.04	1.01	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 103.48 s
Total hot run time: 31.04 s

doris-robot · 2024-08-28T03:41:57Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 26d178a4f69000840a97be5c31d7abfb56f75f86 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.2 seconds inserted 10000000 Rows, about 450K ops/s

bobhan1 · 2024-08-28T05:11:42Z

run buildall

github-actions · 2024-08-28T05:19:28Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-08-28T05:34:08Z

TPC-H: Total hot run time: 49469 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b876d4c89629d19a514aeaf043388270a278df64, data reload: false

------ Round 1 ----------------------------------
q1	18012	4452	4364	4364
q2	2070	158	142	142
q3	10536	1915	1945	1915
q4	10343	1241	1301	1241
q5	8449	3890	3861	3861
q6	227	121	123	121
q7	2054	1589	1623	1589
q8	9555	2757	2709	2709
q9	13779	10239	10182	10182
q10	8661	3547	3504	3504
q11	417	241	250	241
q12	467	296	293	293
q13	18332	3975	4033	3975
q14	350	327	335	327
q15	505	461	454	454
q16	649	573	569	569
q17	1152	937	927	927
q18	7230	6764	6868	6764
q19	1712	1598	1499	1499
q20	556	312	302	302
q21	4442	4099	4091	4091
q22	502	408	399	399
Total cold run time: 120000 ms
Total hot run time: 49469 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4290	4313	4290
q2	321	228	230	228
q3	4184	4137	4136	4136
q4	2744	2738	2746	2738
q5	7235	7142	7070	7070
q6	240	124	122	122
q7	3273	2822	2815	2815
q8	4416	4499	4485	4485
q9	16910	16919	16684	16684
q10	4219	4272	4287	4272
q11	749	703	704	703
q12	1040	874	873	873
q13	7057	3748	3750	3748
q14	447	441	422	422
q15	491	465	453	453
q16	746	697	681	681
q17	3777	3896	3845	3845
q18	8900	8826	8784	8784
q19	1740	1636	1667	1636
q20	2363	2163	2142	2142
q21	8531	8605	8417	8417
q22	1010	943	964	943
Total cold run time: 84759 ms
Total hot run time: 79487 ms

doris-robot · 2024-08-28T05:34:26Z

TeamCity be ut coverage result:
Function Coverage: 37.86% (8140/21499)
Line Coverage: 29.60% (66989/226336)
Region Coverage: 29.09% (34565/118827)
Branch Coverage: 24.99% (17809/71262)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b876d4c89629d19a514aeaf043388270a278df64_b876d4c89629d19a514aeaf043388270a278df64/report/index.html

doris-robot · 2024-08-28T05:46:43Z

TPC-DS: Total hot run time: 202347 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b876d4c89629d19a514aeaf043388270a278df64, data reload: false

query1	924	393	419	393
query2	6529	2368	1984	1984
query3	6919	208	197	197
query4	20211	18087	18014	18014
query5	19754	6537	6579	6537
query6	301	215	218	215
query7	4158	303	314	303
query8	248	271	239	239
query9	3192	2730	2654	2654
query10	410	307	298	298
query11	11398	10826	10842	10826
query12	124	71	76	71
query13	5604	652	651	651
query14	17883	13975	13675	13675
query15	362	217	226	217
query16	6468	296	264	264
query17	1691	1460	875	875
query18	2317	415	407	407
query19	210	150	154	150
query20	79	80	81	80
query21	194	99	92	92
query22	5296	5070	5147	5070
query23	32696	31896	31937	31896
query24	7656	6491	6478	6478
query25	528	442	439	439
query26	873	162	158	158
query27	2204	299	299	299
query28	6054	2294	2259	2259
query29	2921	2699	2782	2699
query30	250	172	165	165
query31	941	742	749	742
query32	69	67	61	61
query33	404	263	254	254
query34	849	472	495	472
query35	1128	931	885	885
query36	1328	1138	1043	1043
query37	92	60	61	60
query38	3048	2924	2992	2924
query39	1388	1315	1349	1315
query40	309	89	95	89
query41	40	38	37	37
query42	89	90	86	86
query43	625	667	573	573
query44	1127	724	721	721
query45	248	232	231	231
query46	1238	967	972	967
query47	1869	1777	1764	1764
query48	976	681	658	658
query49	662	372	374	372
query50	850	609	638	609
query51	4790	4664	4657	4657
query52	97	82	81	81
query53	448	330	320	320
query54	2653	2474	2518	2474
query55	90	76	96	76
query56	227	212	215	212
query57	1187	1117	1116	1116
query58	212	193	197	193
query59	3585	3349	3234	3234
query60	211	205	214	205
query61	98	96	98	96
query62	789	504	495	495
query63	491	343	343	343
query64	2698	1555	1515	1515
query65	3715	3589	3584	3584
query66	805	378	383	378
query67	15702	15135	14947	14947
query68	10011	675	668	668
query69	573	349	365	349
query70	1693	1440	1368	1368
query71	419	308	321	308
query72	6297	3490	3504	3490
query73	747	326	322	322
query74	6413	5914	5932	5914
query75	5382	3660	3632	3632
query76	6435	1170	1210	1170
query77	1082	264	266	264
query78	12585	11634	16833	11634
query79	4675	646	640	640
query80	785	405	410	405
query81	482	238	234	234
query82	317	99	98	98
query83	175	133	132	132
query84	258	71	74	71
query85	859	319	328	319
query86	329	300	303	300
query87	3236	3053	3075	3053
query88	3631	2289	2290	2289
query89	352	285	322	285
query90	1963	218	210	210
query91	158	125	121	121
query92	55	53	52	52
query93	1015	576	563	563
query94	701	209	207	207
query95	1109	1109	1078	1078
query96	633	325	334	325
query97	6461	6448	6358	6358
query98	189	176	173	173
query99	2912	921	935	921
Total cold run time: 304464 ms
Total hot run time: 202347 ms

doris-robot · 2024-08-28T05:52:12Z

ClickBench: Total hot run time: 30.22 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b876d4c89629d19a514aeaf043388270a278df64, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.79	0.06	0.07
query5	0.54	0.52	0.53
query6	1.24	0.64	0.62
query7	0.01	0.02	0.01
query8	0.03	0.03	0.03
query9	0.53	0.49	0.49
query10	0.54	0.55	0.53
query11	0.13	0.09	0.08
query12	0.11	0.10	0.09
query13	0.62	0.61	0.62
query14	0.79	0.77	0.79
query15	0.78	0.77	0.77
query16	0.36	0.36	0.36
query17	1.00	1.00	0.97
query18	0.20	0.27	0.23
query19	1.90	1.84	1.85
query20	0.01	0.01	0.01
query21	15.45	0.57	0.56
query22	2.34	2.32	1.31
query23	17.33	0.89	0.96
query24	5.08	0.80	1.43
query25	0.39	0.12	0.04
query26	0.55	0.17	0.16
query27	0.04	0.05	0.04
query28	7.85	0.73	0.78
query29	12.66	2.29	2.26
query30	0.61	0.54	0.53
query31	2.82	0.39	0.37
query32	3.36	0.49	0.51
query33	3.09	3.12	3.09
query34	15.26	4.81	4.80
query35	4.88	4.82	4.83
query36	1.07	1.01	1.01
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 104.08 s
Total hot run time: 30.22 s

doris-robot · 2024-08-28T05:55:26Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b876d4c89629d19a514aeaf043388270a278df64 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s

bobhan1 · 2024-08-28T11:27:01Z

run external

… column for merge-on-write table (#39958) ## Proposed changes Currently, `BaseTablet::lookup_row_key()` use tablet_meta's schema to decide whether a tablet has sequence column. But users can use `ALTER TABLE tbl ENABLE FEATURE "SEQUENCE_LOAD" WITH ...` to add hidden sequence column on MOW table. This is a light schema change which will not change the BE's tablet meta, thus causing wrong behavior in `BaseTablet::lookup_row_key()`. This PR use the schema of the current load, which is the latest schema, to decide whether a tablet has sequence column and correct the lookup procedure in `BaseTablet::lookup_row_key()` and `Segment::lookup_row_key()`. branch-2.1-pick: #40010 branch-2.0-pick: #40015

… column for merge-on-write table apache#39958 (apache#40015)

pick

26d178a

bobhan1 mentioned this pull request Aug 28, 2024

[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958

Merged

github-actions bot added the kind/test label Aug 28, 2024

fix case

b876d4c

xiaokang merged commit d6ee2ff into apache:branch-2.0 Aug 29, 2024

bobhan1 added a commit to bobhan1/doris that referenced this pull request Jan 21, 2025

[Fix](merge-on-write) Fix duplicate key problem after adding sequence…

23efc88

… column for merge-on-write table apache#39958 (apache#40015)

[branch-2.0] Picks "[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958" #40015

[branch-2.0] Picks "[Fix](merge-on-write) Fix duplicate key problem after adding sequence column for merge-on-write table #39958" #40015

Uh oh!

Conversation

bobhan1 commented Aug 28, 2024

Proposed changes

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

bobhan1 commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

bobhan1 commented Aug 28, 2024

Uh oh!

github-actions bot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

doris-robot commented Aug 28, 2024

Uh oh!

bobhan1 commented Aug 28, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants