Skip to content

[branch-2.0](cherry-pick) partial update should not read old fileds from rows with delete sign (#36210) #36764

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 4 commits into from
Jun 25, 2024
Merged

[branch-2.0](cherry-pick) partial update should not read old fileds from rows with delete sign (#36210) #36764

dataroaring merged 4 commits into from
Jun 25, 2024

Conversation

@zhannngchen
Copy link
Contributor

@zhannngchen zhannngchen commented Jun 24, 2024

cherry-pick #36210

@zhannngchen
[fix](partial update) partial update should not read old fileds from …
3d06aa7 
…rows with delete sign (apache#36210)

Issue Number: close apache#34296

1. When partial update filling in the missing fields, if a load job
previously wrote data with a delete sign, it will also read out the data
in the column with the delete sign, so that the newly written data will
also become invisible
2. This problem was fixed in apache#24877, but was introduced again in apache#26721,
and was never found because the case was changed to the wrong output in
3. The fix in apache#24877 didn't take into account the handling of concurrent
conflicts in the publish phase, the current PR adds this part of the
handling, and adds the corresponding case.
@doris-robot
Copy link

doris-robot commented Jun 24, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Jun 24, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Jun 24, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

TeamCity be ut coverage result:
Function Coverage: 37.86% (8108/21416)
Line Coverage: 29.52% (66362/224778)
Region Coverage: 28.99% (34197/117960)
Branch Coverage: 24.85% (17554/70626)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8ed491004ab1c52f1b9d9e63b3a56e7ee9bb3c7a_8ed491004ab1c52f1b9d9e63b3a56e7ee9bb3c7a/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

TPC-H: Total hot run time: 49953 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ed491004ab1c52f1b9d9e63b3a56e7ee9bb3c7a, data reload: false

------ Round 1 ----------------------------------
q1	17541	4387	4331	4331
q2	2066	172	148	148
q3	10290	1961	1947	1947
q4	10357	1245	1331	1245
q5	8444	3956	3936	3936
q6	236	128	127	127
q7	2093	1587	1599	1587
q8	9494	2712	2711	2711
q9	13856	11371	10239	10239
q10	8673	3526	3556	3526
q11	417	252	253	252
q12	473	302	307	302
q13	18373	3974	4043	3974
q14	358	324	330	324
q15	519	466	452	452
q16	665	574	577	574
q17	1144	939	918	918
q18	7276	6950	6849	6849
q19	1797	1664	1643	1643
q20	576	307	299	299
q21	4470	4116	4124	4116
q22	530	453	460	453
Total cold run time: 119648 ms
Total hot run time: 49953 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4367	4288	4306	4288
q2	321	222	226	222
q3	4213	4188	4187	4187
q4	2754	2754	2731	2731
q5	7110	7069	7046	7046
q6	239	118	120	118
q7	3263	2849	2919	2849
q8	4420	4473	4515	4473
q9	16937	16778	16833	16778
q10	4217	4312	4265	4265
q11	762	702	704	702
q12	1041	852	874	852
q13	6652	3765	3752	3752
q14	460	425	431	425
q15	517	464	458	458
q16	741	682	695	682
q17	3854	3917	3866	3866
q18	8817	8765	8881	8765
q19	1721	1661	1703	1661
q20	2368	2098	2111	2098
q21	8526	8415	8441	8415
q22	1040	976	970	970
Total cold run time: 84340 ms
Total hot run time: 79603 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

TPC-DS: Total hot run time: 203586 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ed491004ab1c52f1b9d9e63b3a56e7ee9bb3c7a, data reload: false

query1	955	389	372	372
query2	6527	2883	2564	2564
query3	6923	209	204	204
query4	20336	18003	17868	17868
query5	19747	6464	6526	6464
query6	289	227	232	227
query7	4160	296	314	296
query8	418	402	406	402
query9	3108	2653	2599	2599
query10	440	313	314	313
query11	11200	10677	10735	10677
query12	122	81	73	73
query13	5605	695	670	670
query14	17671	13453	13415	13415
query15	364	250	245	245
query16	6450	294	267	267
query17	1697	1453	867	867
query18	2326	417	405	405
query19	207	145	154	145
query20	82	79	80	79
query21	189	100	96	96
query22	5187	5059	4968	4968
query23	32420	31964	32129	31964
query24	6993	6479	6577	6479
query25	528	428	419	419
query26	615	157	161	157
query27	2025	292	288	288
query28	6132	2325	2298	2298
query29	2963	2618	2833	2618
query30	241	167	167	167
query31	930	745	765	745
query32	69	64	57	57
query33	404	260	259	259
query34	847	481	487	481
query35	1110	920	978	920
query36	1315	1155	1282	1155
query37	90	61	61	61
query38	3119	2966	2988	2966
query39	1386	1329	1326	1326
query40	204	98	92	92
query41	45	43	46	43
query42	86	86	84	84
query43	737	733	673	673
query44	1135	737	724	724
query45	247	233	239	233
query46	1249	967	983	967
query47	1786	1913	1652	1652
query48	1025	736	693	693
query49	630	365	370	365
query50	875	632	618	618
query51	4874	4651	4635	4635
query52	101	81	85	81
query53	455	315	316	315
query54	2665	2505	2486	2486
query55	91	82	80	80
query56	241	222	209	209
query57	1236	1123	1117	1117
query58	217	196	208	196
query59	4260	4152	4093	4093
query60	216	194	211	194
query61	98	96	99	96
query62	840	475	478	475
query63	480	339	347	339
query64	2620	1549	1503	1503
query65	3669	3564	3574	3564
query66	822	368	379	368
query67	15746	15513	15123	15123
query68	10066	641	671	641
query69	582	357	355	355
query70	1828	1374	1299	1299
query71	420	313	311	311
query72	6664	3526	3479	3479
query73	740	320	315	315
query74	6391	5956	5887	5887
query75	5365	3719	3803	3719
query76	6415	1153	1203	1153
query77	1102	268	249	249
query78	12707	11917	11896	11896
query79	7559	622	655	622
query80	1092	397	399	397
query81	487	236	233	233
query82	1394	101	96	96
query83	187	134	132	132
query84	258	71	74	71
query85	878	319	318	318
query86	318	310	325	310
query87	3254	3029	3059	3029
query88	4511	2277	2262	2262
query89	362	283	301	283
query90	1994	212	208	208
query91	174	143	139	139
query92	60	55	53	53
query93	5654	590	590	590
query94	693	212	205	205
query95	1103	1057	1047	1047
query96	643	314	319	314
query97	6420	6372	6337	6337
query98	192	177	174	174
query99	2886	868	874	868
Total cold run time: 314239 ms
Total hot run time: 203586 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ed491004ab1c52f1b9d9e63b3a56e7ee9bb3c7a, data reload: false

query1	0.02	0.02	0.03
query2	0.06	0.03	0.02
query3	0.25	0.05	0.05
query4	1.78	0.06	0.07
query5	0.54	0.53	0.53
query6	1.28	0.61	0.60
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.50	0.48
query10	0.54	0.54	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.81	0.80	0.78
query15	0.78	0.75	0.76
query16	0.36	0.38	0.36
query17	1.00	1.02	1.02
query18	0.20	0.27	0.23
query19	1.94	1.86	1.86
query20	0.01	0.01	0.01
query21	15.45	0.55	0.57
query22	2.06	2.04	1.66
query23	17.02	1.05	0.98
query24	7.51	1.60	0.86
query25	0.37	0.06	0.05
query26	0.95	0.15	0.15
query27	0.04	0.05	0.04
query28	5.28	0.75	0.73
query29	12.63	2.29	2.34
query30	0.58	0.55	0.53
query31	2.81	0.39	0.38
query32	3.42	0.51	0.50
query33	3.10	3.08	3.10
query34	15.24	4.81	4.78
query35	4.88	4.82	4.84
query36	1.06	1.00	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.79 s
Total hot run time: 30.76 s

@doris-robot
Copy link

doris-robot commented Jun 24, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 8ed491004ab1c52f1b9d9e63b3a56e7ee9bb3c7a with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@zhannngchen
Copy link
Contributor Author

zhannngchen commented Jun 25, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

TPC-H: Total hot run time: 49824 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 70dfbcd62d621318fc0eb0d902a616eef1fbf428, data reload: false

------ Round 1 ----------------------------------
q1	17625	4398	4353	4353
q2	2104	159	158	158
q3	10262	1858	1921	1858
q4	10430	1255	1350	1255
q5	8353	3878	3926	3878
q6	240	149	124	124
q7	2049	1619	1617	1617
q8	9498	2721	2693	2693
q9	13887	10277	10305	10277
q10	8664	3491	3562	3491
q11	410	237	246	237
q12	475	306	308	306
q13	18361	3971	3998	3971
q14	358	328	323	323
q15	513	474	460	460
q16	669	579	570	570
q17	1125	943	903	903
q18	7209	6978	6963	6963
q19	1768	1642	1555	1555
q20	517	316	304	304
q21	4445	4117	4081	4081
q22	534	447	448	447
Total cold run time: 119496 ms
Total hot run time: 49824 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4251	4307	4251
q2	320	229	225	225
q3	4135	4146	4142	4142
q4	2749	2730	2732	2730
q5	7211	7113	7052	7052
q6	244	120	122	120
q7	3240	2864	2873	2864
q8	4358	4536	4492	4492
q9	16911	16620	16695	16620
q10	4212	4258	4271	4258
q11	752	658	693	658
q12	1051	860	868	860
q13	6814	3734	3762	3734
q14	453	440	433	433
q15	492	457	463	457
q16	737	673	664	664
q17	3789	3820	3850	3820
q18	8880	8768	8700	8700
q19	1710	1656	1741	1656
q20	2393	2115	2128	2115
q21	8523	8430	8416	8416
q22	1034	1037	997	997
Total cold run time: 84318 ms
Total hot run time: 79264 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

TeamCity be ut coverage result:
Function Coverage: 37.85% (8106/21416)
Line Coverage: 29.52% (66350/224778)
Region Coverage: 28.98% (34189/117960)
Branch Coverage: 24.85% (17551/70626)
Coverage Report: http://coverage.selectdb-in.cc/coverage/70dfbcd62d621318fc0eb0d902a616eef1fbf428_70dfbcd62d621318fc0eb0d902a616eef1fbf428/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

TPC-DS: Total hot run time: 203426 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 70dfbcd62d621318fc0eb0d902a616eef1fbf428, data reload: false

query1	935	416	388	388
query2	6526	2788	2658	2658
query3	6921	202	201	201
query4	20941	17924	17833	17833
query5	19759	6542	6454	6454
query6	292	213	240	213
query7	4165	296	295	295
query8	437	424	447	424
query9	3041	2635	2571	2571
query10	434	287	284	284
query11	11479	10676	10747	10676
query12	120	78	75	75
query13	5601	685	687	685
query14	18014	13235	13470	13235
query15	404	239	245	239
query16	6471	287	257	257
query17	1716	1442	873	873
query18	2314	414	414	414
query19	198	151	151	151
query20	82	78	83	78
query21	189	95	96	95
query22	5299	4985	5178	4985
query23	32618	31877	31800	31800
query24	6912	6523	6485	6485
query25	529	425	407	407
query26	526	162	158	158
query27	1884	293	293	293
query28	6124	2350	2286	2286
query29	2975	2733	2701	2701
query30	241	164	169	164
query31	908	735	751	735
query32	75	64	58	58
query33	396	246	265	246
query34	838	467	474	467
query35	1114	932	890	890
query36	1339	1284	1258	1258
query37	92	61	63	61
query38	3070	2916	2911	2911
query39	1381	1326	1325	1325
query40	212	98	93	93
query41	46	44	45	44
query42	84	86	84	84
query43	733	667	664	664
query44	1138	718	721	718
query45	252	237	241	237
query46	1216	953	955	953
query47	1878	1800	1834	1800
query48	1008	721	709	709
query49	621	365	383	365
query50	859	571	574	571
query51	4779	4657	4684	4657
query52	90	74	88	74
query53	451	328	320	320
query54	2653	2498	2463	2463
query55	92	85	83	83
query56	231	212	218	212
query57	1321	1115	1156	1115
query58	214	224	198	198
query59	4345	4089	4279	4089
query60	219	206	208	206
query61	98	96	97	96
query62	816	459	488	459
query63	487	335	337	335
query64	2615	1570	1364	1364
query65	3611	3568	3551	3551
query66	824	377	376	376
query67	16644	15797	17578	15797
query68	8218	656	643	643
query69	574	364	357	357
query70	1617	1269	1253	1253
query71	397	307	311	307
query72	6567	3492	3536	3492
query73	739	318	323	318
query74	6367	5890	5834	5834
query75	4627	3733	3615	3615
query76	4612	1166	1197	1166
query77	547	258	251	251
query78	12705	11527	11949	11527
query79	9374	637	637	637
query80	2911	395	394	394
query81	532	234	238	234
query82	1651	102	99	99
query83	187	139	132	132
query84	263	72	71	71
query85	1415	328	323	323
query86	368	297	306	297
query87	3228	3011	3047	3011
query88	5124	2259	2263	2259
query89	386	286	290	286
query90	1747	213	218	213
query91	177	142	141	141
query92	58	51	57	51
query93	5719	549	572	549
query94	887	214	214	214
query95	1101	1049	1050	1049
query96	634	321	324	321
query97	6549	6527	6451	6451
query98	198	182	177	177
query99	2874	792	909	792
Total cold run time: 316250 ms
Total hot run time: 203426 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

ClickBench: Total hot run time: 30.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 70dfbcd62d621318fc0eb0d902a616eef1fbf428, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.04
query4	1.79	0.07	0.06
query5	0.54	0.53	0.53
query6	1.23	0.60	0.60
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.49	0.48
query10	0.54	0.53	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.09
query13	0.62	0.60	0.61
query14	0.78	0.79	0.78
query15	0.77	0.77	0.76
query16	0.36	0.37	0.36
query17	0.99	1.01	1.02
query18	0.22	0.25	0.23
query19	1.91	1.87	1.81
query20	0.01	0.01	0.01
query21	15.47	0.57	0.55
query22	2.00	2.72	1.53
query23	17.16	1.10	0.99
query24	5.26	1.12	1.92
query25	0.39	0.13	0.04
query26	0.69	0.16	0.17
query27	0.05	0.04	0.04
query28	6.84	0.72	0.72
query29	12.63	2.30	2.12
query30	0.61	0.53	0.52
query31	2.83	0.40	0.36
query32	3.37	0.49	0.51
query33	3.07	3.10	3.06
query34	15.22	4.78	4.77
query35	4.89	4.86	4.82
query36	1.04	1.01	1.02
query37	0.05	0.05	0.05
query38	0.03	0.03	0.02
query39	0.02	0.01	0.01
query40	0.16	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 102.8 s
Total hot run time: 30.59 s

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 70dfbcd62d621318fc0eb0d902a616eef1fbf428 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@dataroaring dataroaring merged commit 05253c4 into apache:branch-2.0 Jun 25, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@zhannngchen
[branch-2.0](cherry-pick) partial update should not read old fileds f…
e4830bc 
…rom rows with delete sign (apache#36210) (apache#36764)

cherry-pick apache#36210
bobhan1 pushed a commit to bobhan1/doris that referenced this pull request Jan 20, 2025
[branch-2.0](cherry-pick) partial update should not read old fileds f…
464d26f 
…rom rows with delete sign (apache#36210) (apache#36764) (apache#277)

cherry-pick apache#36210
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhannngchen @doris-robot @dataroaring