Skip to content

[fix] (schema change) ignore the error of repeatedly writing rowset when schema change #29733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jan 10, 2024
Merged

[fix] (schema change) ignore the error of repeatedly writing rowset when schema change #29733

dataroaring merged 1 commit into from
Jan 10, 2024

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Jan 9, 2024 •
edited
Loading

When load and schema change are concurrent, the same rowset may be written repeatedly. This error needs to be ignored.

@luwei16
Copy link
Contributor Author

luwei16 commented Jan 9, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jan 9, 2024

TeamCity be ut coverage result:
Function Coverage: 36.64% (8639/23580)
Line Coverage: 28.65% (70212/245051)
Region Coverage: 27.60% (36315/131566)
Branch Coverage: 24.29% (18555/76394)
Coverage Report: http://coverage.selectdb-in.cc/coverage/346b3419414d5fad7fb0c2869e9bfae955c99532_346b3419414d5fad7fb0c2869e9bfae955c99532/report/index.html

@doris-robot
Copy link

doris-robot commented Jan 9, 2024

TPC-H: Total hot run time: 39230 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 346b3419414d5fad7fb0c2869e9bfae955c99532, data reload: false

------ Round 1 ----------------------------------
q1	18354	5530	5486	5486
q2	2015	157	147	147
q3	10686	1115	1146	1115
q4	10493	788	780	780
q5	7861	2969	2961	2961
q6	205	131	132	131
q7	902	519	504	504
q8	9291	2002	2018	2002
q9	6737	6390	6305	6305
q10	8244	3040	2966	2966
q11	441	221	220	220
q12	392	230	233	230
q13	18283	3462	3456	3456
q14	234	209	210	209
q15	554	531	503	503
q16	455	393	417	393
q17	939	778	669	669
q18	7262	6657	6676	6657
q19	1551	1527	1495	1495
q20	678	328	321	321
q21	2748	2407	2365	2365
q22	370	315	344	315
Total cold run time: 108695 ms
Total hot run time: 39230 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5539	5508	5497	5497
q2	353	257	268	257
q3	3331	3216	3273	3216
q4	2116	2100	2082	2082
q5	5825	5855	5816	5816
q6	208	126	124	124
q7	2303	1897	1884	1884
q8	3411	3591	3527	3527
q9	8896	8899	8899	8899
q10	3822	3876	3873	3873
q11	606	498	518	498
q12	814	623	633	623
q13	6673	3245	3218	3218
q14	288	288	294	288
q15	552	514	517	514
q16	554	498	482	482
q17	2380	2356	2347	2347
q18	8713	8527	8288	8288
q19	1583	1634	1624	1624
q20	2211	1957	1972	1957
q21	5711	5405	5378	5378
q22	562	515	513	513
Total cold run time: 66451 ms
Total hot run time: 60905 ms

@doris-robot
Copy link

doris-robot commented Jan 9, 2024

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.72 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 27.5 seconds inserted 10000000 Rows, about 363K ops/s
storage size: 17184022198 Bytes

@doris-robot
Copy link

doris-robot commented Jan 9, 2024

TPC-DS: Total hot run time: 186631 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 346b3419414d5fad7fb0c2869e9bfae955c99532, data reload: false

run tpcds-sf100 query with default conf and session variables
query1	944	346	338	338
query2	6877	2130	2016	2016
query3	6756	228	221	221
query4	26107	22596	22461	22461
query5	5812	611	595	595
query6	348	212	214	212
query7	4643	310	301	301
query8	263	243	233	233
query9	9723	3332	3272	3272
query10	547	344	315	315
query11	16156	15562	15306	15306
query12	190	116	122	116
query13	1684	349	338	338
query14	13069	7550	7670	7550
query15	275	255	255	255
query16	6490	363	349	349
query17	1742	535	550	535
query18	2001	349	319	319
query19	343	173	174	173
query20	132	120	118	118
query21	181	102	97	97
query22	4754	4502	4621	4502
query23	32136	31627	31529	31529
query24	8744	2998	3016	2998
query25	602	387	384	384
query26	838	199	184	184
query27	2350	317	313	313
query28	5687	2140	2137	2137
query29	883	437	418	418
query30	280	153	154	153
query31	1061	874	858	858
query32	145	103	103	103
query33	667	370	342	342
query34	835	492	477	477
query35	1027	916	835	835
query36	1486	1291	1382	1291
query37	146	107	107	107
query38	3469	3376	3433	3376
query39	1378	1329	1326	1326
query40	186	101	94	94
query41	41	36	37	36
query42	124	117	110	110
query43	624	583	588	583
query44	1183	819	831	819
query45	227	204	215	204
query46	1074	712	719	712
query47	1784	1706	1728	1706
query48	393	284	295	284
query49	1045	335	344	335
query50	809	360	386	360
query51	5495	5434	5319	5319
query52	131	109	114	109
query53	238	184	178	178
query54	1045	706	736	706
query55	107	105	114	105
query56	298	289	280	280
query57	1087	1044	1031	1031
query58	353	335	341	335
query59	3350	3212	3163	3163
query60	392	326	313	313
query61	154	152	149	149
query62	599	468	526	468
query63	237	193	199	193
query64	4334	1818	1793	1793
query65	3510	3412	3399	3399
query66	1077	431	486	431
query67	15844	15347	15200	15200
query68	12577	560	553	553
query69	593	361	390	361
query70	1801	1784	1721	1721
query71	610	305	270	270
query72	5526	3459	3487	3459
query73	3068	344	340	340
query74	7011	6625	6513	6513
query75	5613	2350	2342	2342
query76	6548	1154	1253	1154
query77	799	388	376	376
query78	9122	8816	8751	8751
query79	1086	555	556	555
query80	610	371	362	362
query81	470	221	234	221
query82	244	141	134	134
query83	179	144	150	144
query84	264	65	60	60
query85	939	295	277	277
query86	438	453	445	445
query87	3650	3483	3446	3446
query88	3373	2590	2552	2552
query89	389	297	303	297
query90	1931	297	276	276
query91	128	95	99	95
query92	117	103	93	93
query93	1326	440	503	440
query94	898	260	254	254
query95	608	543	513	513
query96	673	346	349	346
query97	4421	4315	4325	4315
query98	257	229	220	220
query99	1184	923	878	878
Total cold run time: 292965 ms
Total hot run time: 186631 ms

@doris-robot
Copy link

doris-robot commented Jan 9, 2024

ClickBench: Total hot run time: 31.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 346b3419414d5fad7fb0c2869e9bfae955c99532, data reload: false

query1	0.07	0.10	0.06
query2	0.07	0.03	0.03
query3	0.32	0.22	0.21
query4	1.73	0.20	0.19
query5	0.52	0.52	0.53
query6	1.57	0.70	0.69
query7	0.02	0.01	0.01
query8	0.05	0.03	0.03
query9	0.59	0.58	0.56
query10	0.62	0.61	0.60
query11	0.17	0.12	0.12
query12	0.15	0.13	0.12
query13	0.60	0.62	0.59
query14	0.75	0.79	0.78
query15	0.81	0.80	0.80
query16	0.34	0.35	0.36
query17	0.98	0.98	1.00
query18	0.25	0.27	0.27
query19	1.82	1.80	1.80
query20	0.01	0.00	0.01
query21	15.39	0.55	0.52
query22	2.36	3.38	1.63
query23	17.46	1.28	0.95
query24	9.91	0.76	0.33
query25	1.99	0.17	0.17
query26	0.15	0.16	0.16
query27	0.18	0.18	0.18
query28	5.46	0.93	0.90
query29	13.04	3.44	3.49
query30	0.51	0.52	0.53
query31	2.91	0.38	0.37
query32	3.31	0.50	0.50
query33	3.23	3.18	3.27
query34	16.79	4.51	4.50
query35	4.59	4.57	4.55
query36	1.15	1.10	1.07
query37	0.06	0.04	0.05
query38	0.03	0.02	0.03
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 110.25 s
Total hot run time: 31.94 s

@doris-robot
Copy link

doris-robot commented Jan 9, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 346b3419414d5fad7fb0c2869e9bfae955c99532 with default session variables
 stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
 stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
 stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
 insert into select:       18.6 seconds inserted 10000000 Rows, about 537K ops/s

liaoxin01
liaoxin01 approved these changes Jan 9, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Jan 10, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 10, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 10, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 10, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit fd6a040 into apache:master Jan 10, 2024
@wm1581066 wm1581066 added the usercase Important user case type label label Jan 10, 2024
luwei16 added a commit to luwei16/Doris that referenced this pull request Jan 10, 2024
@luwei16
[fix] (schema change) ignore the error of repeatedly writing rowset d…
4c97a2b 
…uring schema change (apache#29733)
@luwei16 luwei16 mentioned this pull request Jan 10, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jan 10, 2024
@luwei16
[fix] (schema change) ignore the error of repeatedly writing rowset d…
a67b643 
…uring schema change (#29733) (#29810)
yiguolei pushed a commit that referenced this pull request Jan 12, 2024
@luwei16 @Doris-Extras
[fix] (schema change) ignore the error of repeatedly writing rowset d…
a94f256 
…uring schema change (#29733)
@zhannngchen zhannngchen mentioned this pull request Apr 11, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.4-merged dev/3.0.0-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@luwei16 @doris-robot @liaoxin01 @dataroaring @xiaokang @wm1581066