Skip to content

[fix](cloud) fix commit txn failed for mow tables #41793

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Oct 20, 2024
Merged

[fix](cloud) fix commit txn failed for mow tables #41793

dataroaring merged 1 commit into from
Oct 20, 2024

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Oct 14, 2024

When s3 load to 2 tables: 1 is non mow table and 1 is mow table. If the tablet commit info for mow table is empty, the commit will get error:

msg:errCode = 2, detailMessage = The partition info is empty, table may be dropped, txnid=570859248863232

@doris-robot
Copy link

doris-robot commented Oct 14, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 14, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 14, 2024

TPC-H: Total hot run time: 41445 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 252bc6720ccf78ae092072ae0451e7d62c06f27a, data reload: false

------ Round 1 ----------------------------------
q1	17586	7444	7301	7301
q2	2024	283	263	263
q3	12223	1049	1189	1049
q4	10596	879	764	764
q5	7762	3066	3111	3066
q6	237	162	152	152
q7	1045	635	601	601
q8	9359	1915	1942	1915
q9	6570	6493	6442	6442
q10	7091	2423	2608	2423
q11	443	255	264	255
q12	412	229	224	224
q13	17769	3023	3044	3023
q14	247	213	212	212
q15	562	530	532	530
q16	640	583	592	583
q17	968	602	513	513
q18	7544	6940	6739	6739
q19	1347	1028	931	931
q20	468	186	180	180
q21	4050	3269	3324	3269
q22	1120	1010	1012	1010
Total cold run time: 110063 ms
Total hot run time: 41445 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7266	7218	7281	7218
q2	344	248	244	244
q3	3071	2988	3019	2988
q4	2146	1818	1834	1818
q5	5728	5782	5733	5733
q6	237	149	153	149
q7	2248	1873	1830	1830
q8	3446	3456	3549	3456
q9	8891	8959	8894	8894
q10	3594	3500	3551	3500
q11	596	498	506	498
q12	885	635	623	623
q13	10393	3240	3255	3240
q14	303	291	271	271
q15	578	535	521	521
q16	680	634	647	634
q17	1873	1653	1620	1620
q18	8277	7806	7737	7737
q19	1761	1620	1560	1560
q20	2137	1848	1899	1848
q21	5799	5659	5314	5314
q22	1220	1073	1062	1062
Total cold run time: 71473 ms
Total hot run time: 60758 ms

@doris-robot
Copy link

doris-robot commented Oct 14, 2024

TPC-DS: Total hot run time: 193051 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 252bc6720ccf78ae092072ae0451e7d62c06f27a, data reload: false

query1	860	394	407	394
query2	6237	2097	2065	2065
query3	8680	195	216	195
query4	34387	23716	23592	23592
query5	3533	473	472	472
query6	279	164	165	164
query7	4184	290	284	284
query8	277	223	213	213
query9	9643	2675	2656	2656
query10	476	284	276	276
query11	17868	15229	15316	15229
query12	159	102	101	101
query13	1595	436	431	431
query14	9506	7711	7703	7703
query15	276	169	181	169
query16	8071	477	516	477
query17	1583	599	580	580
query18	2150	317	308	308
query19	378	152	165	152
query20	123	108	114	108
query21	213	108	107	107
query22	4905	4484	4389	4389
query23	36167	34979	34355	34355
query24	10911	2807	2871	2807
query25	526	427	412	412
query26	723	161	158	158
query27	2186	300	293	293
query28	7526	2402	2395	2395
query29	658	423	420	420
query30	256	156	155	155
query31	1021	802	805	802
query32	97	55	55	55
query33	758	289	295	289
query34	902	514	529	514
query35	905	774	754	754
query36	1133	947	950	947
query37	152	83	80	80
query38	4041	3958	3955	3955
query39	1455	1450	1436	1436
query40	208	99	100	99
query41	46	45	43	43
query42	116	98	99	98
query43	516	493	496	493
query44	1338	832	819	819
query45	194	165	166	165
query46	1155	713	694	694
query47	1932	1884	1844	1844
query48	436	331	350	331
query49	893	421	426	421
query50	806	407	405	405
query51	7215	7034	6982	6982
query52	101	86	87	86
query53	262	190	187	187
query54	1221	425	445	425
query55	79	77	76	76
query56	287	274	268	268
query57	1259	1183	1182	1182
query58	227	241	232	232
query59	3093	2969	3099	2969
query60	301	270	270	270
query61	109	104	102	102
query62	898	692	672	672
query63	228	194	192	192
query64	3818	620	604	604
query65	3286	3222	3230	3222
query66	838	301	306	301
query67	15998	15738	15819	15738
query68	4334	553	554	553
query69	530	287	292	287
query70	1232	1120	1076	1076
query71	360	276	281	276
query72	7445	3997	3929	3929
query73	788	355	358	355
query74	10150	9028	8956	8956
query75	3429	2730	2668	2668
query76	3206	939	914	914
query77	624	307	308	307
query78	10825	9716	9556	9556
query79	2273	599	618	599
query80	1258	450	460	450
query81	591	245	240	240
query82	634	144	138	138
query83	299	136	134	134
query84	273	70	67	67
query85	1375	309	284	284
query86	455	314	307	307
query87	4409	4344	4316	4316
query88	3237	2229	2196	2196
query89	405	293	287	287
query90	1944	195	186	186
query91	148	103	108	103
query92	68	49	48	48
query93	2568	550	541	541
query94	1023	299	291	291
query95	352	245	247	245
query96	634	286	289	286
query97	3277	3134	3151	3134
query98	225	200	198	198
query99	1650	1312	1320	1312
Total cold run time: 301681 ms
Total hot run time: 193051 ms

@doris-robot
Copy link

doris-robot commented Oct 14, 2024

ClickBench: Total hot run time: 32.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 252bc6720ccf78ae092072ae0451e7d62c06f27a, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.49	0.52	0.51
query6	1.14	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.52	0.50
query10	0.56	0.55	0.55
query11	0.14	0.11	0.10
query12	0.13	0.11	0.12
query13	0.61	0.60	0.58
query14	2.71	2.84	2.72
query15	0.90	0.82	0.84
query16	0.39	0.36	0.39
query17	1.05	0.98	1.05
query18	0.20	0.19	0.19
query19	1.93	1.86	1.95
query20	0.01	0.01	0.00
query21	15.37	0.56	0.60
query22	2.81	2.05	1.69
query23	17.04	0.92	0.80
query24	3.49	1.04	0.75
query25	0.27	0.17	0.10
query26	0.47	0.14	0.13
query27	0.04	0.04	0.04
query28	10.72	1.09	1.06
query29	12.56	3.25	3.23
query30	0.24	0.06	0.05
query31	2.87	0.37	0.40
query32	3.27	0.47	0.47
query33	2.99	3.00	3.00
query34	17.26	4.46	4.48
query35	4.58	4.49	4.54
query36	0.64	0.51	0.49
query37	0.07	0.05	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 107.94 s
Total hot run time: 32.22 s

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Oct 14, 2024

run cloud_p0

liaoxin01
liaoxin01 approved these changes Oct 17, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 17, 2024

PR approved by anyone and no changes requested.

@mymeiyi mymeiyi mentioned this pull request Oct 18, 2024
Merged
dataroaring
dataroaring approved these changes Oct 20, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 9c720b6 into apache:master Oct 20, 2024
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Oct 21, 2024
@mymeiyi
[fix](cloud) fix commit txn failed for mow tables (apache#41793)
dc9ba4b 
When s3 load to 2 tables: 1 is non mow table and 1 is mow table. If the
tablet commit info for mow table is empty, the commit will get error:
```
msg:errCode = 2, detailMessage = The partition info is empty, table may be dropped, txnid=570859248863232
```
@mymeiyi mymeiyi mentioned this pull request Oct 21, 2024
Merged
dataroaring pushed a commit that referenced this pull request Oct 21, 2024
@mymeiyi
[fix](cloud) fix commit txn failed for mow tables (#41793) (#42142)
cc0b0be 
pick #41793
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@mymeiyi @doris-robot @liaoxin01 @dataroaring