Skip to content

[fix](mtmv)fix mtmv dead lock #37009

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 28, 2024
Merged

[fix](mtmv)fix mtmv dead lock #37009

dataroaring merged 1 commit into from
Jun 28, 2024

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jun 28, 2024 •
edited
Loading

Cause of occurrence：

  • dropBaseTable: Holding db's writeLock ,notify mtmv alter status to schema_change,need mv's writeLock
  • task(insert overwrite):Holding mv's readLock,when generage plan need db's readLock

fix:

  • mtmv alter status to schema_change need mv's writeMvLock instead of mv's writeLock

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Jun 28, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

TPC-H: Total hot run time: 39403 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd6681415945a3fcfc7b707f5726f9c187aa9d2d, data reload: false

------ Round 1 ----------------------------------
q1	17616	4270	4236	4236
q2	2012	191	193	191
q3	10494	1138	992	992
q4	10182	763	752	752
q5	7461	2632	2611	2611
q6	216	132	135	132
q7	945	582	599	582
q8	9226	2036	2064	2036
q9	8857	6460	6413	6413
q10	9020	3693	3648	3648
q11	433	239	233	233
q12	479	232	225	225
q13	17770	2997	2958	2958
q14	257	228	221	221
q15	525	481	480	480
q16	514	377	371	371
q17	954	697	710	697
q18	7854	7339	7411	7339
q19	1904	1451	1540	1451
q20	643	318	312	312
q21	4888	3179	3895	3179
q22	392	348	344	344
Total cold run time: 112642 ms
Total hot run time: 39403 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4182	4166	4166
q2	373	269	268	268
q3	2931	2772	2649	2649
q4	1903	1685	1727	1685
q5	5539	5565	5543	5543
q6	238	132	131	131
q7	2244	1869	1797	1797
q8	3268	3397	3364	3364
q9	8667	8673	8645	8645
q10	4114	3856	3918	3856
q11	577	488	475	475
q12	748	584	603	584
q13	17095	3184	3124	3124
q14	309	265	276	265
q15	527	495	513	495
q16	490	427	430	427
q17	1793	1526	1496	1496
q18	8076	7996	7744	7744
q19	5933	1665	1541	1541
q20	2052	1842	1900	1842
q21	5085	4846	4927	4846
q22	610	551	558	551
Total cold run time: 76892 ms
Total hot run time: 55494 ms

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

TPC-DS: Total hot run time: 172832 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd6681415945a3fcfc7b707f5726f9c187aa9d2d, data reload: false

query1	933	374	377	374
query2	6453	2503	2268	2268
query3	6634	208	215	208
query4	19365	17680	17239	17239
query5	3613	477	492	477
query6	276	160	165	160
query7	4581	293	286	286
query8	324	301	282	282
query9	8644	2449	2446	2446
query10	559	284	273	273
query11	10574	10062	9983	9983
query12	113	81	80	80
query13	1633	374	376	374
query14	10103	6250	7002	6250
query15	233	190	184	184
query16	7622	264	259	259
query17	1705	521	552	521
query18	1682	270	265	265
query19	189	153	152	152
query20	86	80	82	80
query21	215	132	129	129
query22	4418	4086	4042	4042
query23	34052	33372	33603	33372
query24	10831	2843	2815	2815
query25	588	379	369	369
query26	708	162	163	162
query27	2274	338	329	329
query28	6191	2192	2167	2167
query29	875	619	626	619
query30	256	173	156	156
query31	987	785	756	756
query32	98	54	57	54
query33	680	277	288	277
query34	865	482	479	479
query35	762	640	651	640
query36	1093	993	981	981
query37	141	85	75	75
query38	2966	2853	2817	2817
query39	883	839	832	832
query40	217	130	137	130
query41	62	55	59	55
query42	110	102	107	102
query43	589	553	547	547
query44	1123	754	740	740
query45	192	165	167	165
query46	1060	709	708	708
query47	1840	1766	1736	1736
query48	360	308	298	298
query49	824	390	411	390
query50	755	385	380	380
query51	6863	6851	6728	6728
query52	110	95	98	95
query53	364	301	305	301
query54	856	448	434	434
query55	76	75	79	75
query56	289	258	252	252
query57	1137	1047	1040	1040
query58	243	243	251	243
query59	3375	3260	3417	3260
query60	294	271	266	266
query61	95	96	92	92
query62	614	445	445	445
query63	323	293	289	289
query64	8499	2232	1730	1730
query65	3185	3098	3071	3071
query66	737	320	326	320
query67	15292	14998	14752	14752
query68	5802	554	555	554
query69	629	471	393	393
query70	1204	1154	1089	1089
query71	429	272	268	268
query72	7338	5895	5403	5403
query73	773	334	324	324
query74	5947	5544	5493	5493
query75	3606	2696	2651	2651
query76	3161	1002	923	923
query77	608	301	313	301
query78	10261	9860	9774	9774
query79	2204	511	529	511
query80	937	486	482	482
query81	545	222	224	222
query82	1454	111	106	106
query83	269	186	179	179
query84	237	91	87	87
query85	1333	344	328	328
query86	470	327	320	320
query87	3257	3063	3065	3063
query88	3829	2401	2403	2401
query89	472	395	393	393
query90	1713	199	198	198
query91	144	108	111	108
query92	59	51	53	51
query93	2260	525	519	519
query94	1065	193	198	193
query95	421	321	324	321
query96	594	267	265	265
query97	3220	3027	3002	3002
query98	223	204	203	203
query99	1192	831	833	831
Total cold run time: 269345 ms
Total hot run time: 172832 ms

@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 28, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Jun 28, 2024

ClickBench: Total hot run time: 30.44 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd6681415945a3fcfc7b707f5726f9c187aa9d2d, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.67	0.07	0.07
query5	0.49	0.50	0.51
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.55	0.54
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.60
query14	0.76	0.76	0.79
query15	0.84	0.81	0.81
query16	0.36	0.37	0.37
query17	0.97	1.01	1.02
query18	0.22	0.24	0.25
query19	1.77	1.72	1.69
query20	0.01	0.00	0.00
query21	15.42	0.72	0.65
query22	4.54	6.83	1.79
query23	18.31	1.38	1.30
query24	2.07	0.21	0.23
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.31	1.02	0.99
query29	12.64	3.24	3.26
query30	0.26	0.06	0.05
query31	2.88	0.38	0.38
query32	3.28	0.47	0.48
query33	2.85	2.93	2.89
query34	16.95	4.43	4.43
query35	4.48	4.50	4.46
query36	0.65	0.46	0.48
query37	0.18	0.16	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.13	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.49 s
Total hot run time: 30.44 s

dataroaring
dataroaring approved these changes Jun 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 620653f into apache:master Jun 28, 2024
dataroaring pushed a commit that referenced this pull request Jun 30, 2024
@zddr @dataroaring
[fix](mtmv)fix mtmv dead lock (#37009)
05ca6c6 
Cause of occurrence：
- dropBaseTable: Holding db's writeLock ,notify mtmv alter status to
schema_change,need mv's writeLock
- task(insert overwrite):Holding mv's readLock,when generage plan need
db's readLock

fix:
- mtmv alter status to schema_change need mv's writeMvLock instead of
mv's writeLock
zddr added a commit to zddr/incubator-doris that referenced this pull request Jul 2, 2024
@zddr
[fix](mtmv)fix mtmv dead lock (apache#37009)
60606e5 
Cause of occurrence：
- dropBaseTable: Holding db's writeLock ,notify mtmv alter status to
schema_change,need mv's writeLock
- task(insert overwrite):Holding mv's readLock,when generage plan need
db's readLock

fix:
- mtmv alter status to schema_change need mv's writeMvLock instead of
mv's writeLock
@zddr zddr mentioned this pull request Jul 2, 2024
Merged
morningman pushed a commit that referenced this pull request Jul 2, 2024
@zddr
[fix](mtmv)fix mtmv dead lock (#37009) (#37133)
1777646 
pick #37009
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zddr @doris-robot @dataroaring @morrySnow @morningman @CalvinKirs