Skip to content

[fix](mtmv)fix when related table drop partition,mv partition is sync #36547

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jun 20, 2024
Merged

[fix](mtmv)fix when related table drop partition,mv partition is sync #36547

morrySnow merged 1 commit into from
Jun 20, 2024

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jun 19, 2024

for example:
if mv's partition [mv_p1] is related with t1's partition [p1,p2]

when p2 dropped,
mv_p1'isSync should be false

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Jun 19, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TPC-H: Total hot run time: 40909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87f6f0ac6e1f81cbf99626c05334dba502b7c245, data reload: false

------ Round 1 ----------------------------------
q1	18027	4394	4336	4336
q2	2027	192	190	190
q3	10563	1196	1159	1159
q4	10608	808	806	806
q5	8370	2797	2691	2691
q6	228	139	139	139
q7	977	634	627	627
q8	9610	2071	2109	2071
q9	9163	6598	6590	6590
q10	9578	3685	3766	3685
q11	441	241	241	241
q12	457	249	230	230
q13	17760	2994	2991	2991
q14	261	225	214	214
q15	532	489	479	479
q16	529	386	382	382
q17	963	701	633	633
q18	8020	7649	7419	7419
q19	2204	1588	1493	1493
q20	641	314	330	314
q21	5082	3991	3882	3882
q22	398	345	337	337
Total cold run time: 116439 ms
Total hot run time: 40909 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4289	4261	4261
q2	374	266	259	259
q3	2950	2739	2776	2739
q4	1910	1635	1659	1635
q5	5248	5306	5299	5299
q6	218	129	129	129
q7	2135	1697	1776	1697
q8	3176	3325	3308	3308
q9	8393	8336	8376	8336
q10	3858	3648	3576	3576
q11	582	503	487	487
q12	770	604	610	604
q13	16440	2994	2977	2977
q14	290	265	275	265
q15	515	476	465	465
q16	505	426	431	426
q17	1765	1461	1485	1461
q18	7625	7658	7298	7298
q19	1709	1570	1674	1570
q20	1988	1780	1784	1780
q21	4893	4782	4858	4782
q22	622	535	580	535
Total cold run time: 70354 ms
Total hot run time: 53889 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

TPC-DS: Total hot run time: 170391 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87f6f0ac6e1f81cbf99626c05334dba502b7c245, data reload: false

query1	937	380	376	376
query2	6450	2350	2310	2310
query3	6671	205	206	205
query4	19202	17326	17278	17278
query5	4140	477	465	465
query6	243	159	163	159
query7	4585	296	291	291
query8	330	302	299	299
query9	8504	2411	2386	2386
query10	625	304	293	293
query11	10512	10079	9932	9932
query12	139	88	88	88
query13	1643	381	363	363
query14	10125	7795	7712	7712
query15	235	194	193	193
query16	7930	267	261	261
query17	1911	537	545	537
query18	2025	270	273	270
query19	209	159	158	158
query20	94	86	82	82
query21	213	128	132	128
query22	4332	4026	4046	4026
query23	33760	33150	33249	33150
query24	12015	2815	2778	2778
query25	652	349	356	349
query26	1836	154	159	154
query27	2925	324	324	324
query28	7507	2028	2021	2021
query29	1073	623	624	623
query30	286	150	155	150
query31	934	697	783	697
query32	92	55	56	55
query33	780	289	278	278
query34	935	476	471	471
query35	726	623	623	623
query36	1113	940	931	931
query37	294	74	71	71
query38	2892	2738	2760	2738
query39	877	799	794	794
query40	282	131	130	130
query41	55	52	53	52
query42	125	101	100	100
query43	589	538	534	534
query44	1229	723	726	723
query45	188	169	167	167
query46	1088	770	730	730
query47	1842	1796	1742	1742
query48	379	304	296	296
query49	1201	409	408	408
query50	759	385	390	385
query51	6849	6634	6622	6622
query52	101	90	95	90
query53	364	297	290	290
query54	966	437	435	435
query55	74	72	74	72
query56	271	255	257	255
query57	1151	1064	1018	1018
query58	257	230	240	230
query59	3377	3242	3192	3192
query60	293	279	291	279
query61	97	93	89	89
query62	657	441	438	438
query63	329	301	296	296
query64	9939	2280	1731	1731
query65	3174	3129	3112	3112
query66	1359	358	352	352
query67	15379	14868	14728	14728
query68	4556	540	533	533
query69	445	355	304	304
query70	1190	1080	1135	1080
query71	392	280	265	265
query72	7189	5397	2819	2819
query73	739	327	331	327
query74	5876	5525	5452	5452
query75	3417	2656	2628	2628
query76	2765	1038	863	863
query77	438	290	307	290
query78	10292	9881	9716	9716
query79	2824	516	518	516
query80	1113	473	459	459
query81	584	220	223	220
query82	780	104	99	99
query83	230	170	177	170
query84	243	88	93	88
query85	1613	293	270	270
query86	469	324	326	324
query87	3288	3098	3067	3067
query88	3714	2336	2354	2336
query89	491	404	401	401
query90	1753	196	199	196
query91	142	112	115	112
query92	66	54	54	54
query93	1905	516	501	501
query94	1232	189	187	187
query95	398	310	313	310
query96	605	270	264	264
query97	3241	2998	3050	2998
query98	209	202	189	189
query99	1184	838	846	838
Total cold run time: 275509 ms
Total hot run time: 170391 ms

@doris-robot
Copy link

doris-robot commented Jun 19, 2024

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87f6f0ac6e1f81cbf99626c05334dba502b7c245, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.06
query5	0.47	0.48	0.46
query6	1.16	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.49	0.49
query10	0.53	0.54	0.53
query11	0.15	0.12	0.12
query12	0.14	0.12	0.12
query13	0.58	0.59	0.60
query14	0.80	0.78	0.80
query15	0.85	0.82	0.82
query16	0.36	0.38	0.37
query17	1.00	0.98	0.96
query18	0.22	0.25	0.24
query19	1.82	1.74	1.70
query20	0.02	0.01	0.00
query21	15.43	0.65	0.65
query22	3.76	7.71	2.04
query23	18.23	1.45	1.25
query24	2.14	0.23	0.22
query25	0.17	0.08	0.08
query26	0.26	0.18	0.17
query27	0.09	0.08	0.09
query28	13.20	1.03	1.02
query29	12.66	3.36	3.33
query30	0.25	0.06	0.06
query31	2.84	0.39	0.40
query32	3.29	0.47	0.47
query33	2.88	2.94	2.91
query34	17.11	4.49	4.39
query35	4.55	4.50	4.52
query36	0.66	0.48	0.47
query37	0.17	0.15	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.15
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109 s
Total hot run time: 30.76 s

seawinde
seawinde reviewed Jun 20, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVRefreshSnapshot.java
public Set<String> getSnapshotPartitions(String mtmvPartitionName) {
MTMVRefreshPartitionSnapshot partitionSnapshot = partitionSnapshots.get(mtmvPartitionName);
if (partitionSnapshot == null) {
return Sets.newHashSet();
Copy link
Contributor

@seawinde seawinde Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe use ImmutebleSet.of() is better

@github-actions
Copy link
Contributor

github-actions bot commented Jun 20, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 20, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 2d6852b into apache:master Jun 20, 2024
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 20, 2024
@zddr
[fix](mtmv)fix when related table drop partition,mv partition is sync (…
b19e008 
…apache#36547)

for example:
if mv's partition [mv_p1] is related with t1's partition [p1,p2]

when p2 dropped,
mv_p1'isSync should be false
@zddr zddr mentioned this pull request Jun 20, 2024
Merged
iszhangpch pushed a commit to iszhangpch/doris-p that referenced this pull request Jun 21, 2024
@zddr
[fix](mtmv)fix when related table drop partition,mv partition is sync (…
16f86c5 
…apache#36547)

for example:
if mv's partition [mv_p1] is related with t1's partition [p1,p2]

when p2 dropped,
mv_p1'isSync should be false
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@zddr @dataroaring
[fix](mtmv)fix when related table drop partition,mv partition is sync (…
34bd385 
…#36547)

for example:
if mv's partition [mv_p1] is related with t1's partition [p1,p2]

when p2 dropped,
mv_p1'isSync should be false
morningman pushed a commit that referenced this pull request Jul 2, 2024
@zddr
[fix](mtmv)fix when related table drop partition,mv partition is sync… (
03942f2 
#36602)

pick: #36547
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@seawinde seawinde seawinde approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zddr @doris-robot @morrySnow @seawinde @morningman @dataroaring