[enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change #36513

zddr · 2024-06-19T03:54:24Z

The following behavior will change the state of the materialized view to SCHEMA-CHANGE

column rename
drop column
modify column
replace table
table rename
drop table

doris-robot · 2024-06-19T03:54:28Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zddr · 2024-06-19T03:54:41Z

run buildall

doris-robot · 2024-06-19T04:23:56Z

TPC-H: Total hot run time: 39435 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7e2a1565efaa162a1f83ecf08ec405f3de7e8260, data reload: false

------ Round 1 ----------------------------------
q1	17603	4301	4230	4230
q2	2027	195	192	192
q3	10451	1107	1161	1107
q4	10189	709	803	709
q5	7575	2673	2635	2635
q6	218	135	133	133
q7	945	611	594	594
q8	9228	2040	2077	2040
q9	8953	6462	6408	6408
q10	9049	3723	3709	3709
q11	455	237	237	237
q12	481	225	225	225
q13	17765	2981	2979	2979
q14	283	221	222	221
q15	514	473	455	455
q16	510	375	381	375
q17	949	613	727	613
q18	7971	7341	7318	7318
q19	6099	1497	1520	1497
q20	651	315	321	315
q21	4918	3782	3113	3113
q22	385	330	333	330
Total cold run time: 117219 ms
Total hot run time: 39435 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4212	4221	4212
q2	357	300	253	253
q3	2962	2912	2963	2912
q4	1959	1651	1651	1651
q5	5647	5575	5571	5571
q6	223	136	133	133
q7	2335	1992	1916	1916
q8	3300	3472	3416	3416
q9	8670	8710	8720	8710
q10	4080	3756	3775	3756
q11	589	498	517	498
q12	823	645	652	645
q13	15958	3171	3128	3128
q14	319	291	263	263
q15	523	478	486	478
q16	485	445	436	436
q17	1798	1521	1509	1509
q18	8053	7855	7920	7855
q19	1806	1605	1736	1605
q20	3046	1925	1867	1867
q21	8809	4888	4802	4802
q22	635	526	559	526
Total cold run time: 76758 ms
Total hot run time: 56142 ms

doris-robot · 2024-06-19T04:34:52Z

TPC-DS: Total hot run time: 173206 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7e2a1565efaa162a1f83ecf08ec405f3de7e8260, data reload: false

query1	923	389	380	380
query2	6471	2366	2305	2305
query3	6627	207	209	207
query4	20980	17442	17233	17233
query5	3658	486	470	470
query6	244	163	159	159
query7	4584	311	293	293
query8	305	298	293	293
query9	8514	2430	2405	2405
query10	567	326	283	283
query11	10655	10049	10073	10049
query12	120	92	85	85
query13	1645	370	373	370
query14	9411	6941	7894	6941
query15	245	192	193	192
query16	7796	281	274	274
query17	1871	560	534	534
query18	1929	284	275	275
query19	206	161	161	161
query20	94	86	90	86
query21	209	131	129	129
query22	4579	4109	3946	3946
query23	33833	33766	33630	33630
query24	10709	2873	2834	2834
query25	596	404	369	369
query26	706	160	158	158
query27	2253	320	323	320
query28	6086	2114	2083	2083
query29	887	638	612	612
query30	268	159	157	157
query31	964	776	757	757
query32	90	50	56	50
query33	653	286	275	275
query34	885	483	482	482
query35	749	682	650	650
query36	1150	984	987	984
query37	148	70	72	70
query38	2973	2822	2861	2822
query39	872	832	832	832
query40	210	129	128	128
query41	65	60	51	51
query42	115	96	100	96
query43	565	547	531	531
query44	1101	710	706	706
query45	198	164	158	158
query46	1067	728	724	724
query47	1844	1793	1803	1793
query48	358	290	285	285
query49	847	395	402	395
query50	761	385	388	385
query51	6779	6637	6656	6637
query52	100	100	93	93
query53	361	298	286	286
query54	843	450	428	428
query55	74	71	76	71
query56	273	249	252	249
query57	1119	1036	1065	1036
query58	234	240	248	240
query59	3404	3231	3120	3120
query60	289	265	267	265
query61	117	87	98	87
query62	621	433	455	433
query63	317	286	289	286
query64	8633	2238	1727	1727
query65	3141	3114	3113	3113
query66	718	323	322	322
query67	15332	14999	15037	14999
query68	6177	539	544	539
query69	648	524	386	386
query70	1203	1080	1157	1080
query71	445	311	261	261
query72	8149	5175	5527	5175
query73	768	319	321	319
query74	5838	5567	5498	5498
query75	3579	2665	2664	2664
query76	3203	944	923	923
query77	645	294	295	294
query78	10357	9811	9737	9737
query79	2104	511	510	510
query80	1123	458	456	456
query81	565	217	222	217
query82	1228	104	105	104
query83	209	166	170	166
query84	255	87	87	87
query85	1219	293	279	279
query86	466	337	324	324
query87	3256	3083	3034	3034
query88	4041	2337	2345	2337
query89	490	382	407	382
query90	1692	188	191	188
query91	146	100	102	100
query92	60	50	54	50
query93	2114	505	493	493
query94	1023	256	190	190
query95	418	313	315	313
query96	581	271	265	265
query97	3211	3024	3049	3024
query98	228	197	192	192
query99	1168	820	836	820
Total cold run time: 271550 ms
Total hot run time: 173206 ms

doris-robot · 2024-06-19T04:40:22Z

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7e2a1565efaa162a1f83ecf08ec405f3de7e8260, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.22	0.05	0.04
query4	1.68	0.07	0.08
query5	0.50	0.48	0.49
query6	1.12	0.72	0.73
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.48	0.49
query10	0.56	0.56	0.55
query11	0.14	0.12	0.11
query12	0.15	0.13	0.13
query13	0.59	0.59	0.58
query14	0.77	0.78	0.79
query15	0.83	0.81	0.81
query16	0.37	0.36	0.38
query17	1.00	0.99	0.95
query18	0.22	0.25	0.24
query19	1.75	1.82	1.70
query20	0.02	0.01	0.01
query21	15.41	0.64	0.64
query22	4.59	7.05	2.03
query23	18.30	1.34	1.32
query24	2.11	0.22	0.21
query25	0.15	0.09	0.09
query26	0.27	0.18	0.18
query27	0.08	0.07	0.08
query28	13.29	1.04	0.99
query29	12.59	3.30	3.28
query30	0.26	0.06	0.06
query31	2.85	0.40	0.38
query32	3.27	0.47	0.47
query33	2.87	2.91	2.97
query34	17.00	4.34	4.38
query35	4.46	4.50	4.56
query36	0.67	0.45	0.48
query37	0.17	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.06
query43	0.04	0.05	0.04
Total cold run time: 109.56 s
Total hot run time: 30.65 s

zddr · 2024-06-19T06:58:22Z

run p0

zddr · 2024-06-19T10:23:38Z

run buildall

doris-robot · 2024-06-19T10:52:12Z

TPC-H: Total hot run time: 39701 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdc0bc36f976189b3253dcc35e598b0e14dbc68c, data reload: false

------ Round 1 ----------------------------------
q1	17620	4287	4202	4202
q2	2023	191	199	191
q3	10449	1160	1087	1087
q4	10193	898	778	778
q5	7492	2658	2649	2649
q6	223	137	140	137
q7	975	631	616	616
q8	9215	2055	2089	2055
q9	9081	6485	6471	6471
q10	8999	3724	3701	3701
q11	450	237	248	237
q12	460	238	231	231
q13	17762	3033	3035	3033
q14	264	227	225	225
q15	530	470	474	470
q16	540	384	390	384
q17	979	630	663	630
q18	7985	7411	7342	7342
q19	6514	1424	1431	1424
q20	660	315	322	315
q21	4904	3177	3237	3177
q22	396	347	346	346
Total cold run time: 117714 ms
Total hot run time: 39701 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4393	4210	4251	4210
q2	374	274	279	274
q3	2991	3011	2895	2895
q4	2016	1674	1741	1674
q5	5618	5531	5509	5509
q6	219	135	147	135
q7	2205	1914	1884	1884
q8	3334	3462	3392	3392
q9	8668	8702	8786	8702
q10	4117	3880	3768	3768
q11	599	491	501	491
q12	832	629	638	629
q13	16263	3172	3228	3172
q14	307	290	279	279
q15	524	487	487	487
q16	494	439	454	439
q17	1823	1519	1492	1492
q18	8179	7924	7946	7924
q19	1850	1512	1605	1512
q20	3056	1875	1848	1848
q21	5205	4988	4756	4756
q22	626	543	588	543
Total cold run time: 73693 ms
Total hot run time: 56015 ms

doris-robot · 2024-06-19T11:03:08Z

TPC-DS: Total hot run time: 174037 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bdc0bc36f976189b3253dcc35e598b0e14dbc68c, data reload: false

query1	920	377	382	377
query2	6454	2461	2369	2369
query3	6634	204	207	204
query4	19366	17337	17298	17298
query5	3629	498	453	453
query6	237	160	163	160
query7	4588	299	290	290
query8	329	309	300	300
query9	8729	2377	2366	2366
query10	561	301	274	274
query11	10626	10040	9937	9937
query12	118	81	95	81
query13	1633	356	377	356
query14	9392	7728	7743	7728
query15	249	200	200	200
query16	7731	275	271	271
query17	1768	554	528	528
query18	1915	283	273	273
query19	224	162	153	153
query20	105	85	85	85
query21	209	143	132	132
query22	4652	4413	4570	4413
query23	34060	33376	33492	33376
query24	10994	2775	2822	2775
query25	588	357	355	355
query26	732	156	152	152
query27	2229	310	317	310
query28	6393	2043	2034	2034
query29	862	612	608	608
query30	216	152	150	150
query31	958	744	735	735
query32	94	50	57	50
query33	776	303	289	289
query34	885	475	475	475
query35	746	610	635	610
query36	1095	930	966	930
query37	137	70	69	69
query38	2829	2733	2748	2733
query39	859	799	838	799
query40	207	127	129	127
query41	57	52	53	52
query42	114	98	98	98
query43	585	545	531	531
query44	1234	728	726	726
query45	201	171	164	164
query46	1080	701	701	701
query47	1882	1751	1779	1751
query48	374	291	333	291
query49	861	397	412	397
query50	770	395	394	394
query51	6828	6595	6600	6595
query52	109	91	93	91
query53	360	287	284	284
query54	920	454	433	433
query55	76	73	70	70
query56	282	264	255	255
query57	1102	1086	1004	1004
query58	253	246	236	236
query59	3474	3021	3254	3021
query60	287	275	278	275
query61	92	91	122	91
query62	613	456	459	456
query63	317	286	295	286
query64	8748	2237	1720	1720
query65	3168	3085	3100	3085
query66	748	340	327	327
query67	15421	15108	14808	14808
query68	4849	590	556	556
query69	625	461	437	437
query70	1184	1132	1105	1105
query71	485	270	267	267
query72	8218	5956	5754	5754
query73	777	323	325	323
query74	5814	5460	5387	5387
query75	3679	2601	2664	2601
query76	3474	960	930	930
query77	656	304	313	304
query78	10290	9818	9934	9818
query79	2205	517	522	517
query80	1478	467	461	461
query81	591	221	223	221
query82	814	101	102	101
query83	264	165	165	165
query84	258	85	89	85
query85	1401	276	271	271
query86	477	331	339	331
query87	3261	3098	3069	3069
query88	3786	2326	2327	2326
query89	470	387	386	386
query90	1718	191	188	188
query91	127	104	97	97
query92	58	48	53	48
query93	2024	501	508	501
query94	1086	189	184	184
query95	400	322	320	320
query96	597	263	270	263
query97	3201	3042	3031	3031
query98	231	279	198	198
query99	1184	854	825	825
Total cold run time: 270367 ms
Total hot run time: 174037 ms

doris-robot · 2024-06-19T11:08:38Z

ClickBench: Total hot run time: 30.48 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bdc0bc36f976189b3253dcc35e598b0e14dbc68c, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.08
query5	0.50	0.48	0.50
query6	1.13	0.74	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.54	0.53
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.77	0.79
query15	0.84	0.82	0.81
query16	0.37	0.37	0.36
query17	1.03	1.02	1.03
query18	0.22	0.24	0.25
query19	1.79	1.68	1.69
query20	0.01	0.01	0.00
query21	15.42	0.65	0.64
query22	4.23	7.61	1.79
query23	18.27	1.40	1.28
query24	2.10	0.23	0.20
query25	0.16	0.09	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.22	1.02	1.01
query29	12.66	3.35	3.32
query30	0.26	0.06	0.06
query31	2.86	0.39	0.38
query32	3.27	0.47	0.47
query33	2.88	2.92	2.91
query34	17.01	4.40	4.48
query35	4.47	4.50	4.49
query36	0.66	0.49	0.47
query37	0.17	0.14	0.14
query38	0.15	0.14	0.14
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.10	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.28 s
Total hot run time: 30.48 s

seawinde · 2024-06-20T07:35:00Z

fe/fe-core/src/main/java/org/apache/doris/analysis/AlterTableClause.java


    public abstract boolean allowOpMTMV();
+
+    public abstract boolean needChangeMTMVState();


this may be add a default impl and value such as true, when add new cluse later, maybe safe

This is intentional to avoid ignoring this method when creating new classes, and adding some comments in other PRs

github-actions · 2024-06-20T07:40:09Z

PR approved by anyone and no changes requested.

github-actions · 2024-06-20T12:23:44Z

PR approved by at least one committer and no changes requested.

…ange to schema_change (apache#36513) The following behavior will change the state of the materialized view to SCHEMA-CHANGE - column rename - drop column - modify column - replace table - table rename - drop table

…ange to schema_change (#36513) The following behavior will change the state of the materialized view to SCHEMA-CHANGE - column rename - drop column - modify column - replace table - table rename - drop table

…ange to schema_change (apache#36513) The following behavior will change the state of the materialized view to SCHEMA-CHANGE - column rename - drop column - modify column - replace table - table rename - drop table

…ange to schema_change (#36513) (#37122) pick from #36513

1

7e2a156

1

d6b8b96

zddr marked this pull request as draft June 19, 2024 09:18

zddr changed the title ~~[fix](mtmv)fix when related table drop partition,mv partition is sync~~ [enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change Jun 19, 2024

zddr added 6 commits June 19, 2024 17:23

1

402b395

1

ae36197

1

3dca313

1

53ac3b8

1

9e06a57

1

7b1f118

zddr marked this pull request as ready for review June 19, 2024 10:22

1

bdc0bc3

seawinde reviewed Jun 20, 2024

View reviewed changes

seawinde approved these changes Jun 20, 2024

View reviewed changes

github-actions bot added the reviewed label Jun 20, 2024

morrySnow approved these changes Jun 20, 2024

View reviewed changes

morrySnow added the dev/2.1.x label Jun 20, 2024

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024

morrySnow merged commit 7bb1944 into apache:master Jun 20, 2024

zddr mentioned this pull request Jul 2, 2024

[enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change (#36513) #37122

Merged

morningman pushed a commit that referenced this pull request Jul 3, 2024

[enhance](mtmv)reduce the behavior of triggering the mtmv state to ch…

6804ae5

…ange to schema_change (#36513) (#37122) pick from #36513

morningman added dev/2.1.5-merged and removed dev/2.1.x labels Jul 3, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024


		public abstract boolean allowOpMTMV();

		public abstract boolean needChangeMTMVState();

[enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change #36513

[enhance](mtmv)reduce the behavior of triggering the mtmv state to change to schema_change #36513

Uh oh!

Conversation

zddr commented Jun 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

zddr commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

zddr commented Jun 19, 2024

Uh oh!

zddr commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

seawinde Jun 20, 2024

Choose a reason for hiding this comment

Uh oh!

zddr Jun 20, 2024

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

zddr commented Jun 19, 2024 •

edited

Loading