Skip to content

Conversation

@seawinde
Copy link
Contributor

Proposed changes

Support to use cast when create sync materialized view
Such as sync mv def is as following:

you can use cast(FLOOR(MINUTE(time) / 15) as decimal(9, 0)) in mv def which is not allowed before.

CREATE MATERIALIZED VIEW sync_mv
AS
SELECT 
  decision,
  code, 
  app_name, 
  event_id, 
  event_type, 
  date_trunc(time, 'minute'), 
  DATE_FORMAT(
    `time`, '%Y-%m-%d'
  ), 
  cast(FLOOR(MINUTE(time) / 15) as decimal(9, 0)),
  count(id) as cnt
from 
  test 
group by 
  code, 
  app_name, 
  event_id, 
  event_type, 
  date_trunc(time, 'minute'), 
  decision, 
  DATE_FORMAT(time, '%Y-%m-%d'), 
  cast(FLOOR(MINUTE(`time`) / 15) as decimal(9, 0));

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow
Copy link
Contributor

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40302 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 33500e8ee2abbc705bebb1746c66a40d2e94c9ef, data reload: false

------ Round 1 ----------------------------------
q1	18143	4427	4409	4409
q2	2872	198	196	196
q3	11081	1182	1179	1179
q4	10286	966	753	753
q5	7583	2790	2657	2657
q6	224	140	142	140
q7	978	619	614	614
q8	9490	2106	2112	2106
q9	8853	6549	6504	6504
q10	8781	3793	3741	3741
q11	449	248	249	248
q12	396	234	230	230
q13	18919	3041	2983	2983
q14	273	229	238	229
q15	527	494	478	478
q16	475	388	388	388
q17	966	708	675	675
q18	8053	7518	7536	7518
q19	7761	1446	1413	1413
q20	675	340	349	340
q21	4903	3216	3931	3216
q22	355	299	285	285
Total cold run time: 122043 ms
Total hot run time: 40302 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4424	4299	4239	4239
q2	384	280	255	255
q3	3021	2767	2752	2752
q4	1895	1591	1647	1591
q5	5295	5329	5335	5329
q6	224	135	136	135
q7	2124	1753	1761	1753
q8	3238	3365	3340	3340
q9	8417	8418	8430	8418
q10	3923	3660	3700	3660
q11	586	507	488	488
q12	759	602	612	602
q13	17578	2973	3000	2973
q14	300	267	286	267
q15	520	477	479	477
q16	474	414	402	402
q17	1752	1494	1508	1494
q18	7726	7548	7420	7420
q19	1654	1455	1675	1455
q20	1983	1778	1762	1762
q21	4815	4743	4725	4725
q22	596	511	504	504
Total cold run time: 71688 ms
Total hot run time: 54041 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172949 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 33500e8ee2abbc705bebb1746c66a40d2e94c9ef, data reload: false

query1	916	373	373	373
query2	6475	1936	1920	1920
query3	6683	209	216	209
query4	28575	17467	17404	17404
query5	4241	472	494	472
query6	263	173	160	160
query7	4587	290	288	288
query8	257	203	199	199
query9	8617	2516	2507	2507
query10	443	303	275	275
query11	12533	10101	10021	10021
query12	130	87	84	84
query13	1632	378	378	378
query14	10311	7830	6893	6893
query15	225	167	165	165
query16	7710	326	321	321
query17	1666	560	537	537
query18	1889	291	281	281
query19	193	155	153	153
query20	93	82	82	82
query21	211	129	142	129
query22	4464	3939	3925	3925
query23	33899	33080	33198	33080
query24	12078	2895	2889	2889
query25	689	393	395	393
query26	1778	155	152	152
query27	2915	281	280	280
query28	7642	2052	2046	2046
query29	1086	646	624	624
query30	291	149	151	149
query31	941	747	752	747
query32	99	53	63	53
query33	805	316	304	304
query34	912	502	482	482
query35	688	595	588	588
query36	1113	947	957	947
query37	184	82	83	82
query38	2911	2755	2748	2748
query39	867	805	836	805
query40	281	121	123	121
query41	49	47	46	46
query42	124	98	97	97
query43	500	468	477	468
query44	1235	749	748	748
query45	194	160	161	160
query46	1093	731	756	731
query47	1875	1811	1816	1811
query48	379	309	301	301
query49	1211	427	431	427
query50	773	401	399	399
query51	6901	6815	6776	6776
query52	101	97	90	90
query53	355	293	291	291
query54	995	468	470	468
query55	78	77	76	76
query56	302	288	294	288
query57	1240	1087	1043	1043
query58	253	240	258	240
query59	3045	2687	2566	2566
query60	296	276	276	276
query61	96	90	107	90
query62	840	657	665	657
query63	326	294	284	284
query64	10495	2237	1712	1712
query65	3199	3265	3155	3155
query66	1375	339	360	339
query67	15538	15103	15027	15027
query68	9134	571	573	571
query69	751	473	357	357
query70	1452	1118	1117	1117
query71	537	283	281	281
query72	9028	5674	5764	5674
query73	2259	326	326	326
query74	6396	5724	5695	5695
query75	4851	2680	2675	2675
query76	5508	925	927	925
query77	708	314	300	300
query78	9710	9085	8931	8931
query79	11131	543	531	531
query80	1584	517	461	461
query81	566	224	231	224
query82	378	131	129	129
query83	245	165	160	160
query84	275	86	92	86
query85	1038	307	315	307
query86	360	305	314	305
query87	3371	3093	3108	3093
query88	4736	2463	2474	2463
query89	556	384	396	384
query90	2012	191	189	189
query91	131	115	99	99
query92	59	49	48	48
query93	7125	527	523	523
query94	1388	217	213	213
query95	408	315	326	315
query96	636	275	273	273
query97	3163	3048	3048	3048
query98	220	196	200	196
query99	1535	1278	1218	1218
Total cold run time: 315412 ms
Total hot run time: 172949 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 33500e8ee2abbc705bebb1746c66a40d2e94c9ef, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.06	0.05
query4	1.67	0.08	0.07
query5	0.49	0.52	0.49
query6	1.14	0.71	0.72
query7	0.03	0.01	0.01
query8	0.06	0.04	0.04
query9	0.56	0.50	0.50
query10	0.54	0.54	0.55
query11	0.14	0.11	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.59
query14	0.76	0.78	0.78
query15	0.85	0.83	0.80
query16	0.35	0.38	0.37
query17	0.98	0.96	1.03
query18	0.23	0.22	0.22
query19	1.78	1.68	1.69
query20	0.01	0.00	0.01
query21	15.42	0.77	0.66
query22	4.19	6.57	2.12
query23	18.23	1.39	1.28
query24	2.09	0.22	0.23
query25	0.16	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.24	0.23
query28	13.28	1.02	0.98
query29	12.61	3.32	3.29
query30	0.27	0.06	0.05
query31	2.87	0.40	0.39
query32	3.26	0.48	0.47
query33	2.84	2.93	2.96
query34	17.09	4.36	4.41
query35	4.45	4.40	4.39
query36	0.66	0.47	0.46
query37	0.19	0.15	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.14	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.59 s
Total hot run time: 30.82 s

@morrySnow morrySnow merged commit 607b86a into apache:master Jul 24, 2024
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
…iew (#38008)

Support to use cast when create sync materialized view
Such as sync mv def is as following:

you can use cast(FLOOR(MINUTE(`time`) / 15) as decimal(9, 0)) in mv
def which is not allowed before.

CREATE MATERIALIZED VIEW sync_mv
AS
SELECT 
  decision,
  code, 
  app_name, 
  event_id, 
  event_type, 
  date_trunc(time, 'minute'), 
  DATE_FORMAT(
    `time`, '%Y-%m-%d'
  ), 
  cast(FLOOR(MINUTE(time) / 15) as decimal(9, 0)),
  count(id) as cnt
from 
  test 
group by 
  code, 
  app_name, 
  event_id, 
  event_type, 
  date_trunc(time, 'minute'), 
  decision, 
  DATE_FORMAT(time, '%Y-%m-%d'), 
  cast(FLOOR(MINUTE(`time`) / 15) as decimal(9, 0));
seawinde added a commit to seawinde/doris that referenced this pull request Aug 6, 2024
…iew (apache#38008)

Support to use cast when create sync materialized view
Such as sync mv def is as following:

you can use cast(FLOOR(MINUTE(`time`) / 15) as decimal(9, 0)) in mv
def which is not allowed before.

CREATE MATERIALIZED VIEW sync_mv
AS
SELECT 
  decision,
  code, 
  app_name, 
  event_id, 
  event_type, 
  date_trunc(time, 'minute'), 
  DATE_FORMAT(
    `time`, '%Y-%m-%d'
  ), 
  cast(FLOOR(MINUTE(time) / 15) as decimal(9, 0)),
  count(id) as cnt
from 
  test 
group by 
  code, 
  app_name, 
  event_id, 
  event_type, 
  date_trunc(time, 'minute'), 
  decision, 
  DATE_FORMAT(time, '%Y-%m-%d'), 
  cast(FLOOR(MINUTE(`time`) / 15) as decimal(9, 0));
yiguolei pushed a commit that referenced this pull request Aug 7, 2024
## Proposed changes

pr: #38115
commitId: 2b29288

pr: #38008
commitId: c6b924d

pr: #37929
commitId: d44fcdc
seawinde added a commit to seawinde/doris that referenced this pull request Aug 14, 2024
dataroaring pushed a commit that referenced this pull request Aug 14, 2024
…lized view #38008 (#39338)

Please enter the commit message for your changes. Lines starting
seawinde added a commit to seawinde/doris that referenced this pull request Aug 14, 2024
yiguolei pushed a commit that referenced this pull request Aug 15, 2024
…lized view #38008 (#39378)

## Proposed changes

this is brought by #38008
if use cast(FLOOR(MINUTE(time) / 15) as decimal(9, 0)) in group by
clause when sync materialized view. if downgrade from 2.1.6 to 2.1.5 or
upgrade 2.1.6 to 3.0.0
this may cause fe can not run. So revert the function.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants