Skip to content

[fix](mtmv)Fix exception when create materialized view with cte #33988

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Apr 24, 2024
Merged

[fix](mtmv)Fix exception when create materialized view with cte #33988

morrySnow merged 1 commit into from
Apr 24, 2024

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Apr 22, 2024

Proposed changes

Fix exception when create materialized view with cte, after this fix, can create materialized view with following

        CREATE MATERIALIZED VIEW mv_with_cte
            BUILD IMMEDIATE REFRESH AUTO ON MANUAL
            DISTRIBUTED BY RANDOM BUCKETS 2
            PROPERTIES ('replication_num' = '1')
            AS
            with `test_with` AS (
            select l_partkey, l_suppkey
            from lineitem
            union
            select
              ps_partkey, ps_suppkey
            from
            partsupp)
            select * from test_with;

this is brought from #28144

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 22, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

seawinde commented Apr 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 22, 2024

TPC-H: Total hot run time: 38455 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f127a715fa6177d90e401e10b17ba092112a187a, data reload: false

------ Round 1 ----------------------------------
q1	17611	4479	4413	4413
q2	2002	180	177	177
q3	10461	1127	1173	1127
q4	10197	671	725	671
q5	7514	2672	2603	2603
q6	214	128	127	127
q7	997	612	570	570
q8	9229	2033	2080	2033
q9	7326	6587	6482	6482
q10	8537	3492	3507	3492
q11	455	228	233	228
q12	562	228	214	214
q13	17766	2942	2933	2933
q14	273	227	222	222
q15	516	477	478	477
q16	518	384	370	370
q17	943	700	699	699
q18	7449	6731	6713	6713
q19	6750	1532	1502	1502
q20	645	303	297	297
q21	3788	2811	2804	2804
q22	357	313	301	301
Total cold run time: 114110 ms
Total hot run time: 38455 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4343	4172	4174	4172
q2	368	255	261	255
q3	2998	2692	2711	2692
q4	1869	1559	1627	1559
q5	5331	5344	5294	5294
q6	208	120	125	120
q7	2235	1865	1854	1854
q8	3199	3333	3322	3322
q9	8583	8497	8505	8497
q10	3904	3651	3720	3651
q11	580	489	496	489
q12	744	582	583	582
q13	17210	2957	2943	2943
q14	306	288	265	265
q15	503	469	461	461
q16	477	412	409	409
q17	1761	1469	1454	1454
q18	7957	8225	7920	7920
q19	1680	1552	1554	1552
q20	2019	1851	1873	1851
q21	10375	5093	4961	4961
q22	576	492	480	480
Total cold run time: 77226 ms
Total hot run time: 54783 ms

@github-actions
Copy link
Contributor

github-actions bot commented Apr 23, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 23, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 23, 2024

PR approved by at least one committer and no changes requested.

@morrySnow
Copy link
Contributor

morrySnow commented Apr 23, 2024

run p0

@morrySnow morrySnow merged commit 19fed26 into apache:master Apr 24, 2024
yiguolei pushed a commit that referenced this pull request Apr 24, 2024
@seawinde @Doris-Extras
[fix](mtmv) Fix exception when create materialized view with cte (#33988
ec0235d 
)

Fix exception when create materialized view with cte, after this fix, can create materialized view with following
```
        CREATE MATERIALIZED VIEW mv_with_cte
            BUILD IMMEDIATE REFRESH AUTO ON MANUAL
            DISTRIBUTED BY RANDOM BUCKETS 2
            PROPERTIES ('replication_num' = '1')
            AS
            with `test_with` AS (
            select l_partkey, l_suppkey
            from lineitem
            union
            select
              ps_partkey, ps_suppkey
            from
            partsupp)
            select * from test_with;
```

this is brought from #28144
yiguolei pushed a commit that referenced this pull request Apr 25, 2024
@seawinde @Doris-Extras
[fix](mtmv) Fix exception when create materialized view with cte (#33988
a15a8e1 
)

Fix exception when create materialized view with cte, after this fix, can create materialized view with following
```
        CREATE MATERIALIZED VIEW mv_with_cte
            BUILD IMMEDIATE REFRESH AUTO ON MANUAL
            DISTRIBUTED BY RANDOM BUCKETS 2
            PROPERTIES ('replication_num' = '1')
            AS
            with `test_with` AS (
            select l_partkey, l_suppkey
            from lineitem
            union
            select
              ps_partkey, ps_suppkey
            from
            partsupp)
            select * from test_with;
```

this is brought from #28144
@seawinde seawinde mentioned this pull request Dec 2, 2024
Merged
16 tasks
morrySnow pushed a commit that referenced this pull request Dec 5, 2024
@seawinde
[fix](mtmv) Fix refresh materialized view fail when mv def contains c…
8ea48af 
…te (#44857)

### What problem does this PR solve?

Related PR: #33988

Problem Summary:

if mv def contain cte and is partition mv, such as following:

    CREATE MATERIALIZED VIEW mv_name
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        PARTITION BY (l_shipdate)
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    WITH scan_data_cte as (
        select t1.l_shipdate, t1.L_LINENUMBER, orders.O_CUSTKEY, l_suppkey
        from (select * from lineitem where L_LINENUMBER > 1) t1
        left join orders on t1.L_ORDERKEY = orders.O_ORDERKEY
    )
    SELECT *  FROM scan_data_cte; 

if run refresh cmd as following, this would fail, throw exception `no
partition for this tuple`, this pr fix this

refresh materialized view mv_name partitions(p_20231210_20231211);

### Release note

Fix refresh materialized view fail when mv def contains cte
github-actions bot pushed a commit that referenced this pull request Dec 5, 2024
@seawinde
[fix](mtmv) Fix refresh materialized view fail when mv def contains c…
245ce79 
…te (#44857)

### What problem does this PR solve?

Related PR: #33988

Problem Summary:

if mv def contain cte and is partition mv, such as following:

    CREATE MATERIALIZED VIEW mv_name
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        PARTITION BY (l_shipdate)
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    WITH scan_data_cte as (
        select t1.l_shipdate, t1.L_LINENUMBER, orders.O_CUSTKEY, l_suppkey
        from (select * from lineitem where L_LINENUMBER > 1) t1
        left join orders on t1.L_ORDERKEY = orders.O_ORDERKEY
    )
    SELECT *  FROM scan_data_cte; 

if run refresh cmd as following, this would fail, throw exception `no
partition for this tuple`, this pr fix this

refresh materialized view mv_name partitions(p_20231210_20231211);

### Release note

Fix refresh materialized view fail when mv def contains cte
github-actions bot pushed a commit that referenced this pull request Dec 5, 2024
@seawinde
[fix](mtmv) Fix refresh materialized view fail when mv def contains c…
5732812 
…te (#44857)

### What problem does this PR solve?

Related PR: #33988

Problem Summary:

if mv def contain cte and is partition mv, such as following:

    CREATE MATERIALIZED VIEW mv_name
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        PARTITION BY (l_shipdate)
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    WITH scan_data_cte as (
        select t1.l_shipdate, t1.L_LINENUMBER, orders.O_CUSTKEY, l_suppkey
        from (select * from lineitem where L_LINENUMBER > 1) t1
        left join orders on t1.L_ORDERKEY = orders.O_ORDERKEY
    )
    SELECT *  FROM scan_data_cte; 

if run refresh cmd as following, this would fail, throw exception `no
partition for this tuple`, this pr fix this

refresh materialized view mv_name partitions(p_20231210_20231211);

### Release note

Fix refresh materialized view fail when mv def contains cte
yiguolei pushed a commit that referenced this pull request Dec 6, 2024
@seawinde @yiguolei
[fix](mtmv) Fix refresh materialized view fail when mv def contains c…
290532e 
…te (#44857)

### What problem does this PR solve?

Related PR: #33988

Problem Summary:

if mv def contain cte and is partition mv, such as following:

    CREATE MATERIALIZED VIEW mv_name
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        PARTITION BY (l_shipdate)
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    WITH scan_data_cte as (
        select t1.l_shipdate, t1.L_LINENUMBER, orders.O_CUSTKEY, l_suppkey
        from (select * from lineitem where L_LINENUMBER > 1) t1
        left join orders on t1.L_ORDERKEY = orders.O_ORDERKEY
    )
    SELECT *  FROM scan_data_cte; 

if run refresh cmd as following, this would fail, throw exception `no
partition for this tuple`, this pr fix this

refresh materialized view mv_name partitions(p_20231210_20231211);

### Release note

Fix refresh materialized view fail when mv def contains cte
seawinde added a commit to seawinde/doris that referenced this pull request Dec 6, 2024
@seawinde
[fix](mtmv) Fix refresh materialized view fail when mv def contains c…
bc8258a 
…te (apache#44857)

### What problem does this PR solve?

Related PR: apache#33988

Problem Summary:

if mv def contain cte and is partition mv, such as following:

    CREATE MATERIALIZED VIEW mv_name
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        PARTITION BY (l_shipdate)
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    WITH scan_data_cte as (
        select t1.l_shipdate, t1.L_LINENUMBER, orders.O_CUSTKEY, l_suppkey
        from (select * from lineitem where L_LINENUMBER > 1) t1
        left join orders on t1.L_ORDERKEY = orders.O_ORDERKEY
    )
    SELECT *  FROM scan_data_cte; 

if run refresh cmd as following, this would fail, throw exception `no
partition for this tuple`, this pr fix this

refresh materialized view mv_name partitions(p_20231210_20231211);

### Release note

Fix refresh materialized view fail when mv def contains cte
seawinde added a commit to seawinde/doris that referenced this pull request Dec 6, 2024
@seawinde
[fix](mtmv) Fix refresh materialized view fail when mv def contains c…
d40c427 
…te (apache#44857)

### What problem does this PR solve?

Related PR: apache#33988

Problem Summary:

if mv def contain cte and is partition mv, such as following:

    CREATE MATERIALIZED VIEW mv_name
        BUILD IMMEDIATE REFRESH COMPLETE ON MANUAL
        PARTITION BY (l_shipdate)
        DISTRIBUTED BY RANDOM BUCKETS 2
        PROPERTIES ('replication_num' = '1') 
        AS
    WITH scan_data_cte as (
        select t1.l_shipdate, t1.L_LINENUMBER, orders.O_CUSTKEY, l_suppkey
        from (select * from lineitem where L_LINENUMBER > 1) t1
        left join orders on t1.L_ORDERKEY = orders.O_ORDERKEY
    )
    SELECT *  FROM scan_data_cte; 

if run refresh cmd as following, this would fail, throw exception `no
partition for this tuple`, this pr fix this

refresh materialized view mv_name partitions(p_20231210_20231211);

### Release note

Fix refresh materialized view fail when mv def contains cte
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@keanji-x keanji-x keanji-x approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@seawinde @doris-robot @morrySnow @keanji-x