Skip to content

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Aug 6, 2024

Proposed changes

mv def is

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

if we query the sql as following, it will rewrite fail by mv, the fail info is mv can not offer any partition for query

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

This pr fix this problem. it would be rewritten by mv successfully.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 6, 2024
@seawinde
Copy link
Contributor Author

seawinde commented Aug 6, 2024

run buildall

@seawinde seawinde force-pushed the hive_mv_rewrite_fail branch from 53c2111 to 84f3630 Compare August 6, 2024 14:42
@seawinde
Copy link
Contributor Author

seawinde commented Aug 6, 2024

run buildall

Copy link
Contributor

@zfr9527 zfr9527 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Aug 7, 2024

PR approved by anyone and no changes requested.

@seawinde seawinde force-pushed the hive_mv_rewrite_fail branch from 84f3630 to 78fc743 Compare August 8, 2024 07:37
@seawinde
Copy link
Contributor Author

seawinde commented Aug 8, 2024

run buildall

@seawinde seawinde force-pushed the hive_mv_rewrite_fail branch from 78fc743 to be40355 Compare August 9, 2024 02:55
@seawinde
Copy link
Contributor Author

seawinde commented Aug 9, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39685 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be40355cc42d473743c186873eb3e23a1ddede63, data reload: false

------ Round 1 ----------------------------------
q1	18537	4441	4365	4365
q2	2515	179	178	178
q3	11673	1183	1140	1140
q4	10590	776	729	729
q5	7831	2580	2563	2563
q6	229	156	145	145
q7	973	595	594	594
q8	9216	1901	1920	1901
q9	8666	6560	6517	6517
q10	7031	2220	2208	2208
q11	451	254	252	252
q12	389	217	215	215
q13	17763	3006	2966	2966
q14	288	236	237	236
q15	521	486	486	486
q16	490	387	389	387
q17	976	652	672	652
q18	7953	7381	7370	7370
q19	4283	944	1063	944
q20	659	334	346	334
q21	5337	4523	4672	4523
q22	1093	995	980	980
Total cold run time: 117464 ms
Total hot run time: 39685 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4499	4252	4243	4243
q2	384	275	265	265
q3	2858	2644	2654	2644
q4	1910	1634	1571	1571
q5	5256	5252	5252	5252
q6	223	137	136	136
q7	2039	1648	1675	1648
q8	3141	3365	3332	3332
q9	8435	8372	8441	8372
q10	3388	3151	3152	3151
q11	631	496	484	484
q12	756	625	640	625
q13	17659	2979	2986	2979
q14	309	280	273	273
q15	545	481	485	481
q16	476	421	432	421
q17	1752	1460	1465	1460
q18	7608	7462	7479	7462
q19	1662	1498	1670	1498
q20	1981	1773	1780	1773
q21	5180	5145	5139	5139
q22	1083	990	1007	990
Total cold run time: 71775 ms
Total hot run time: 54199 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 202500 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be40355cc42d473743c186873eb3e23a1ddede63, data reload: false

query1	963	411	399	399
query2	6456	1979	1902	1902
query3	6652	212	214	212
query4	34143	23232	23071	23071
query5	4202	495	508	495
query6	297	196	176	176
query7	4581	299	297	297
query8	249	210	196	196
query9	8508	2488	2445	2445
query10	889	900	850	850
query11	17463	15046	14959	14959
query12	159	124	94	94
query13	1651	375	364	364
query14	9773	7803	7838	7803
query15	406	320	317	317
query16	7739	478	470	470
query17	1718	550	523	523
query18	2060	384	409	384
query19	254	217	221	217
query20	111	103	108	103
query21	210	106	100	100
query22	4340	4233	4076	4076
query23	33921	33121	33213	33121
query24	12332	2907	2802	2802
query25	663	372	396	372
query26	1777	155	157	155
query27	3057	278	280	278
query28	7796	2040	2014	2014
query29	1141	403	401	401
query30	306	145	145	145
query31	959	725	758	725
query32	97	54	56	54
query33	743	294	279	279
query34	998	465	472	465
query35	953	861	827	827
query36	1080	918	941	918
query37	289	82	84	82
query38	4319	4181	4088	4088
query39	1417	1353	1345	1345
query40	268	115	117	115
query41	48	44	46	44
query42	111	95	99	95
query43	538	480	482	480
query44	1180	723	737	723
query45	392	373	370	370
query46	1156	759	792	759
query47	1836	1747	1791	1747
query48	368	303	302	302
query49	1209	424	425	424
query50	807	415	419	415
query51	6770	6682	6679	6679
query52	99	94	91	91
query53	257	192	185	185
query54	992	453	443	443
query55	78	73	78	73
query56	281	248	258	248
query57	1142	1081	1070	1070
query58	235	216	219	216
query59	2939	2903	3105	2903
query60	282	259	267	259
query61	101	93	130	93
query62	825	642	655	642
query63	207	186	178	178
query64	10536	2439	1946	1946
query65	3202	3123	3170	3123
query66	1366	348	336	336
query67	15292	14807	14836	14807
query68	4545	547	567	547
query69	413	364	394	364
query70	1136	1106	1092	1092
query71	402	332	265	265
query72	17839	16868	16344	16344
query73	757	332	340	332
query74	9356	8707	8840	8707
query75	3363	2683	2692	2683
query76	2579	1031	910	910
query77	464	319	306	306
query78	12959	9348	8889	8889
query79	2888	512	521	512
query80	1160	496	527	496
query81	585	225	234	225
query82	551	143	129	129
query83	325	154	158	154
query84	283	84	82	82
query85	730	320	311	311
query86	484	298	282	282
query87	4799	4532	4567	4532
query88	4778	2508	2580	2508
query89	402	288	291	288
query90	2037	202	206	202
query91	147	133	208	133
query92	63	50	51	50
query93	3556	543	534	534
query94	920	302	284	284
query95	360	273	267	267
query96	615	282	274	274
query97	3219	3039	3029	3029
query98	237	198	201	198
query99	1623	1281	1290	1281
Total cold run time: 320764 ms
Total hot run time: 202500 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.02 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be40355cc42d473743c186873eb3e23a1ddede63, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.07
query5	0.50	0.48	0.48
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.49	0.50
query10	0.53	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.61	0.57
query14	0.76	0.78	0.79
query15	0.88	0.82	0.81
query16	0.37	0.36	0.37
query17	1.02	0.96	0.95
query18	0.24	0.22	0.22
query19	1.82	1.70	1.72
query20	0.01	0.02	0.01
query21	15.43	0.73	0.65
query22	4.13	8.24	1.40
query23	18.29	1.35	1.26
query24	2.13	0.21	0.22
query25	0.16	0.09	0.08
query26	0.29	0.20	0.21
query27	0.45	0.24	0.23
query28	13.32	1.01	1.00
query29	12.64	3.35	3.29
query30	0.24	0.05	0.05
query31	2.90	0.41	0.39
query32	3.25	0.48	0.48
query33	2.91	2.91	2.89
query34	17.08	4.33	4.33
query35	4.43	4.45	4.41
query36	0.64	0.48	0.47
query37	0.19	0.15	0.15
query38	0.16	0.16	0.14
query39	0.05	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.83 s
Total hot run time: 30.02 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 9, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Aug 9, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 08c9e05 into apache:master Aug 9, 2024
seawinde added a commit to seawinde/doris that referenced this pull request Aug 9, 2024
…le (apache#38909)

mv def is 

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

if we query the sql as following, it will rewrite fail by mv, the fail
info is `mv can not offer any partition for query`

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

This pr fix this problem. it would be rewritten by mv successfully.
yiguolei pushed a commit that referenced this pull request Aug 10, 2024
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
…le (#38909)

mv def is 

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

if we query the sql as following, it will rewrite fail by mv, the fail
info is `mv can not offer any partition for query`

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

This pr fix this problem. it would be rewritten by mv successfully.
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
…le (apache#38909)

mv def is 

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

if we query the sql as following, it will rewrite fail by mv, the fail
info is `mv can not offer any partition for query`

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

This pr fix this problem. it would be rewritten by mv successfully.
dataroaring pushed a commit that referenced this pull request Aug 16, 2024
…le (#38909)

mv def is 

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

if we query the sql as following, it will rewrite fail by mv, the fail
info is `mv can not offer any partition for query`

            select l_orderkey, l_partkey, o_custkey, l_shipdate, o_orderdate 
            from ${hive_catalog_name}.${hive_database}.${hive_table} 
            left join ${internal_catalog}.${olap_db}.${olap_table} on l_orderkey = o_orderkey 

This pr fix this problem. it would be rewritten by mv successfully.
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.2-merged doing reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants