Skip to content

Conversation

@keanji-x
Copy link
Contributor

@keanji-x keanji-x commented Dec 5, 2023

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@keanji-x keanji-x marked this pull request as draft December 5, 2023 09:02
@keanji-x keanji-x marked this pull request as ready for review December 5, 2023 12:01
@keanji-x
Copy link
Contributor Author

keanji-x commented Dec 5, 2023

run buildall

1 similar comment
@keanji-x
Copy link
Contributor Author

keanji-x commented Dec 6, 2023

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.02 seconds
stream load tsv: 558 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17162364046 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit db93a5d665495d3609868e4e9da061fd8af69229, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4702	4423	4414	4414
q2	358	184	156	156
q3	1461	1254	1253	1253
q4	1110	941	931	931
q5	3199	3212	3191	3191
q6	251	130	127	127
q7	1005	501	481	481
q8	2224	2227	2205	2205
q9	6705	6723	6679	6679
q10	3204	3262	3289	3262
q11	316	210	211	210
q12	359	219	216	216
q13	4584	3805	3946	3805
q14	249	214	217	214
q15	556	526	528	526
q16	445	397	392	392
q17	1008	595	556	556
q18	7638	6935	6946	6935
q19	1537	1439	1356	1356
q20	572	1180	313	313
q21	3104	2694	2679	2679
q22	355	288	299	288
Total cold run time: 44942 ms
Total hot run time: 40189 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4384	4353	4379	4353
q2	268	161	175	161
q3	3532	3543	3518	3518
q4	2384	2382	2368	2368
q5	5743	5762	5725	5725
q6	235	123	120	120
q7	2405	1899	1842	1842
q8	3519	3530	3538	3530
q9	9094	9039	8998	8998
q10	3923	3980	4021	3980
q11	503	387	368	368
q12	767	605	597	597
q13	4307	3578	3593	3578
q14	278	246	255	246
q15	569	531	523	523
q16	491	448	458	448
q17	1872	1864	1858	1858
q18	8606	8096	8678	8096
q19	1724	1763	1752	1752
q20	2270	1947	1952	1947
q21	6525	6178	6191	6178
q22	508	433	440	433
Total cold run time: 63907 ms
Total hot run time: 60619 ms

@keanji-x
Copy link
Contributor Author

keanji-x commented Dec 7, 2023

run buildall

@keanji-x
Copy link
Contributor Author

keanji-x commented Dec 7, 2023

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.4 seconds
stream load tsv: 586 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17214655315 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 8cca69f532955f7676cc04011764d23e92c0fd3b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4669	4465	4498	4465
q2	384	172	159	159
q3	1456	1272	1199	1199
q4	1113	942	919	919
q5	3132	3130	3134	3130
q6	250	134	129	129
q7	1005	483	476	476
q8	2192	2208	2171	2171
q9	6698	6689	6664	6664
q10	3218	3262	3259	3259
q11	325	195	207	195
q12	347	201	208	201
q13	4583	3808	3801	3801
q14	238	206	215	206
q15	560	521	529	521
q16	439	383	383	383
q17	994	570	579	570
q18	7478	7249	6966	6966
q19	1514	1374	1406	1374
q20	527	304	312	304
q21	3068	2629	2664	2629
q22	346	283	281	281
Total cold run time: 44536 ms
Total hot run time: 40002 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4392	4416	4416	4416
q2	268	162	166	162
q3	3520	3526	3520	3520
q4	2387	2377	2368	2368
q5	5720	5728	5738	5728
q6	244	122	123	122
q7	2403	1857	1878	1857
q8	3501	3503	3524	3503
q9	8991	9034	8962	8962
q10	3884	4000	3993	3993
q11	495	388	381	381
q12	764	585	601	585
q13	4330	3549	3559	3549
q14	290	252	259	252
q15	573	524	518	518
q16	502	429	499	429
q17	1872	1834	1861	1834
q18	8723	8241	8267	8241
q19	1722	1690	1712	1690
q20	2259	1930	1929	1929
q21	6520	6196	6166	6166
q22	500	432	424	424
Total cold run time: 63860 ms
Total hot run time: 60629 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 11, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@jackwener jackwener merged commit c2d6fbb into apache:master Dec 11, 2023
xzj7019 pushed a commit to xzj7019/doris that referenced this pull request Dec 13, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
morrySnow pushed a commit that referenced this pull request Nov 14, 2024
…n not push down (#43539)

### What problem does this PR solve?

Related PR: #28006

Problem Summary:

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this` leftExtendedNodes` Shoule be contained both orders and lineitem,
which should be 3, but now is 2

This may cause associate wrongly when use hyper graph and materialized
view which depends on it rewrite wrongly

### Release note

Fix generate hyper graph wrongly when has filter which can not push down
github-actions bot pushed a commit that referenced this pull request Nov 14, 2024
…n not push down (#43539)

### What problem does this PR solve?

Related PR: #28006

Problem Summary:

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this` leftExtendedNodes` Shoule be contained both orders and lineitem,
which should be 3, but now is 2

This may cause associate wrongly when use hyper graph and materialized
view which depends on it rewrite wrongly

### Release note

Fix generate hyper graph wrongly when has filter which can not push down
github-actions bot pushed a commit that referenced this pull request Nov 14, 2024
…n not push down (#43539)

### What problem does this PR solve?

Related PR: #28006

Problem Summary:

When query is as following:

 select
              o_orderdate,
              o_shippriority,
              o_comment,
              l_orderkey, 
              l_partkey
            from
              orders left
              join lineitem on l_orderkey = o_orderkey
              left join partsupp on ps_partkey = l_partkey and l_suppkey = ps_suppkey;

hyper graph is as following:
this` leftExtendedNodes` Shoule be contained both orders and lineitem,
which should be 3, but now is 2

This may cause associate wrongly when use hyper graph and materialized
view which depends on it rewrite wrongly

### Release note

Fix generate hyper graph wrongly when has filter which can not push down
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants