Skip to content

Conversation

@LiBinfeng-01
Copy link
Contributor

isLeading in cascade context means do we have leading in this scope
isLeadingDisableJoinReorder means does leading used
we need to copy it after analyze outerside of cte

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@LiBinfeng-01
Copy link
Contributor Author

run buildall

CascadesContext outerCascadesCtx = CascadesContext.newContextWithCteContext(
ctx.cascadesContext, logicalCTE.child(), result.first);
outerCascadesCtx.newAnalyzer().analyze();
ctx.cascadesContext.setLeadingDisableJoinReorder(outerCascadesCtx.isLeadingDisableJoinReorder());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add test case please

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@doris-robot
Copy link

TPC-H: Total hot run time: 41958 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 469bfb819f4aa1db0fb540d4c991ebbc0993e0a3, data reload: false

------ Round 1 ----------------------------------
q1	18124	7527	7392	7392
q2	3136	170	178	170
q3	11830	1156	1218	1156
q4	10580	789	804	789
q5	7766	3123	3099	3099
q6	243	150	151	150
q7	1034	636	616	616
q8	9425	2029	2041	2029
q9	6713	6366	6404	6366
q10	7025	2294	2294	2294
q11	432	251	257	251
q12	406	227	222	222
q13	17789	3016	3002	3002
q14	244	216	219	216
q15	575	523	506	506
q16	671	617	602	602
q17	988	824	819	819
q18	7181	6781	6658	6658
q19	1404	1049	1014	1014
q20	582	310	285	285
q21	4010	3388	3315	3315
q22	1121	1011	1007	1007
Total cold run time: 111279 ms
Total hot run time: 41958 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7289	7268	7268
q2	334	223	234	223
q3	2897	2780	2756	2756
q4	1932	1700	1777	1700
q5	5350	5398	5378	5378
q6	228	143	148	143
q7	2101	1676	1717	1676
q8	3171	3314	3311	3311
q9	8404	8445	8350	8350
q10	3400	3317	3328	3317
q11	569	461	479	461
q12	819	585	579	579
q13	5863	2988	3045	2988
q14	287	260	269	260
q15	570	519	501	501
q16	699	689	663	663
q17	1764	1560	1544	1544
q18	7778	7354	7301	7301
q19	1693	1625	1676	1625
q20	2057	1808	1789	1789
q21	5523	5426	5340	5340
q22	1150	1030	1008	1008
Total cold run time: 63887 ms
Total hot run time: 58181 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194491 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 469bfb819f4aa1db0fb540d4c991ebbc0993e0a3, data reload: false

query1	950	367	363	363
query2	6512	2016	2018	2016
query3	6711	213	223	213
query4	34197	23394	23372	23372
query5	4377	487	461	461
query6	260	182	157	157
query7	4621	292	301	292
query8	272	212	212	212
query9	9738	2663	2662	2662
query10	489	286	277	277
query11	18259	15078	15230	15078
query12	151	97	100	97
query13	1616	424	386	386
query14	10227	7398	7590	7398
query15	267	172	189	172
query16	7603	499	475	475
query17	1674	573	552	552
query18	1905	319	303	303
query19	363	155	154	154
query20	116	113	110	110
query21	215	105	104	104
query22	4356	4307	4392	4307
query23	34611	34784	33942	33942
query24	11288	2969	2919	2919
query25	578	404	406	404
query26	748	158	168	158
query27	2615	291	290	290
query28	8039	2464	2423	2423
query29	662	440	436	436
query30	324	160	148	148
query31	995	782	815	782
query32	99	56	57	56
query33	771	294	304	294
query34	942	497	512	497
query35	882	720	725	720
query36	1084	956	959	956
query37	150	92	90	90
query38	4082	3860	3862	3860
query39	1471	1405	1421	1405
query40	215	97	99	97
query41	53	49	51	49
query42	116	100	99	99
query43	519	489	493	489
query44	1250	830	795	795
query45	199	164	171	164
query46	1147	794	769	769
query47	1881	1780	1824	1780
query48	471	368	363	363
query49	1141	417	417	417
query50	822	398	399	398
query51	6897	6946	6745	6745
query52	99	85	90	85
query53	260	190	181	181
query54	1201	468	460	460
query55	83	77	78	77
query56	285	260	254	254
query57	1188	1082	1124	1082
query58	260	227	231	227
query59	3175	2939	2832	2832
query60	295	273	278	273
query61	130	123	127	123
query62	880	668	659	659
query63	227	222	190	190
query64	5165	650	645	645
query65	3302	3189	3217	3189
query66	1325	299	304	299
query67	15898	15468	15377	15377
query68	3117	577	566	566
query69	431	290	292	290
query70	1213	1149	1139	1139
query71	336	262	280	262
query72	5765	4014	3990	3990
query73	761	328	325	325
query74	9553	8986	9018	8986
query75	3348	2660	2634	2634
query76	2307	891	906	891
query77	409	293	285	285
query78	9848	9618	9530	9530
query79	1167	877	874	874
query80	867	587	575	575
query81	512	249	248	248
query82	854	240	228	228
query83	230	161	159	159
query84	250	109	106	106
query85	772	382	362	362
query86	320	312	324	312
query87	4397	4344	4317	4317
query88	4697	4050	4073	4050
query89	383	366	367	366
query90	1839	309	312	309
query91	185	164	169	164
query92	76	71	71	71
query93	929	913	903	903
query94	718	387	378	378
query95	463	422	414	414
query96	486	488	485	485
query97	3121	3117	3125	3117
query98	227	224	223	223
query99	1453	1280	1273	1273
Total cold run time: 293738 ms
Total hot run time: 194491 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 469bfb819f4aa1db0fb540d4c991ebbc0993e0a3, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.66	0.10	0.10
query5	0.52	0.49	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.57	0.51	0.49
query10	0.54	0.58	0.54
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	2.94	2.93	3.09
query15	0.89	0.82	0.81
query16	0.38	0.38	0.38
query17	1.04	1.01	1.02
query18	0.22	0.21	0.21
query19	1.89	1.76	1.99
query20	0.01	0.01	0.01
query21	15.37	0.60	0.61
query22	2.53	2.51	1.47
query23	17.07	0.86	0.75
query24	2.97	0.70	1.66
query25	0.22	0.08	0.05
query26	0.53	0.15	0.13
query27	0.04	0.04	0.04
query28	10.35	1.11	1.08
query29	12.61	3.30	3.26
query30	0.24	0.05	0.06
query31	2.91	0.38	0.38
query32	3.27	0.46	0.46
query33	3.02	3.00	3.05
query34	16.90	4.40	4.38
query35	4.39	4.45	4.40
query36	0.66	0.49	0.49
query37	0.08	0.05	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.64 s
Total hot run time: 31.94 s

@LiBinfeng-01
Copy link
Contributor Author

run buildall

@LiBinfeng-01
Copy link
Contributor Author

run buildall

@LiBinfeng-01
Copy link
Contributor Author

run p0

@LiBinfeng-01
Copy link
Contributor Author

run buildall

@LiBinfeng-01
Copy link
Contributor Author

run feut

@LiBinfeng-01
Copy link
Contributor Author

run external

@LiBinfeng-01
Copy link
Contributor Author

run cloud_p0

1 similar comment
@LiBinfeng-01
Copy link
Contributor Author

run cloud_p0

@LiBinfeng-01
Copy link
Contributor Author

run buildall

1 similar comment
@LiBinfeng-01
Copy link
Contributor Author

run buildall

@LiBinfeng-01
Copy link
Contributor Author

run compile

@LiBinfeng-01
Copy link
Contributor Author

run p0

@LiBinfeng-01
Copy link
Contributor Author

run cloud_p0

morrySnow
morrySnow previously approved these changes Oct 10, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 10, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Oct 12, 2024
@LiBinfeng-01
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 14, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 1813ec2 into apache:master Oct 14, 2024
qzsee pushed a commit to qzsee/incubator-doris that referenced this pull request Oct 16, 2024
…che#40958)

isLeading in cascade context means do we have leading in this scope
isLeadingDisableJoinReorder means does leading used
we need to copy it after analyze outerside of cte
LiBinfeng-01 added a commit to LiBinfeng-01/doris that referenced this pull request Oct 16, 2024
…che#40958)

isLeading in cascade context means do we have leading in this scope
isLeadingDisableJoinReorder means does leading used
we need to copy it after analyze outerside of cte
@morrySnow morrySnow added the p0_w label Oct 17, 2024
LiBinfeng-01 added a commit to LiBinfeng-01/doris that referenced this pull request Oct 17, 2024
…che#40958)

isLeading in cascade context means do we have leading in this scope
isLeadingDisableJoinReorder means does leading used
we need to copy it after analyze outerside of cte
englefly pushed a commit that referenced this pull request Oct 17, 2024
) (#41934)

cherry-pick: (#40958)

isLeading in cascade context means do we have leading in this scope
isLeadingDisableJoinReorder means does leading used we need to copy it
after analyze outerside of cte

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
morrySnow pushed a commit that referenced this pull request Oct 17, 2024
…#41869)

pick: #35925 #39715 #40167 #40958
Add feat of force use/nouse cbo rule hint and fix pr

introduce
when not using this hint, cbo rules like INFER_SET_OPERATOR_DISTINCT
would generate two plans and compare their cost
and nereids optimizer would decide which is better. But when we want to
control the behavior of cbo rules we could use this force cbo rule hint
usage example
explain shape plan
select /*+ USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) */
*
from t1
union
select * from t2;
the USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) hint would force rule
INFER_SET_OPERATOR_DISTINCT to be used
and generate plan like, which hashAgg below union is generated by this
rule:

-- !with_hint_union_distinct --
----hashAgg[GLOBAL]
--------hashAgg[LOCAL]
----------PhysicalUnion
--------------hashAgg[LOCAL]
----------------PhysicalOlapScan[t1]
--------------hashAgg[LOCAL]
----------------PhysicalOlapScan[t2]
Hint log:
Used: INFER_SET_OPERATOR_DISTINCT
UnUsed:
SyntaxError:
When we want to force disable this rule, we could use
explain shape plan select /*+
NO_USE_CBO_RULE(INFER_SET_OPERATOR_DISTINCT) */ * from t1 union select *
from t2;
which would generate plan with this rule:

-- !with_hint_no_union_distinct --
----hashAgg[GLOBAL]
--------hashAgg[LOCAL]
----------PhysicalUnion
--------------PhysicalOlapScan[t1]
--------------PhysicalOlapScan[t2]
Hint log:
Used: NO_INFER_SET_OPERATOR_DISTINCT
UnUsed:
SyntaxError:
change sessionvariable enableNereidsRules to varType.remove
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged p0_w reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants