Skip to content

[enhance](nereids) add rewrite rule SplitJoinForNullSkew #44357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
feiniaofeiafei wants to merge 6 commits into from
Closed

[enhance](nereids) add rewrite rule SplitJoinForNullSkew #44357

feiniaofeiafei wants to merge 6 commits into from

Conversation

@feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Nov 20, 2024 •
edited
Loading

What problem does this PR solve?

add transform rule:

 LogicalJoin(type:left join, hashConjuncts:t1.a=t2.a)
    +--Plan1(output:t1.a)
    +--Plan2(output:t2.a)
  ->
  LogicalUnion
    +--LogicalFilter(t1.a is null)
      +--Plan1
    +--LogicalJoin(type:left join, t1.a=t2.a)
      +--LogicalFilter(t1.a is not null)
        +--Plan1
      +--Plan2

only support left join with only 1 hash join conjuncts , and without other join conjuncts.
Since there is sometimes null value skew on the join key, which can lead to prolonged execution times, the join is split into two parts based on whether the join key is null. This can accelerate the query.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

doris-robot commented Nov 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 20, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the null_skew branch 2 times, most recently from 7c0ed99 to 82e7176 Compare November 20, 2024 12:10
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 20, 2024

run buildall

@feiniaofeiafei feiniaofeiafei force-pushed the null_skew branch 2 times, most recently from c96d557 to 1b40cc5 Compare November 20, 2024 12:58
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 20, 2024

run buidall

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 21, 2024

run buildall

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Nov 21, 2024

run p0

@doris-robot
Copy link

doris-robot commented Nov 21, 2024

TPC-H: Total hot run time: 39963 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b40cc50be96015ccb1eb034d44273a10393bea4, data reload: false

------ Round 1 ----------------------------------
q1	17954	7517	7285	7285
q2	2046	173	170	170
q3	10794	1123	1210	1123
q4	10487	747	760	747
q5	7592	2756	2692	2692
q6	247	146	148	146
q7	975	622	609	609
q8	9263	1875	1960	1875
q9	6639	6471	6402	6402
q10	6954	2293	2328	2293
q11	452	265	261	261
q12	428	222	214	214
q13	17796	3062	3037	3037
q14	250	213	216	213
q15	581	525	523	523
q16	647	595	601	595
q17	987	614	586	586
q18	7403	6656	6760	6656
q19	1346	1007	935	935
q20	491	187	177	177
q21	3970	3269	3115	3115
q22	380	322	309	309
Total cold run time: 107682 ms
Total hot run time: 39963 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7295	7334	7268	7268
q2	329	225	231	225
q3	2909	2869	2967	2869
q4	2127	1900	1908	1900
q5	5677	5682	5755	5682
q6	222	140	142	140
q7	2240	1831	1843	1831
q8	3391	3563	3569	3563
q9	8741	8913	8977	8913
q10	3620	3566	3585	3566
q11	601	523	507	507
q12	860	623	606	606
q13	10727	3303	3204	3204
q14	321	286	265	265
q15	590	526	541	526
q16	703	638	652	638
q17	1857	1646	1593	1593
q18	8484	7881	7796	7796
q19	2247	1642	1590	1590
q20	2112	1878	1948	1878
q21	5638	5455	5443	5443
q22	650	562	570	562
Total cold run time: 71341 ms
Total hot run time: 60565 ms

@doris-robot
Copy link

doris-robot commented Nov 21, 2024

TPC-DS: Total hot run time: 198243 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b40cc50be96015ccb1eb034d44273a10393bea4, data reload: false

query1	1253	952	910	910
query2	6239	2113	2119	2113
query3	10789	4028	4231	4028
query4	68158	29670	23621	23621
query5	5279	489	483	483
query6	445	184	200	184
query7	5871	313	295	295
query8	321	242	231	231
query9	9350	2683	2675	2675
query10	496	260	264	260
query11	17828	15780	15935	15780
query12	154	113	120	113
query13	1621	447	423	423
query14	11559	8163	8193	8163
query15	224	183	198	183
query16	7030	434	451	434
query17	1080	561	566	561
query18	1781	294	293	293
query19	195	173	156	156
query20	120	112	110	110
query21	205	102	110	102
query22	4625	4390	4273	4273
query23	34975	34188	34174	34174
query24	5319	2469	2511	2469
query25	522	388	391	388
query26	646	153	152	152
query27	1694	288	283	283
query28	4320	2421	2393	2393
query29	728	418	416	416
query30	205	147	147	147
query31	1028	813	831	813
query32	68	56	56	56
query33	408	280	298	280
query34	982	519	526	519
query35	859	765	725	725
query36	1145	962	981	962
query37	126	75	76	75
query38	4347	4289	4261	4261
query39	1487	1455	1437	1437
query40	207	101	100	100
query41	46	41	44	41
query42	112	101	100	100
query43	556	498	494	494
query44	1208	830	809	809
query45	188	167	167	167
query46	1197	735	709	709
query47	1910	1814	1828	1814
query48	420	324	326	324
query49	735	403	399	399
query50	872	389	418	389
query51	7353	7211	7113	7113
query52	100	87	91	87
query53	270	183	180	180
query54	544	402	406	402
query55	81	78	73	73
query56	265	243	247	243
query57	1307	1174	1165	1165
query58	231	220	224	220
query59	3171	3020	2925	2925
query60	292	258	258	258
query61	113	109	110	109
query62	824	683	701	683
query63	229	188	192	188
query64	1428	678	661	661
query65	3394	3298	3260	3260
query66	721	335	330	330
query67	15791	15614	15836	15614
query68	4259	594	566	566
query69	415	261	267	261
query70	1200	1154	1146	1146
query71	372	274	251	251
query72	6321	4125	4138	4125
query73	817	374	364	364
query74	10181	9028	9035	9028
query75	3545	2698	2699	2698
query76	1937	1132	1273	1132
query77	516	291	290	290
query78	10595	9572	9362	9362
query79	1626	598	602	598
query80	876	436	443	436
query81	531	241	238	238
query82	232	119	125	119
query83	266	158	162	158
query84	266	71	71	71
query85	970	309	330	309
query86	369	312	303	303
query87	4900	4573	4724	4573
query88	3859	2273	2236	2236
query89	439	302	303	302
query90	2169	196	192	192
query91	141	105	112	105
query92	62	49	55	49
query93	2256	542	549	542
query94	917	288	300	288
query95	342	252	254	252
query96	684	275	283	275
query97	2949	2659	2700	2659
query98	224	204	199	199
query99	1712	1323	1318	1318
Total cold run time: 322995 ms
Total hot run time: 198243 ms

@doris-robot
Copy link

doris-robot commented Nov 21, 2024

ClickBench: Total hot run time: 32.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b40cc50be96015ccb1eb034d44273a10393bea4, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.63	0.10	0.11
query5	0.43	0.41	0.41
query6	1.17	0.64	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.12
query13	0.61	0.59	0.60
query14	2.70	2.70	2.83
query15	0.91	0.83	0.81
query16	0.38	0.37	0.38
query17	1.07	1.00	1.04
query18	0.20	0.20	0.20
query19	1.99	1.89	2.06
query20	0.01	0.01	0.01
query21	15.36	0.56	0.59
query22	2.37	3.09	1.72
query23	17.25	0.90	0.77
query24	3.45	0.65	1.65
query25	0.28	0.05	0.10
query26	0.48	0.13	0.14
query27	0.05	0.04	0.04
query28	10.45	1.09	1.08
query29	12.56	3.23	3.22
query30	0.25	0.06	0.06
query31	2.87	0.37	0.36
query32	3.30	0.46	0.48
query33	2.97	3.12	3.03
query34	17.06	4.52	4.48
query35	4.53	4.50	4.50
query36	0.65	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.03
query43	0.04	0.03	0.02
Total cold run time: 107.29 s
Total hot run time: 32.02 s

morrySnow
morrySnow reviewed Nov 29, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/JoinSplitForNullSkew.java
public Rule build() {
return logicalJoin(any(), any())
.when(join -> join.getJoinType().isLeftJoin())
.when(join -> join.getHashJoinConjuncts().size() == 1)
Copy link
Contributor

@morrySnow morrySnow Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should check mark join conjuncts

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/JoinSplitForNullSkew.java Outdated
.when(join -> join.getJoinType().isLeftJoin())
.when(join -> join.getHashJoinConjuncts().size() == 1)
.thenApply(ctx -> {
Set<Integer> enableNereidsRules = ctx.cascadesContext.getConnectContext()
Copy link
Contributor

@morrySnow morrySnow Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need add this? enableRules removed in this PR: #44769

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/JoinSplitForNullSkew.java Outdated
@Override
public Rule build() {
return logicalJoin(any(), any())
.when(join -> join.getJoinType().isLeftJoin())
Copy link
Contributor

@morrySnow morrySnow Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not process right outer join?

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/JoinSplitForNullSkew.java Outdated
Plan deepCopyJoin = LogicalPlanDeepCopier.INSTANCE.deepCopy(newJoin, new DeepCopierContext());

// avoid duplicate application of rules
if (left instanceof LogicalFilter) {
Copy link
Contributor

@morrySnow morrySnow Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not just check filter's conjuncts contain is not null?

Copy link
Contributor Author

@feiniaofeiafei feiniaofeiafei Feb 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are right, I make it too complicated.

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/JoinSplitForNullSkew.java
.toRule(RuleType.JOIN_SPLIT_FOR_NULL_SKEW);
}

private Plan splitJoin(LogicalJoin<Plan, Plan> join) {
Copy link
Contributor

@morrySnow morrySnow Nov 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should not rewrite if leftExpr is not null already

Copy link
Contributor Author

@feiniaofeiafei feiniaofeiafei Feb 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@feiniaofeiafei feiniaofeiafei force-pushed the null_skew branch 2 times, most recently from ae357a1 to ff5921c Compare December 2, 2024 13:15
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Dec 2, 2024

run buildall

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Dec 3, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 3, 2024

TPC-H: Total hot run time: 39859 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94c526af1767abdd5a77707bcc5925660a0f7dc1, data reload: false

------ Round 1 ----------------------------------
q1	17652	7388	7265	7265
q2	2040	173	167	167
q3	10615	1079	1176	1079
q4	10593	819	779	779
q5	7595	2735	2712	2712
q6	236	147	145	145
q7	988	617	600	600
q8	9248	1865	1930	1865
q9	6609	6455	6476	6455
q10	7016	2291	2276	2276
q11	469	258	260	258
q12	417	221	217	217
q13	17791	3052	3095	3052
q14	243	219	212	212
q15	579	547	512	512
q16	653	589	598	589
q17	989	504	521	504
q18	7213	6629	6655	6629
q19	1329	1064	976	976
q20	470	179	174	174
q21	3984	3231	3081	3081
q22	367	317	312	312
Total cold run time: 107096 ms
Total hot run time: 39859 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7216	7218	7225	7218
q2	328	226	228	226
q3	2925	2799	2957	2799
q4	2099	1861	1859	1859
q5	5657	5696	5631	5631
q6	230	145	146	145
q7	2224	1854	1819	1819
q8	3392	3540	3510	3510
q9	8921	9030	9006	9006
q10	3602	3557	3547	3547
q11	597	513	509	509
q12	841	614	596	596
q13	13610	3247	3222	3222
q14	297	273	274	273
q15	564	523	518	518
q16	677	669	667	667
q17	1858	1622	1597	1597
q18	8278	7852	7465	7465
q19	1706	1468	1528	1468
q20	2108	1934	1841	1841
q21	5539	5446	5459	5446
q22	641	560	576	560
Total cold run time: 73310 ms
Total hot run time: 59922 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2024

TPC-DS: Total hot run time: 198205 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 94c526af1767abdd5a77707bcc5925660a0f7dc1, data reload: false

query1	1235	957	959	957
query2	6211	2110	2057	2057
query3	10918	4448	4433	4433
query4	66188	28582	23960	23960
query5	5007	463	436	436
query6	423	184	177	177
query7	5502	302	302	302
query8	330	241	236	236
query9	8548	2707	2684	2684
query10	463	248	250	248
query11	17067	15254	15853	15254
query12	158	99	114	99
query13	1506	433	450	433
query14	10971	7640	7127	7127
query15	212	194	205	194
query16	7285	457	487	457
query17	1188	576	582	576
query18	1891	294	309	294
query19	201	152	175	152
query20	116	109	106	106
query21	210	101	104	101
query22	4598	4642	4434	4434
query23	34885	34694	34654	34654
query24	5440	2528	2473	2473
query25	497	392	382	382
query26	644	155	150	150
query27	1991	281	290	281
query28	4746	2492	2452	2452
query29	672	441	421	421
query30	222	161	156	156
query31	1017	837	863	837
query32	68	56	51	51
query33	409	282	310	282
query34	949	532	507	507
query35	897	769	746	746
query36	1091	953	962	953
query37	119	79	71	71
query38	4445	4416	4478	4416
query39	1538	1476	1444	1444
query40	202	98	98	98
query41	43	43	44	43
query42	113	98	98	98
query43	534	506	497	497
query44	1203	828	831	828
query45	197	174	171	171
query46	1181	727	724	724
query47	2045	1916	1928	1916
query48	432	313	332	313
query49	742	402	403	402
query50	847	391	387	387
query51	7450	7098	7019	7019
query52	96	87	88	87
query53	256	175	183	175
query54	504	388	389	388
query55	74	73	78	73
query56	236	239	245	239
query57	1252	1116	1099	1099
query58	216	222	203	203
query59	3132	3026	3005	3005
query60	269	243	245	243
query61	106	101	106	101
query62	795	662	662	662
query63	205	183	193	183
query64	1369	653	624	624
query65	3272	3220	3199	3199
query66	698	295	315	295
query67	16030	15480	15547	15480
query68	4063	569	548	548
query69	427	247	244	244
query70	1200	1116	1115	1115
query71	358	247	246	246
query72	7357	4855	4996	4855
query73	761	363	359	359
query74	10199	9005	8987	8987
query75	3393	2659	2673	2659
query76	1941	1138	1037	1037
query77	496	257	293	257
query78	10478	9510	9456	9456
query79	1316	598	592	592
query80	835	430	421	421
query81	485	240	224	224
query82	1318	118	121	118
query83	250	143	136	136
query84	279	70	81	70
query85	889	299	285	285
query86	344	312	294	294
query87	4713	4522	4497	4497
query88	3664	2197	2143	2143
query89	424	292	287	287
query90	2017	183	180	180
query91	145	101	103	101
query92	66	49	50	49
query93	1688	541	546	541
query94	767	278	285	278
query95	340	247	249	247
query96	601	271	285	271
query97	2888	2668	2671	2668
query98	223	197	192	192
query99	1794	1329	1303	1303
Total cold run time: 319076 ms
Total hot run time: 198205 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2024

ClickBench: Total hot run time: 32.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 94c526af1767abdd5a77707bcc5925660a0f7dc1, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.04
query3	0.24	0.08	0.07
query4	1.61	0.10	0.10
query5	0.43	0.39	0.41
query6	1.14	0.66	0.66
query7	0.01	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.53	0.51
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.13	0.13	0.11
query13	0.61	0.61	0.61
query14	2.71	2.88	2.84
query15	0.92	0.83	0.82
query16	0.39	0.38	0.37
query17	1.06	1.06	1.05
query18	0.21	0.21	0.21
query19	1.85	1.89	2.03
query20	0.01	0.00	0.01
query21	15.36	0.59	0.58
query22	2.60	1.86	1.73
query23	16.98	0.87	0.90
query24	2.91	0.54	0.90
query25	0.15	0.11	0.09
query26	0.48	0.14	0.15
query27	0.05	0.04	0.04
query28	11.47	1.09	1.06
query29	12.57	3.25	3.23
query30	0.25	0.06	0.08
query31	2.84	0.39	0.37
query32	3.27	0.46	0.46
query33	3.00	3.00	3.03
query34	17.02	4.52	4.46
query35	4.52	4.44	4.48
query36	0.68	0.50	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.36 s
Total hot run time: 32.28 s

@feiniaofeiafei feiniaofeiafei force-pushed the null_skew branch 2 times, most recently from b688cb2 to 9bcb488 Compare February 14, 2025 12:00
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Feb 14, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 14, 2025

TPC-H: Total hot run time: 31583 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9bcb48891ffc313583961fe56459a3a2ae829a72, data reload: false

------ Round 1 ----------------------------------
q1	17602	5241	5107	5107
q2	2043	316	177	177
q3	10375	1313	737	737
q4	10219	1013	553	553
q5	7521	2429	2385	2385
q6	189	184	142	142
q7	919	749	629	629
q8	9304	1289	1058	1058
q9	4841	4623	4508	4508
q10	6855	2328	1884	1884
q11	463	281	261	261
q12	357	357	220	220
q13	17761	3675	3049	3049
q14	242	241	215	215
q15	504	469	466	466
q16	642	626	593	593
q17	583	870	359	359
q18	6594	6250	6163	6163
q19	1081	942	554	554
q20	319	326	206	206
q21	2909	2276	2007	2007
q22	386	334	310	310
Total cold run time: 101709 ms
Total hot run time: 31583 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5203	5183	5189	5183
q2	241	327	231	231
q3	2127	2656	2305	2305
q4	1414	1837	1454	1454
q5	4237	4172	4192	4172
q6	212	165	126	126
q7	1857	1813	1697	1697
q8	2636	2674	2569	2569
q9	7312	7056	7092	7056
q10	3020	3208	2802	2802
q11	579	507	485	485
q12	660	768	624	624
q13	3518	3916	3376	3376
q14	272	289	280	280
q15	503	464	468	464
q16	643	667	619	619
q17	1184	1575	1341	1341
q18	7540	7355	7449	7355
q19	848	869	847	847
q20	1976	2046	1887	1887
q21	5406	4825	4813	4813
q22	651	627	573	573
Total cold run time: 52039 ms
Total hot run time: 50259 ms

@doris-robot
Copy link

doris-robot commented Feb 14, 2025

TPC-DS: Total hot run time: 185766 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9bcb48891ffc313583961fe56459a3a2ae829a72, data reload: false

query1	973	367	398	367
query2	6533	1842	1842	1842
query3	6806	214	212	212
query4	26864	23669	23385	23385
query5	4379	706	485	485
query6	297	206	190	190
query7	4606	523	291	291
query8	288	233	218	218
query9	8624	2497	2515	2497
query10	446	329	277	277
query11	15176	15187	16329	15187
query12	163	111	105	105
query13	1665	528	399	399
query14	10127	6713	6898	6713
query15	195	197	188	188
query16	7785	670	467	467
query17	1234	727	575	575
query18	1989	418	305	305
query19	207	187	156	156
query20	123	119	119	119
query21	214	138	103	103
query22	4101	4533	4558	4533
query23	34233	33207	33032	33032
query24	7648	2355	2399	2355
query25	534	458	377	377
query26	1224	283	151	151
query27	2091	513	323	323
query28	3896	2383	2377	2377
query29	765	538	413	413
query30	229	182	150	150
query31	933	861	766	766
query32	69	62	64	62
query33	571	348	308	308
query34	797	902	487	487
query35	788	832	723	723
query36	976	1021	905	905
query37	124	97	78	78
query38	4138	4110	4102	4102
query39	1461	1425	1386	1386
query40	209	115	105	105
query41	51	50	51	50
query42	128	108	105	105
query43	499	508	478	478
query44	1309	778	768	768
query45	182	166	164	164
query46	911	1088	642	642
query47	1747	1790	1734	1734
query48	382	421	301	301
query49	752	495	427	427
query50	767	763	423	423
query51	4141	4204	4194	4194
query52	102	107	111	107
query53	229	270	184	184
query54	506	478	403	403
query55	87	86	78	78
query56	249	276	258	258
query57	1128	1173	1068	1068
query58	243	239	232	232
query59	2526	2605	2625	2605
query60	286	275	292	275
query61	126	125	125	125
query62	860	704	665	665
query63	231	186	189	186
query64	4364	988	659	659
query65	3189	3126	3189	3126
query66	1059	410	309	309
query67	15808	15525	15448	15448
query68	7282	776	497	497
query69	486	299	264	264
query70	1232	1142	1128	1128
query71	418	306	264	264
query72	6688	4542	4686	4542
query73	753	773	346	346
query74	8908	8871	8912	8871
query75	3203	3158	2690	2690
query76	3291	1233	749	749
query77	496	385	279	279
query78	9994	10211	9295	9295
query79	1987	884	579	579
query80	736	553	504	504
query81	532	272	233	233
query82	407	127	98	98
query83	176	169	153	153
query84	241	105	81	81
query85	755	345	310	310
query86	395	282	265	265
query87	4305	4637	4407	4407
query88	2879	2183	2182	2182
query89	392	319	278	278
query90	1769	194	192	192
query91	131	143	109	109
query92	70	60	58	58
query93	1440	1057	586	586
query94	654	436	314	314
query95	350	265	257	257
query96	491	593	267	267
query97	2769	2828	2766	2766
query98	221	219	198	198
query99	1284	1423	1276	1276
Total cold run time: 270078 ms
Total hot run time: 185766 ms

@doris-robot
Copy link

doris-robot commented Feb 14, 2025

ClickBench: Total hot run time: 30.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9bcb48891ffc313583961fe56459a3a2ae829a72, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.62	0.10	0.10
query5	0.43	0.43	0.40
query6	1.15	0.66	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.59	0.53	0.52
query10	0.58	0.57	0.59
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.70	2.71	2.69
query15	0.92	0.85	0.87
query16	0.36	0.39	0.38
query17	1.00	1.01	1.02
query18	0.21	0.19	0.19
query19	1.93	1.86	2.02
query20	0.02	0.01	0.01
query21	15.36	0.91	0.56
query22	0.75	1.17	0.71
query23	14.88	1.39	0.66
query24	9.00	1.35	0.60
query25	0.58	0.16	0.07
query26	0.69	0.19	0.14
query27	0.05	0.05	0.04
query28	9.44	0.85	0.43
query29	12.54	3.91	3.27
query30	0.25	0.09	0.07
query31	2.82	0.58	0.38
query32	3.23	0.56	0.46
query33	3.00	3.04	3.00
query34	15.82	5.17	4.52
query35	4.56	4.51	4.54
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.16	0.14	0.14
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.97 s
Total hot run time: 30.46 s

@feiniaofeiafei feiniaofeiafei force-pushed the null_skew branch 2 times, most recently from 861da34 to 6bb80b0 Compare February 17, 2025 07:21
@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Feb 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 17, 2025

TPC-H: Total hot run time: 31676 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6bb80b0ff7efee50dd4f5d25133b0124a0bb08ce, data reload: false

------ Round 1 ----------------------------------
q1	17588	5230	5116	5116
q2	2066	297	165	165
q3	10442	1252	742	742
q4	10243	1018	560	560
q5	7975	2410	2330	2330
q6	197	167	133	133
q7	901	751	608	608
q8	9297	1275	1069	1069
q9	5010	4671	4661	4661
q10	6880	2316	1884	1884
q11	495	276	261	261
q12	361	356	218	218
q13	17767	3721	3052	3052
q14	230	239	219	219
q15	522	479	464	464
q16	618	619	587	587
q17	588	882	338	338
q18	6597	6374	6186	6186
q19	1296	959	574	574
q20	310	330	193	193
q21	2811	2332	2003	2003
q22	362	336	313	313
Total cold run time: 102556 ms
Total hot run time: 31676 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5217	5186	5215	5186
q2	244	362	230	230
q3	2177	2713	2280	2280
q4	1484	1908	1413	1413
q5	4262	4188	4192	4188
q6	208	172	127	127
q7	1861	1798	1754	1754
q8	2645	2679	2581	2581
q9	7198	7188	7113	7113
q10	3040	3237	2788	2788
q11	590	536	490	490
q12	702	791	638	638
q13	3423	3919	3291	3291
q14	281	293	287	287
q15	506	469	456	456
q16	652	679	630	630
q17	1164	1584	1418	1418
q18	7519	7312	7301	7301
q19	833	869	847	847
q20	2084	2056	1865	1865
q21	5514	5110	4776	4776
q22	631	638	549	549
Total cold run time: 52235 ms
Total hot run time: 50208 ms

@doris-robot
Copy link

doris-robot commented Feb 17, 2025

TPC-DS: Total hot run time: 190308 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6bb80b0ff7efee50dd4f5d25133b0124a0bb08ce, data reload: false

query1	1336	967	974	967
query2	6197	1834	1814	1814
query3	11101	4598	4637	4598
query4	54458	24638	22910	22910
query5	5365	569	468	468
query6	430	204	184	184
query7	5410	516	293	293
query8	341	248	237	237
query9	7580	2503	2505	2503
query10	407	319	260	260
query11	15417	15070	14851	14851
query12	162	109	108	108
query13	1342	520	395	395
query14	10583	6456	6889	6456
query15	199	192	187	187
query16	6920	681	530	530
query17	1114	741	593	593
query18	1569	415	317	317
query19	224	215	166	166
query20	131	125	121	121
query21	217	127	114	114
query22	4314	4486	4380	4380
query23	34006	33307	33294	33294
query24	5952	2465	2422	2422
query25	461	469	396	396
query26	688	279	152	152
query27	1740	493	341	341
query28	2846	2440	2423	2423
query29	585	558	423	423
query30	211	195	160	160
query31	927	877	868	868
query32	78	60	60	60
query33	434	355	306	306
query34	760	860	487	487
query35	791	829	758	758
query36	954	1006	929	929
query37	124	109	77	77
query38	4313	4347	4151	4151
query39	1487	1412	1463	1412
query40	207	123	103	103
query41	51	50	50	50
query42	124	103	110	103
query43	507	509	468	468
query44	1329	792	798	792
query45	180	175	167	167
query46	863	1046	672	672
query47	1862	1856	1797	1797
query48	400	417	313	313
query49	721	515	413	413
query50	722	761	439	439
query51	4377	4265	4228	4228
query52	108	106	99	99
query53	251	262	187	187
query54	488	496	410	410
query55	83	86	81	81
query56	276	268	257	257
query57	1124	1179	1112	1112
query58	254	244	241	241
query59	2707	2922	2597	2597
query60	288	278	254	254
query61	119	120	128	120
query62	752	763	680	680
query63	246	202	193	193
query64	1474	1065	675	675
query65	3259	3158	3139	3139
query66	716	386	299	299
query67	16037	15907	15441	15441
query68	5145	768	501	501
query69	523	303	263	263
query70	1189	1084	1134	1084
query71	425	297	264	264
query72	7046	4649	4711	4649
query73	932	741	336	336
query74	9049	9198	8833	8833
query75	3196	3136	2706	2706
query76	4100	1158	751	751
query77	522	358	281	281
query78	9977	9874	9315	9315
query79	2727	812	585	585
query80	714	518	448	448
query81	511	270	234	234
query82	674	129	93	93
query83	181	173	148	148
query84	283	89	72	72
query85	781	345	305	305
query86	386	312	288	288
query87	4492	4573	4383	4383
query88	3568	2158	2161	2158
query89	411	303	283	283
query90	1726	189	188	188
query91	141	142	111	111
query92	75	55	54	54
query93	2248	1022	566	566
query94	706	417	265	265
query95	351	268	254	254
query96	494	566	268	268
query97	2817	2887	2724	2724
query98	223	212	213	212
query99	1382	1559	1245	1245
Total cold run time: 297694 ms
Total hot run time: 190308 ms

@doris-robot
Copy link

doris-robot commented Feb 17, 2025

ClickBench: Total hot run time: 30.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6bb80b0ff7efee50dd4f5d25133b0124a0bb08ce, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.08	0.06
query4	1.62	0.10	0.10
query5	0.41	0.42	0.40
query6	1.15	0.66	0.66
query7	0.02	0.02	0.02
query8	0.03	0.03	0.03
query9	0.58	0.52	0.52
query10	0.58	0.59	0.57
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.62	0.61	0.60
query14	2.69	2.74	2.71
query15	0.92	0.86	0.86
query16	0.38	0.37	0.38
query17	1.07	1.02	1.03
query18	0.21	0.19	0.19
query19	1.91	1.77	1.99
query20	0.02	0.01	0.01
query21	15.36	0.90	0.54
query22	0.73	1.19	0.65
query23	14.97	1.39	0.62
query24	11.81	1.06	0.39
query25	0.32	0.13	0.08
query26	0.90	0.19	0.14
query27	0.05	0.04	0.04
query28	6.07	0.81	0.44
query29	12.53	4.01	3.29
query30	0.25	0.09	0.06
query31	2.84	0.59	0.39
query32	3.22	0.54	0.46
query33	3.04	3.01	3.02
query34	15.63	5.17	4.50
query35	4.56	4.59	4.54
query36	0.67	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.04	0.03
Total cold run time: 106.25 s
Total hot run time: 30.13 s

@feiniaofeiafei
Copy link
Contributor Author

feiniaofeiafei commented Feb 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 17, 2025

TPC-H: Total hot run time: 31933 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2b183026be8558156e98920eb9e053b43df96fb6, data reload: false

------ Round 1 ----------------------------------
q1	17680	5367	5156	5156
q2	2055	296	179	179
q3	10403	1340	726	726
q4	10227	1012	550	550
q5	7533	2419	2372	2372
q6	188	166	139	139
q7	931	750	600	600
q8	9303	1283	1168	1168
q9	5192	4706	4891	4706
q10	6846	2317	1908	1908
q11	480	280	256	256
q12	362	360	216	216
q13	17757	3705	3074	3074
q14	233	224	203	203
q15	518	463	455	455
q16	623	594	578	578
q17	595	867	357	357
q18	6778	6233	6302	6233
q19	1211	943	564	564
q20	308	338	200	200
q21	3031	2254	1987	1987
q22	372	344	306	306
Total cold run time: 102626 ms
Total hot run time: 31933 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5239	5313	5272	5272
q2	234	327	232	232
q3	2187	2688	2314	2314
q4	1480	1856	1374	1374
q5	4262	4178	4170	4170
q6	210	161	124	124
q7	1868	1833	1773	1773
q8	2635	2655	2600	2600
q9	7200	7053	7014	7014
q10	3007	3217	2754	2754
q11	584	511	490	490
q12	686	760	591	591
q13	3650	3928	3287	3287
q14	275	286	290	286
q15	504	460	472	460
q16	649	670	624	624
q17	1155	1568	1376	1376
q18	7538	7231	7227	7227
q19	852	791	842	791
q20	2028	2032	1901	1901
q21	5415	5072	4868	4868
q22	615	623	552	552
Total cold run time: 52273 ms
Total hot run time: 50080 ms

@doris-robot
Copy link

doris-robot commented Feb 17, 2025

TPC-DS: Total hot run time: 184228 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2b183026be8558156e98920eb9e053b43df96fb6, data reload: false

query1	965	376	383	376
query2	6523	1849	1837	1837
query3	6799	211	211	211
query4	26053	23724	22901	22901
query5	4322	684	509	509
query6	309	204	193	193
query7	4616	521	303	303
query8	296	250	247	247
query9	8605	2489	2496	2489
query10	467	310	252	252
query11	15413	15161	14937	14937
query12	163	104	103	103
query13	1646	523	372	372
query14	8992	6915	6592	6592
query15	206	197	175	175
query16	7238	609	472	472
query17	1180	725	561	561
query18	1944	394	310	310
query19	205	197	163	163
query20	121	122	123	122
query21	209	123	104	104
query22	4165	4095	4337	4095
query23	34562	33391	32930	32930
query24	7761	2422	2467	2422
query25	555	476	412	412
query26	1224	280	165	165
query27	2131	505	327	327
query28	3916	2393	2350	2350
query29	788	621	408	408
query30	229	183	149	149
query31	923	874	785	785
query32	79	61	62	61
query33	569	349	299	299
query34	783	858	492	492
query35	817	805	736	736
query36	953	988	873	873
query37	120	95	72	72
query38	4092	4249	4189	4189
query39	1442	1442	1403	1403
query40	214	113	114	113
query41	53	51	52	51
query42	127	99	102	99
query43	501	508	483	483
query44	1302	792	772	772
query45	178	172	162	162
query46	887	1033	659	659
query47	1798	1797	1728	1728
query48	366	411	293	293
query49	785	527	418	418
query50	724	731	423	423
query51	4152	4139	4062	4062
query52	104	108	95	95
query53	226	261	195	195
query54	485	495	401	401
query55	82	78	79	78
query56	326	307	249	249
query57	1123	1149	1058	1058
query58	250	233	244	233
query59	2657	2674	2363	2363
query60	280	277	285	277
query61	118	134	117	117
query62	810	731	665	665
query63	230	193	189	189
query64	4368	983	656	656
query65	3224	3149	3146	3146
query66	1167	401	290	290
query67	15738	15777	15474	15474
query68	8051	777	486	486
query69	486	298	262	262
query70	1180	1150	1093	1093
query71	409	298	253	253
query72	6766	4609	4720	4609
query73	757	753	377	377
query74	9071	9254	8901	8901
query75	3167	3191	2715	2715
query76	3270	1176	776	776
query77	542	373	307	307
query78	10087	9889	9346	9346
query79	3305	816	588	588
query80	942	542	464	464
query81	562	270	249	249
query82	1260	125	99	99
query83	215	176	149	149
query84	239	93	78	78
query85	759	353	304	304
query86	415	283	283	283
query87	4436	4560	4342	4342
query88	4068	2179	2168	2168
query89	408	316	293	293
query90	1816	193	193	193
query91	137	144	112	112
query92	77	58	61	58
query93	2736	1007	572	572
query94	798	420	286	286
query95	352	267	260	260
query96	486	583	272	272
query97	2795	2836	2742	2742
query98	233	208	200	200
query99	1314	1413	1245	1245
Total cold run time: 274795 ms
Total hot run time: 184228 ms

@doris-robot
Copy link

doris-robot commented Feb 17, 2025

ClickBench: Total hot run time: 30.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2b183026be8558156e98920eb9e053b43df96fb6, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.43	0.41
query6	1.18	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.59	0.51	0.54
query10	0.58	0.58	0.57
query11	0.16	0.11	0.10
query12	0.15	0.12	0.11
query13	0.62	0.60	0.61
query14	2.68	2.72	2.68
query15	0.93	0.85	0.86
query16	0.39	0.38	0.38
query17	1.05	1.02	1.04
query18	0.21	0.19	0.19
query19	1.89	1.98	1.82
query20	0.02	0.02	0.01
query21	15.36	0.91	0.55
query22	0.78	1.08	0.77
query23	14.93	1.41	0.65
query24	12.27	1.12	0.41
query25	0.36	0.27	0.07
query26	0.88	0.18	0.14
query27	0.05	0.04	0.05
query28	5.97	0.77	0.44
query29	12.56	4.00	3.28
query30	0.25	0.08	0.06
query31	2.81	0.59	0.38
query32	3.23	0.55	0.46
query33	2.98	2.99	3.00
query34	15.66	5.11	4.50
query35	4.56	4.53	4.50
query36	0.66	0.49	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.66 s
Total hot run time: 30.22 s

@morrySnow morrySnow marked this pull request as draft February 19, 2025 08:54
@morrySnow morrySnow marked this pull request as ready for review February 19, 2025 08:54
@feiniaofeiafei feiniaofeiafei marked this pull request as draft March 17, 2025 02:59
@feiniaofeiafei
initial commit
defb2d9 
rule can execute

Avoid duplicate application of rules

add comment

add test

remove unrelated code

add test

modify code by comments

fix compile
@feiniaofeiafei
add test
1ed5934
@feiniaofeiafei
fix feut
56f4fae
@feiniaofeiafei
add test
7111d92
@feiniaofeiafei
simplify avoid duplicate application of rules
9daeb11
@feiniaofeiafei
add agg skew rewrite
2068444
@morrySnow morrySnow closed this Jun 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@feiniaofeiafei @doris-robot @morrySnow