Skip to content

Conversation

@yujun777
Copy link
Contributor

@yujun777 yujun777 commented Jan 24, 2025

What problem does this PR solve?

Add an expression rewrite rule, it will simplify like:

x and not x =>   falseOrNull(x)
x or not x    =>  trueOrNull(x)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777
Copy link
Contributor Author

run buildall

@yujun777
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32353 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 62162e4e6974491a82afa0a1aac921204189b822, data reload: false

------ Round 1 ----------------------------------
q1	17582	5483	5369	5369
q2	2049	309	166	166
q3	10452	1230	772	772
q4	10525	958	551	551
q5	9394	2412	2165	2165
q6	203	162	133	133
q7	903	757	604	604
q8	9235	1342	1176	1176
q9	5235	4932	4893	4893
q10	6854	2353	1916	1916
q11	475	281	259	259
q12	346	360	225	225
q13	17764	3712	3105	3105
q14	229	241	201	201
q15	531	464	477	464
q16	650	615	582	582
q17	566	866	338	338
q18	6913	6351	6616	6351
q19	1635	952	542	542
q20	310	327	196	196
q21	2780	2246	2033	2033
q22	364	328	312	312
Total cold run time: 104995 ms
Total hot run time: 32353 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5495	5548	6150	5548
q2	246	333	238	238
q3	2227	2679	2318	2318
q4	1442	1822	1388	1388
q5	4332	4789	4783	4783
q6	177	171	131	131
q7	2040	1994	1844	1844
q8	2660	2882	2750	2750
q9	7343	7272	7356	7272
q10	3038	3328	2747	2747
q11	566	503	492	492
q12	651	765	625	625
q13	3613	3967	3316	3316
q14	300	286	284	284
q15	518	480	467	467
q16	648	690	671	671
q17	1208	1752	1286	1286
q18	7714	7635	7281	7281
q19	805	1198	1062	1062
q20	2023	2072	1934	1934
q21	5846	5144	4992	4992
q22	611	604	574	574
Total cold run time: 53503 ms
Total hot run time: 52003 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192929 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 62162e4e6974491a82afa0a1aac921204189b822, data reload: false

query1	1326	956	955	955
query2	6373	1988	2065	1988
query3	11122	4582	4582	4582
query4	32298	23395	23153	23153
query5	3644	625	466	466
query6	303	211	193	193
query7	3985	488	321	321
query8	295	256	232	232
query9	9460	2580	2572	2572
query10	448	320	261	261
query11	17828	15438	15085	15085
query12	164	110	106	106
query13	1570	532	395	395
query14	9612	7536	7020	7020
query15	261	210	186	186
query16	7998	644	479	479
query17	1597	744	563	563
query18	2165	414	317	317
query19	190	191	159	159
query20	124	121	110	110
query21	213	126	104	104
query22	4721	4467	4466	4466
query23	34633	34340	33696	33696
query24	6470	2340	2311	2311
query25	477	453	410	410
query26	721	250	159	159
query27	1987	481	346	346
query28	5763	2478	2420	2420
query29	571	540	459	459
query30	214	189	167	167
query31	999	884	838	838
query32	94	66	59	59
query33	503	359	303	303
query34	758	890	517	517
query35	843	839	761	761
query36	1028	1059	944	944
query37	131	103	81	81
query38	4405	4258	4303	4258
query39	1491	1483	1463	1463
query40	203	123	109	109
query41	59	60	55	55
query42	131	112	112	112
query43	534	525	509	509
query44	1346	850	867	850
query45	184	179	166	166
query46	892	1050	658	658
query47	1908	1919	1855	1855
query48	394	415	323	323
query49	729	508	398	398
query50	664	682	402	402
query51	4315	4305	4258	4258
query52	109	107	99	99
query53	237	262	200	200
query54	499	518	427	427
query55	85	77	80	77
query56	262	250	264	250
query57	1199	1201	1129	1129
query58	268	231	236	231
query59	3179	3240	3170	3170
query60	276	280	276	276
query61	117	129	123	123
query62	779	733	693	693
query63	242	196	196	196
query64	3719	1042	665	665
query65	3425	3275	3369	3275
query66	793	412	316	316
query67	16289	15866	15688	15688
query68	1979	784	558	558
query69	464	318	271	271
query70	1300	1110	1102	1102
query71	348	284	259	259
query72	5336	3790	3880	3790
query73	660	749	364	364
query74	9947	8931	8792	8792
query75	3219	3138	2656	2656
query76	2865	1198	794	794
query77	506	358	278	278
query78	10068	10246	9496	9496
query79	1005	881	587	587
query80	1203	537	465	465
query81	527	280	236	236
query82	655	198	122	122
query83	256	167	152	152
query84	236	96	79	79
query85	761	343	304	304
query86	397	321	308	308
query87	4468	4509	4404	4404
query88	3713	2206	2278	2206
query89	412	336	306	306
query90	1577	193	193	193
query91	130	133	104	104
query92	59	55	54	54
query93	975	871	543	543
query94	636	416	298	298
query95	334	264	264	264
query96	492	614	292	292
query97	2805	2915	2776	2776
query98	226	229	195	195
query99	1291	1426	1266	1266
Total cold run time: 278472 ms
Total hot run time: 192929 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 62162e4e6974491a82afa0a1aac921204189b822, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.11	0.11
query5	0.43	0.42	0.41
query6	1.17	0.65	0.65
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.51	0.50
query10	0.57	0.59	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.63	0.60	0.59
query14	2.74	2.83	2.75
query15	0.90	0.83	0.84
query16	0.38	0.38	0.38
query17	1.01	1.08	1.07
query18	0.23	0.21	0.22
query19	1.93	1.84	2.01
query20	0.02	0.01	0.02
query21	15.37	0.97	0.60
query22	0.76	0.76	0.63
query23	15.32	1.41	0.57
query24	3.05	0.65	1.36
query25	0.23	0.07	0.21
query26	0.39	0.15	0.14
query27	0.04	0.06	0.05
query28	13.71	1.06	0.45
query29	12.56	3.88	3.24
query30	0.25	0.10	0.07
query31	2.82	0.60	0.38
query32	3.23	0.55	0.46
query33	3.04	3.17	3.01
query34	16.61	5.23	4.57
query35	4.61	4.56	4.57
query36	0.63	0.50	0.48
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.05 s
Total hot run time: 30.55 s

@yujun777
Copy link
Contributor Author

run p0

2 similar comments
@yujun777
Copy link
Contributor Author

run p0

@yujun777
Copy link
Contributor Author

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 14, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@starocean999 starocean999 merged commit 46f12ee into apache:master Feb 14, 2025
25 of 26 checks passed
starocean999 added a commit that referenced this pull request Feb 19, 2025
…ly to join (#47985)

example sql:
`SELECT *
FROM 
    t1
WHERE 
    t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);`

the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be
replaced as `FALSE` by SimplifyConflictCompound rule. So we need check
if correlated filter exists before converting apply to join

Related PR: (#47385)
github-actions bot pushed a commit that referenced this pull request Feb 19, 2025
…ly to join (#47985)

example sql:
`SELECT *
FROM 
    t1
WHERE 
    t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);`

the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be
replaced as `FALSE` by SimplifyConflictCompound rule. So we need check
if correlated filter exists before converting apply to join

Related PR: (#47385)
github-actions bot pushed a commit that referenced this pull request Feb 19, 2025
…ly to join (#47985)

example sql:
`SELECT *
FROM 
    t1
WHERE 
    t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);`

the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be
replaced as `FALSE` by SimplifyConflictCompound rule. So we need check
if correlated filter exists before converting apply to join

Related PR: (#47385)
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
Add an expression rewrite rule,  it will simplify like:

```
x and not x =>   falseOrNull(x)
x or not x    =>  trueOrNull(x)
```
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
…ly to join (apache#47985)

example sql:
`SELECT *
FROM 
    t1
WHERE 
    t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);`

the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be
replaced as `FALSE` by SimplifyConflictCompound rule. So we need check
if correlated filter exists before converting apply to join

Related PR: (apache#47385)
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
Add an expression rewrite rule,  it will simplify like:

```
x and not x =>   falseOrNull(x)
x or not x    =>  trueOrNull(x)
```
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…ly to join (apache#47985)

example sql:
`SELECT *
FROM 
    t1
WHERE 
    t1.x IN (SELECT x FROM t2 WHERE t1.y is NULL AND t1.y is NOT NULL);`

the correlated filter `t1.y is NULL AND t1.y is NOT NULL` will be
replaced as `FALSE` by SimplifyConflictCompound rule. So we need check
if correlated filter exists before converting apply to join

Related PR: (apache#47385)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants