[fix](nereids)OrToIn and SimplifyRange rules dead loop #41689

englefly · 2024-10-11T02:54:29Z

intro by #41222

[bug]
OrToIn Rule set state flag in Or expression to avoid repeatedly process the same expression. But SimplifyRange Rule may remove this flag, and hence OrToIn and SimplifyRange Rules go into dead loop.

[fix]
if the input expression and output expression of SimplifyRange Rule are the same, SimplifyRange will return input expression to keep expression state.

doris-robot · 2024-10-11T02:54:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

englefly · 2024-10-11T02:54:42Z

run buildall

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionOptimization.java

englefly · 2024-10-11T03:28:16Z

run buildall

englefly · 2024-10-11T11:46:48Z

run p0

englefly · 2024-10-11T11:47:01Z

run performance

doris-robot · 2024-10-11T12:40:35Z

TPC-H: Total hot run time: 40751 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7839b3287d1745ac4cd9b339d11b411196c64b6b, data reload: false

------ Round 1 ----------------------------------
q1	17614	8067	7265	7265
q2	2027	279	280	279
q3	12056	1077	1220	1077
q4	10562	767	744	744
q5	7742	2902	2736	2736
q6	245	154	150	150
q7	990	624	615	615
q8	9349	1921	2020	1921
q9	6594	6413	6416	6413
q10	6964	2302	2285	2285
q11	436	246	245	245
q12	408	221	219	219
q13	17771	2984	2999	2984
q14	254	208	207	207
q15	566	510	522	510
q16	643	584	587	584
q17	971	552	551	551
q18	7334	6605	6714	6605
q19	1348	1068	970	970
q20	485	202	204	202
q21	4046	3182	3252	3182
q22	1134	1009	1007	1007
Total cold run time: 109539 ms
Total hot run time: 40751 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7300	7627	7249	7249
q2	324	235	235	235
q3	3103	2940	2972	2940
q4	2127	1824	1796	1796
q5	5727	5759	5784	5759
q6	259	146	146	146
q7	2280	1879	1834	1834
q8	3405	3527	3496	3496
q9	8879	8884	8823	8823
q10	3616	3548	3518	3518
q11	594	491	496	491
q12	846	627	602	602
q13	8782	3193	3163	3163
q14	312	270	267	267
q15	587	540	526	526
q16	684	645	633	633
q17	1835	1611	1611	1611
q18	8333	7722	7625	7625
q19	1706	1594	1530	1530
q20	2136	1858	1919	1858
q21	5705	5457	5385	5385
q22	1130	1053	1069	1053
Total cold run time: 69670 ms
Total hot run time: 60540 ms

doris-robot · 2024-10-11T12:52:39Z

TPC-DS: Total hot run time: 191179 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7839b3287d1745ac4cd9b339d11b411196c64b6b, data reload: false

query1	968	399	396	396
query2	6237	2057	2058	2057
query3	8677	197	201	197
query4	34273	23602	23502	23502
query5	3591	475	456	456
query6	272	163	166	163
query7	4201	307	303	303
query8	286	230	234	230
query9	9354	2707	2716	2707
query10	468	280	285	280
query11	17711	15312	15317	15312
query12	154	97	96	96
query13	1549	457	424	424
query14	8831	6671	6811	6671
query15	230	177	177	177
query16	7696	464	450	450
query17	1615	571	598	571
query18	2001	339	322	322
query19	204	157	168	157
query20	120	110	112	110
query21	211	108	107	107
query22	4742	4453	4639	4453
query23	35246	34194	34101	34101
query24	12616	2807	2864	2807
query25	708	423	399	399
query26	1787	168	166	166
query27	2875	306	296	296
query28	7735	2424	2404	2404
query29	1172	462	448	448
query30	271	163	154	154
query31	1033	798	819	798
query32	99	51	56	51
query33	759	302	307	302
query34	967	525	518	518
query35	895	745	749	745
query36	1098	963	970	963
query37	294	85	86	85
query38	4036	3914	3898	3898
query39	1492	1388	1414	1388
query40	260	97	99	97
query41	46	44	43	43
query42	121	98	99	98
query43	518	493	479	479
query44	1275	800	828	800
query45	196	169	163	163
query46	1137	715	716	715
query47	1944	1826	1841	1826
query48	432	351	352	351
query49	1030	422	405	405
query50	837	419	423	419
query51	7044	6904	6920	6904
query52	97	84	90	84
query53	259	179	181	179
query54	1271	479	486	479
query55	83	78	77	77
query56	287	252	274	252
query57	1281	1167	1138	1138
query58	230	229	233	229
query59	3145	2945	3037	2945
query60	296	275	263	263
query61	103	99	103	99
query62	859	663	664	663
query63	223	181	181	181
query64	4892	635	615	615
query65	3297	3183	3171	3171
query66	1414	300	301	300
query67	15980	15594	15369	15369
query68	4517	574	556	556
query69	498	299	305	299
query70	1166	1101	1130	1101
query71	352	282	311	282
query72	7023	3928	3897	3897
query73	775	345	348	345
query74	10169	8941	8890	8890
query75	3404	2730	2693	2693
query76	2943	870	862	862
query77	393	294	294	294
query78	10388	9587	9543	9543
query79	1146	598	586	586
query80	2155	437	445	437
query81	572	238	241	238
query82	761	137	132	132
query83	282	143	131	131
query84	261	76	81	76
query85	1495	289	281	281
query86	386	311	298	298
query87	4429	4391	4245	4245
query88	3293	2491	2420	2420
query89	402	294	295	294
query90	2210	189	192	189
query91	139	100	104	100
query92	61	46	46	46
query93	1081	539	543	539
query94	1143	300	308	300
query95	359	262	247	247
query96	608	281	282	281
query97	3272	3146	3087	3087
query98	222	206	195	195
query99	1547	1287	1311	1287
Total cold run time: 302483 ms
Total hot run time: 191179 ms

doris-robot · 2024-10-11T12:58:06Z

ClickBench: Total hot run time: 32.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7839b3287d1745ac4cd9b339d11b411196c64b6b, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.53	0.51	0.50
query6	1.15	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.56	0.52	0.50
query10	0.56	0.55	0.57
query11	0.14	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.61	0.59
query14	2.71	2.73	2.73
query15	0.90	0.83	0.81
query16	0.39	0.39	0.38
query17	1.00	1.06	1.11
query18	0.20	0.20	0.20
query19	1.99	1.81	2.03
query20	0.01	0.01	0.01
query21	15.36	0.61	0.57
query22	3.23	1.63	1.80
query23	17.12	0.89	0.78
query24	2.84	1.95	0.59
query25	0.23	0.11	0.05
query26	0.45	0.15	0.14
query27	0.05	0.04	0.04
query28	10.47	1.10	1.08
query29	12.51	3.25	3.22
query30	0.24	0.06	0.05
query31	2.88	0.39	0.39
query32	3.27	0.46	0.47
query33	2.99	3.05	3.03
query34	16.95	4.47	4.45
query35	4.51	4.56	4.52
query36	0.68	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.16 s
Total hot run time: 32.13 s

englefly · 2024-10-14T02:09:15Z

run p0

englefly · 2024-10-14T09:02:21Z

run buildall

doris-robot · 2024-10-14T09:41:23Z

TPC-H: Total hot run time: 41140 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2a3e6cba326da401f7d798f4317a89fa143baec3, data reload: false

------ Round 1 ----------------------------------
q1	17583	7351	7262	7262
q2	2018	278	276	276
q3	12262	1066	1205	1066
q4	10562	891	861	861
q5	7751	3058	3058	3058
q6	235	154	151	151
q7	1017	607	607	607
q8	9434	1926	1977	1926
q9	7376	6475	6374	6374
q10	7087	2398	2423	2398
q11	446	249	246	246
q12	409	220	219	219
q13	17774	2986	3007	2986
q14	248	210	206	206
q15	577	520	528	520
q16	645	590	581	581
q17	961	580	517	517
q18	7196	6599	6785	6599
q19	1378	1003	913	913
q20	475	192	185	185
q21	3949	3273	3177	3177
q22	1129	1012	1022	1012
Total cold run time: 110512 ms
Total hot run time: 41140 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7293	7234	7242	7234
q2	327	234	229	229
q3	3086	2981	2970	2970
q4	2062	1844	1780	1780
q5	5761	5753	5761	5753
q6	243	144	147	144
q7	2189	1841	1801	1801
q8	3398	3501	3492	3492
q9	8991	8915	8838	8838
q10	3587	3580	3583	3580
q11	605	509	510	509
q12	847	611	611	611
q13	11256	3186	3182	3182
q14	305	265	282	265
q15	579	538	531	531
q16	695	650	665	650
q17	1853	1603	1621	1603
q18	8181	7691	7717	7691
q19	1707	1389	1468	1389
q20	2146	1871	1962	1871
q21	5670	5360	5396	5360
q22	1131	1046	1039	1039
Total cold run time: 71912 ms
Total hot run time: 60522 ms

doris-robot · 2024-10-14T09:53:27Z

TPC-DS: Total hot run time: 192426 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2a3e6cba326da401f7d798f4317a89fa143baec3, data reload: false

query1	957	383	405	383
query2	6260	2083	2031	2031
query3	8682	193	198	193
query4	34252	23716	23669	23669
query5	3516	472	458	458
query6	271	164	158	158
query7	4184	300	294	294
query8	278	213	217	213
query9	9553	2652	2647	2647
query10	454	271	264	264
query11	17879	15259	15239	15239
query12	155	98	96	96
query13	1562	416	405	405
query14	9434	7529	7416	7416
query15	253	170	179	170
query16	7791	441	438	438
query17	1640	611	585	585
query18	2084	305	330	305
query19	316	173	151	151
query20	124	119	113	113
query21	211	130	113	113
query22	4628	4638	4422	4422
query23	35237	34503	34296	34296
query24	10977	2781	2772	2772
query25	630	420	396	396
query26	1135	162	159	159
query27	2256	285	287	285
query28	7456	2388	2382	2382
query29	820	422	411	411
query30	269	149	152	149
query31	1033	804	797	797
query32	98	55	59	55
query33	754	300	300	300
query34	940	501	508	501
query35	880	765	750	750
query36	1174	949	971	949
query37	153	86	94	86
query38	4060	3871	3906	3871
query39	1482	1485	1428	1428
query40	203	97	97	97
query41	45	44	45	44
query42	121	100	97	97
query43	539	470	476	470
query44	1301	833	818	818
query45	189	166	162	162
query46	1164	701	715	701
query47	1957	1850	1842	1842
query48	416	322	338	322
query49	963	418	397	397
query50	815	395	396	395
query51	7152	6973	6966	6966
query52	99	87	87	87
query53	259	184	179	179
query54	1247	421	435	421
query55	82	76	80	76
query56	283	263	263	263
query57	1318	1157	1175	1157
query58	233	264	279	264
query59	3126	3036	2995	2995
query60	301	266	265	265
query61	101	111	127	111
query62	858	678	654	654
query63	215	194	188	188
query64	4006	637	627	627
query65	3258	3358	3245	3245
query66	823	296	296	296
query67	15962	15648	15735	15648
query68	4255	575	564	564
query69	535	292	293	292
query70	1182	1074	1093	1074
query71	421	275	264	264
query72	7544	3967	4014	3967
query73	774	353	354	353
query74	9713	8942	9073	8942
query75	3629	2692	2708	2692
query76	3295	930	860	860
query77	626	314	316	314
query78	10680	9945	9606	9606
query79	1622	592	600	592
query80	2411	474	461	461
query81	594	237	248	237
query82	736	139	138	138
query83	307	138	132	132
query84	285	74	68	68
query85	1614	299	291	291
query86	438	304	296	296
query87	4490	4379	4418	4379
query88	3117	2201	2148	2148
query89	403	294	290	290
query90	2061	188	189	188
query91	133	100	100	100
query92	65	45	45	45
query93	1082	527	540	527
query94	1179	281	291	281
query95	352	256	254	254
query96	617	277	277	277
query97	3320	3178	3149	3149
query98	220	199	198	198
query99	1552	1310	1326	1310
Total cold run time: 300126 ms
Total hot run time: 192426 ms

doris-robot · 2024-10-14T09:58:54Z

ClickBench: Total hot run time: 32.44 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2a3e6cba326da401f7d798f4317a89fa143baec3, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.64	0.10	0.10
query5	0.50	0.50	0.48
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.05	0.03
query9	0.56	0.50	0.49
query10	0.55	0.54	0.55
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.61
query14	2.82	2.70	2.73
query15	0.90	0.83	0.82
query16	0.39	0.37	0.38
query17	1.03	1.03	0.97
query18	0.20	0.20	0.20
query19	1.81	1.88	2.00
query20	0.02	0.01	0.00
query21	15.37	0.62	0.60
query22	2.78	2.16	1.58
query23	16.96	0.97	0.92
query24	3.94	1.59	0.98
query25	0.34	0.24	0.05
query26	0.41	0.14	0.14
query27	0.04	0.04	0.04
query28	9.82	1.11	1.08
query29	12.56	3.25	3.29
query30	0.24	0.06	0.05
query31	2.86	0.39	0.38
query32	3.28	0.45	0.46
query33	3.02	3.06	3.03
query34	17.04	4.44	4.44
query35	4.51	4.49	4.44
query36	0.69	0.48	0.50
query37	0.08	0.06	0.05
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.15 s
Total hot run time: 32.44 s

englefly · 2024-10-15T02:25:05Z

run feut

github-actions · 2024-10-15T04:29:44Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-15T04:29:46Z

PR approved by anyone and no changes requested.

…e#41689)" This reverts commit af7eb15.

englefly changed the title ~~[fix](nereids)Or to in simplify range~~ [fix](nereids)OrToIn and SimplifyRange rules dead loop Oct 11, 2024

morrySnow reviewed Oct 11, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/ExpressionOptimization.java Outdated Show resolved Hide resolved

fix bug: or-to-in and simplify-range conflict

7839b32

englefly force-pushed the or-to-in-simplify-range branch from 3015021 to 7839b32 Compare October 11, 2024 03:27

update tpcds 13 shape, orToInTest add case for further optmize

2a3e6cb

starocean999 approved these changes Oct 15, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 15, 2024

github-actions bot added the reviewed label Oct 15, 2024

924060929 approved these changes Oct 15, 2024

View reviewed changes

englefly merged commit af7eb15 into apache:master Oct 15, 2024

englefly deleted the or-to-in-simplify-range branch October 15, 2024 06:48

morrySnow added the dev/3.0.x label Oct 15, 2024

englefly added a commit to englefly/incubator-doris that referenced this pull request Oct 15, 2024

pick apache#41689

181a00e

englefly mentioned this pull request Oct 15, 2024

[fix](nereids)OrToIn and SimplifyRange rules dead loop #41689 (3.0) #41887

Closed

englefly added a commit to englefly/incubator-doris that referenced this pull request Oct 17, 2024

Revert "[fix](nereids)OrToIn and SimplifyRange rules dead loop (apach…

34d3406

…e#41689)" This reverts commit af7eb15.

englefly removed the dev/3.0.x label Oct 22, 2024

[fix](nereids)OrToIn and SimplifyRange rules dead loop #41689

[fix](nereids)OrToIn and SimplifyRange rules dead loop #41689

Uh oh!

Conversation

englefly commented Oct 11, 2024 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Oct 11, 2024

Uh oh!

englefly commented Oct 11, 2024

Uh oh!

Uh oh!

englefly commented Oct 11, 2024

Uh oh!

englefly commented Oct 11, 2024

Uh oh!

englefly commented Oct 11, 2024

Uh oh!

doris-robot commented Oct 11, 2024

Uh oh!

doris-robot commented Oct 11, 2024

Uh oh!

doris-robot commented Oct 11, 2024

Uh oh!

englefly commented Oct 14, 2024

Uh oh!

englefly commented Oct 14, 2024

Uh oh!

doris-robot commented Oct 14, 2024

Uh oh!

doris-robot commented Oct 14, 2024

Uh oh!

doris-robot commented Oct 14, 2024

Uh oh!

englefly commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

englefly commented Oct 11, 2024 •

edited by morrySnow

Loading