[fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 #46224

englefly · 2024-12-31T12:22:57Z

What problem does this PR solve?

Issue Number: close #xxx
pick #46406
Related PR: #40815

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

Thearas · 2024-12-31T12:23:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

englefly · 2024-12-31T12:23:18Z

run buildall

doris-robot · 2024-12-31T13:52:38Z

TPC-H: Total hot run time: 41212 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fcfe2539b3e6d5f3ce89e6b9d2ed62328b7913ff, data reload: false

------ Round 1 ----------------------------------
q1	17573	7382	7338	7338
q2	2079	172	178	172
q3	10556	1095	1226	1095
q4	10524	752	775	752
q5	7738	2916	2834	2834
q6	243	149	147	147
q7	996	623	612	612
q8	9359	1974	2039	1974
q9	6680	6458	6418	6418
q10	7004	2291	2460	2291
q11	469	263	263	263
q12	425	217	205	205
q13	17780	2988	3042	2988
q14	247	211	217	211
q15	564	537	512	512
q16	673	630	617	617
q17	974	579	584	579
q18	7241	6724	6736	6724
q19	1389	1057	1134	1057
q20	466	210	196	196
q21	4072	3238	3253	3238
q22	1127	1006	989	989
Total cold run time: 108179 ms
Total hot run time: 41212 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7242	7223	7223
q2	323	236	227	227
q3	2937	2932	3003	2932
q4	2057	1840	1840	1840
q5	5798	5766	5811	5766
q6	233	147	146	146
q7	2254	1868	1797	1797
q8	3386	3638	3490	3490
q9	8991	8955	8936	8936
q10	3587	3579	3536	3536
q11	598	516	511	511
q12	790	611	614	611
q13	9520	3220	3176	3176
q14	305	293	273	273
q15	568	529	525	525
q16	737	681	703	681
q17	1882	1621	1605	1605
q18	8323	7721	7622	7622
q19	1688	1609	1642	1609
q20	2119	1878	1855	1855
q21	5587	5387	5604	5387
q22	1119	1054	1030	1030
Total cold run time: 70092 ms
Total hot run time: 60778 ms

doris-robot · 2024-12-31T14:05:18Z

TPC-DS: Total hot run time: 197507 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fcfe2539b3e6d5f3ce89e6b9d2ed62328b7913ff, data reload: false

query1	1316	917	912	912
query2	6230	2119	2080	2080
query3	10813	4253	4357	4253
query4	65796	29289	23366	23366
query5	5059	474	466	466
query6	410	170	165	165
query7	5532	308	305	305
query8	312	235	233	233
query9	8767	2736	2725	2725
query10	458	268	254	254
query11	17309	15191	15844	15191
query12	163	105	95	95
query13	1476	442	422	422
query14	10354	7275	6766	6766
query15	225	181	193	181
query16	7186	418	472	418
query17	1073	606	588	588
query18	1941	320	323	320
query19	213	163	158	158
query20	122	113	113	113
query21	64	48	44	44
query22	4671	4627	4610	4610
query23	35832	34388	34566	34388
query24	6093	2963	2953	2953
query25	540	435	417	417
query26	669	171	165	165
query27	2029	305	305	305
query28	4377	2585	2504	2504
query29	718	476	449	449
query30	250	172	161	161
query31	1009	847	859	847
query32	68	56	60	56
query33	460	288	280	280
query34	922	522	502	502
query35	850	744	729	729
query36	1072	982	972	972
query37	120	69	78	69
query38	4171	4029	4108	4029
query39	1501	1478	1452	1452
query40	139	82	81	81
query41	50	46	48	46
query42	119	100	102	100
query43	563	499	501	499
query44	1223	823	828	823
query45	185	175	169	169
query46	1154	778	741	741
query47	2023	1901	1921	1901
query48	464	372	394	372
query49	719	386	388	386
query50	851	427	419	419
query51	7452	7226	7263	7226
query52	98	83	88	83
query53	251	184	176	176
query54	550	455	443	443
query55	75	78	75	75
query56	255	260	249	249
query57	1228	1116	1102	1102
query58	209	198	199	198
query59	3316	2969	3058	2969
query60	287	275	263	263
query61	112	108	108	108
query62	777	691	696	691
query63	215	182	185	182
query64	1412	674	651	651
query65	3271	3222	3222	3222
query66	698	301	309	301
query67	15992	15509	15474	15474
query68	4086	574	587	574
query69	437	264	274	264
query70	1210	1101	1146	1101
query71	341	255	258	255
query72	6187	4048	4094	4048
query73	759	348	350	348
query74	10315	9023	8954	8954
query75	3344	2622	2658	2622
query76	1907	1006	1033	1006
query77	505	274	276	274
query78	10701	9664	9606	9606
query79	1473	594	592	592
query80	856	444	420	420
query81	513	244	242	242
query82	1255	116	113	113
query83	220	140	141	140
query84	294	89	80	80
query85	878	331	296	296
query86	354	296	290	290
query87	4412	4380	4301	4301
query88	3795	2467	2392	2392
query89	421	292	288	288
query90	1961	189	185	185
query91	183	145	148	145
query92	64	51	50	50
query93	1900	549	553	549
query94	769	282	288	282
query95	353	258	260	258
query96	624	285	284	284
query97	3312	3184	3169	3169
query98	211	202	203	202
query99	1591	1284	1297	1284
Total cold run time: 319090 ms
Total hot run time: 197507 ms

doris-robot · 2024-12-31T14:10:49Z

ClickBench: Total hot run time: 33.12 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fcfe2539b3e6d5f3ce89e6b9d2ed62328b7913ff, data reload: false

query1	0.03	0.02	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.53	0.49	0.50
query6	1.13	0.72	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.48
query10	0.55	0.54	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.59	0.59
query14	3.09	3.12	2.96
query15	0.89	0.86	0.82
query16	0.38	0.38	0.38
query17	1.00	0.99	0.98
query18	0.24	0.22	0.21
query19	1.86	1.83	1.98
query20	0.01	0.01	0.02
query21	15.36	0.59	0.58
query22	2.73	2.16	2.05
query23	17.00	0.99	0.86
query24	3.42	1.28	0.94
query25	0.21	0.17	0.08
query26	0.47	0.14	0.14
query27	0.05	0.04	0.05
query28	10.39	1.11	1.07
query29	12.58	3.22	3.22
query30	0.25	0.06	0.05
query31	2.85	0.38	0.38
query32	3.26	0.46	0.47
query33	2.99	3.01	3.06
query34	17.19	4.49	4.54
query35	4.53	4.48	4.59
query36	0.67	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.61 s
Total hot run time: 33.12 s

xzj7019 · 2025-01-02T03:42:50Z

regression-test/suites/query_p0/runtime_filter/runtime_filter.groovy

+    ('ad', 'bc', 'cc', '2024-07-03 01:15:30', 4);
+    """
+
+    qt_1 """


reference out is missing

morrySnow · 2025-01-02T04:18:03Z

please add desc

englefly · 2025-01-03T14:26:27Z

run buildall

doris-robot · 2025-01-03T15:59:45Z

TPC-H: Total hot run time: 41092 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c4ef6f21288fb61249e9dca3bef439b3ab73b3f, data reload: false

------ Round 1 ----------------------------------
q1	17739	7497	7294	7294
q2	2048	183	198	183
q3	10539	1098	1186	1098
q4	10564	751	733	733
q5	7732	2887	2876	2876
q6	236	149	143	143
q7	973	612	612	612
q8	9366	1997	2023	1997
q9	6589	6439	6455	6439
q10	7008	2288	2316	2288
q11	470	269	263	263
q12	411	216	211	211
q13	17801	3003	2986	2986
q14	252	209	223	209
q15	581	523	507	507
q16	690	630	603	603
q17	988	532	528	528
q18	7286	6724	6704	6704
q19	1405	1030	1112	1030
q20	478	205	196	196
q21	4132	3204	3212	3204
q22	1116	988	999	988
Total cold run time: 108404 ms
Total hot run time: 41092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7410	7230	7231	7230
q2	329	238	237	237
q3	2944	2981	2938	2938
q4	2082	1776	1858	1776
q5	5729	5790	5733	5733
q6	226	144	142	142
q7	2313	1812	1820	1812
q8	3404	3628	3548	3548
q9	8919	8907	8889	8889
q10	3616	3540	3539	3539
q11	609	511	512	511
q12	856	585	602	585
q13	10388	3173	3169	3169
q14	297	270	277	270
q15	585	547	516	516
q16	725	674	669	669
q17	1859	1644	1601	1601
q18	8420	7848	7577	7577
q19	1696	1545	1523	1523
q20	2137	1875	1886	1875
q21	5632	5495	5406	5406
q22	1167	1029	1035	1029
Total cold run time: 71343 ms
Total hot run time: 60575 ms

doris-robot · 2025-01-03T16:12:29Z

TPC-DS: Total hot run time: 197891 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c4ef6f21288fb61249e9dca3bef439b3ab73b3f, data reload: false

query1	1310	937	934	934
query2	6276	2136	2088	2088
query3	10932	4576	4226	4226
query4	66958	29098	23637	23637
query5	5096	493	491	491
query6	437	196	194	194
query7	5658	315	317	315
query8	399	307	311	307
query9	9352	2727	2725	2725
query10	501	288	277	277
query11	17472	15369	15873	15369
query12	162	105	112	105
query13	1568	455	439	439
query14	10825	7471	7192	7192
query15	208	189	187	187
query16	7208	464	467	464
query17	1346	627	641	627
query18	1934	354	346	346
query19	233	171	169	169
query20	119	115	115	115
query21	67	49	51	49
query22	4717	4525	4258	4258
query23	34740	34243	34377	34243
query24	6452	2926	2960	2926
query25	551	457	467	457
query26	679	179	181	179
query27	1779	321	332	321
query28	4380	2522	2460	2460
query29	724	439	443	439
query30	251	177	170	170
query31	991	808	884	808
query32	64	55	54	54
query33	424	297	290	290
query34	910	489	508	489
query35	883	738	756	738
query36	1115	950	979	950
query37	118	84	76	76
query38	4149	4112	4002	4002
query39	1549	1490	1472	1472
query40	150	84	80	80
query41	57	49	46	46
query42	116	102	102	102
query43	548	494	504	494
query44	1188	839	836	836
query45	182	170	175	170
query46	1159	746	744	744
query47	2024	1901	1933	1901
query48	505	420	404	404
query49	742	377	380	377
query50	846	443	439	439
query51	7362	7244	6994	6994
query52	103	89	84	84
query53	261	182	183	182
query54	561	454	451	451
query55	75	76	80	76
query56	268	260	253	253
query57	1213	1094	1085	1085
query58	212	201	203	201
query59	3209	3307	2940	2940
query60	285	253	273	253
query61	131	108	106	106
query62	779	666	681	666
query63	223	189	188	188
query64	1420	663	647	647
query65	3289	3188	3175	3175
query66	703	330	302	302
query67	15865	15445	15402	15402
query68	4428	591	572	572
query69	433	268	275	268
query70	1190	1108	1071	1071
query71	365	254	246	246
query72	6580	4062	4076	4062
query73	747	357	351	351
query74	9908	8977	8937	8937
query75	3367	2645	2660	2645
query76	1863	1125	1173	1125
query77	481	274	279	274
query78	10471	9576	9625	9576
query79	1766	597	605	597
query80	1373	439	427	427
query81	527	247	247	247
query82	1254	117	115	115
query83	165	148	151	148
query84	284	84	77	77
query85	1007	298	292	292
query86	408	295	319	295
query87	4539	4306	4365	4306
query88	3701	2398	2367	2367
query89	427	292	289	289
query90	1947	188	189	188
query91	187	148	151	148
query92	69	50	50	50
query93	2467	554	551	551
query94	854	294	303	294
query95	380	259	256	256
query96	625	283	294	283
query97	3359	3163	3187	3163
query98	218	213	201	201
query99	1612	1292	1284	1284
Total cold run time: 323015 ms
Total hot run time: 197891 ms

doris-robot · 2025-01-03T16:18:02Z

ClickBench: Total hot run time: 33.91 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c4ef6f21288fb61249e9dca3bef439b3ab73b3f, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.62	0.10	0.10
query5	0.54	0.52	0.51
query6	1.14	0.74	0.73
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.51
query10	0.55	0.54	0.56
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.60	0.59	0.59
query14	2.93	2.93	3.04
query15	0.90	0.83	0.82
query16	0.39	0.39	0.39
query17	1.02	1.08	1.02
query18	0.23	0.21	0.22
query19	1.93	1.92	1.89
query20	0.02	0.01	0.01
query21	15.35	0.57	0.57
query22	2.96	2.37	2.22
query23	16.96	1.04	0.86
query24	3.44	1.38	1.24
query25	0.28	0.19	0.07
query26	0.41	0.14	0.12
query27	0.03	0.05	0.05
query28	9.95	1.12	1.07
query29	12.61	3.30	3.25
query30	0.24	0.06	0.06
query31	2.86	0.38	0.38
query32	3.25	0.46	0.47
query33	2.97	3.07	3.06
query34	16.70	4.57	4.56
query35	4.55	4.55	4.54
query36	0.67	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.02	0.03
query40	0.15	0.12	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.83 s
Total hot run time: 33.91 s

englefly · 2025-01-03T23:13:20Z

run p0

englefly · 2025-01-03T23:13:28Z

run cloud_p0

…asTransferMap apache#46224 (apache#3709)

xzj7019 reviewed Jan 2, 2025

View reviewed changes

morrySnow added dev/2.1.x dev/3.0.x usercase Important user case type label p0_b labels Jan 2, 2025

englefly closed this Jan 3, 2025

englefly reopened this Jan 3, 2025

englefly changed the title ~~[fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap~~ [fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 Jan 3, 2025

do not check aliasMap for equal.right

3c4ef6f

englefly force-pushed the rf-missOnAgg branch from fcfe253 to 3c4ef6f Compare January 3, 2025 14:26

englefly removed dev/2.1.x dev/3.0.x labels Jan 4, 2025

englefly merged commit f4c3e39 into apache:branch-3.0 Jan 4, 2025
19 of 20 checks passed

BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Feb 7, 2025

[fix](nereids)create runtime filter even if equal.right is not in ali…

500a324

…asTransferMap apache#46224 (apache#3709)

englefly deleted the rf-missOnAgg branch June 24, 2025 03:47

[fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 #46224

[fix](nereids) Create Runtime filter even if equal.right is not in aliasTransferMap-branch3.0 #46224

Uh oh!

Conversation

englefly commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

Thearas commented Dec 31, 2024

Uh oh!

englefly commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

doris-robot commented Dec 31, 2024

Uh oh!

xzj7019 Jan 2, 2025

Choose a reason for hiding this comment

Uh oh!

morrySnow commented Jan 2, 2025

Uh oh!

englefly commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

doris-robot commented Jan 3, 2025

Uh oh!

englefly commented Jan 3, 2025

Uh oh!

englefly commented Jan 3, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

englefly commented Dec 31, 2024 •

edited

Loading