[opt](lambda) let lambda expression support refer outer slot #44365

qzsee · 2024-11-21T02:33:12Z

What problem does this PR solve?

CREATE TABLE `array_test` (
  `id` int(11) NULL COMMENT "",
  `c_array` ARRAY<int(11)> NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`id`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
);

this sql is ok

select *, array_map(x -> x+1, c_array) from array_test

but lambda expressions in array_map do not support field arguments

for example:

select *, array_map(x -> x+id, c_array) from array_test

This pr solves two problems

support column arg in lambda expression
prevent high memory usage due to large block data

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-11-21T02:33:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

be/src/vec/exprs/lambda_function/varray_map_function.cpp

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java

zhangstar333 · 2024-11-29T03:51:56Z

run buildall

zhangstar333

the code compile failed, could check it
and maybe could add some desc to explain want to do in this PR

be/src/vec/core/block.cpp

qzsee · 2024-11-29T08:05:46Z

run buildall

shuke987 · 2024-11-29T08:16:24Z

run buildall

doris-robot · 2024-11-29T08:49:50Z

TeamCity be ut coverage result:
Function Coverage: 38.32% (9976/26031)
Line Coverage: 29.41% (83543/284110)
Region Coverage: 28.52% (42974/150668)
Branch Coverage: 25.15% (21826/86798)
Coverage Report: http://coverage.selectdb-in.cc/coverage/124fde4adb7f8706648139da4a078d5e1f3fcdc5_124fde4adb7f8706648139da4a078d5e1f3fcdc5/report/index.html

doris-robot · 2024-11-29T09:44:53Z

TPC-H: Total hot run time: 40130 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 124fde4adb7f8706648139da4a078d5e1f3fcdc5, data reload: false

------ Round 1 ----------------------------------
q1	17718	7509	7447	7447
q2	2061	181	175	175
q3	10683	1087	1188	1087
q4	10560	742	773	742
q5	7607	2829	2745	2745
q6	244	151	146	146
q7	991	633	598	598
q8	9258	1908	1898	1898
q9	6707	6452	6406	6406
q10	6944	2311	2302	2302
q11	454	262	255	255
q12	405	220	214	214
q13	17770	3084	3011	3011
q14	250	210	210	210
q15	578	528	526	526
q16	638	593	583	583
q17	977	542	537	537
q18	7411	6741	6642	6642
q19	1331	1022	938	938
q20	464	201	190	190
q21	3928	3339	3153	3153
q22	385	327	325	325
Total cold run time: 107364 ms
Total hot run time: 40130 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7297	7235	7232	7232
q2	329	226	238	226
q3	2936	2850	2949	2850
q4	2080	1812	1806	1806
q5	5689	5672	5663	5663
q6	238	139	140	139
q7	2241	1818	1808	1808
q8	3372	3562	3520	3520
q9	8978	9071	9041	9041
q10	3602	3553	3559	3553
q11	599	518	505	505
q12	817	614	643	614
q13	10976	3245	3222	3222
q14	299	264	266	264
q15	574	534	518	518
q16	699	650	654	650
q17	1884	1635	1642	1635
q18	8395	7897	7668	7668
q19	1715	1482	1554	1482
q20	2103	1906	1870	1870
q21	5591	5413	5313	5313
q22	681	573	585	573
Total cold run time: 71095 ms
Total hot run time: 60152 ms

doris-robot · 2024-11-29T09:57:36Z

TPC-DS: Total hot run time: 199281 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 124fde4adb7f8706648139da4a078d5e1f3fcdc5, data reload: false

query1	1290	970	941	941
query2	6230	2272	2192	2192
query3	10984	4446	4445	4445
query4	66377	28596	23736	23736
query5	4872	476	505	476
query6	426	188	179	179
query7	5578	311	308	308
query8	330	242	242	242
query9	8675	2684	2667	2667
query10	447	274	262	262
query11	17352	15418	15982	15418
query12	159	124	109	109
query13	1502	437	436	436
query14	11082	7358	7531	7358
query15	233	202	209	202
query16	7187	496	488	488
query17	1070	591	600	591
query18	1971	325	323	323
query19	236	213	156	156
query20	119	109	115	109
query21	201	105	102	102
query22	4736	4695	4608	4608
query23	35111	34563	34454	34454
query24	5393	2599	2496	2496
query25	494	388	394	388
query26	640	153	153	153
query27	1737	290	298	290
query28	4509	2491	2473	2473
query29	681	434	417	417
query30	214	156	156	156
query31	995	824	843	824
query32	66	55	59	55
query33	492	301	312	301
query34	904	539	558	539
query35	875	776	767	767
query36	1103	971	994	971
query37	123	72	79	72
query38	4506	4449	4383	4383
query39	1523	1501	1498	1498
query40	212	105	105	105
query41	45	43	43	43
query42	117	103	100	100
query43	568	526	512	512
query44	1195	881	863	863
query45	199	174	199	174
query46	1189	768	779	768
query47	2018	1925	1924	1924
query48	424	317	319	317
query49	778	394	414	394
query50	846	406	407	406
query51	7413	7292	7222	7222
query52	96	88	89	88
query53	262	181	180	180
query54	523	408	411	408
query55	81	77	74	74
query56	254	243	235	235
query57	1235	1133	1153	1133
query58	210	217	215	215
query59	3405	3195	3131	3131
query60	272	238	253	238
query61	106	108	121	108
query62	777	689	664	664
query63	221	194	187	187
query64	1359	718	643	643
query65	3316	3230	3257	3230
query66	637	302	334	302
query67	15898	15610	15707	15610
query68	4218	569	578	569
query69	431	254	263	254
query70	1229	1184	1120	1120
query71	339	251	254	251
query72	6411	4111	4047	4047
query73	759	358	359	358
query74	10409	9258	9053	9053
query75	3419	2663	2708	2663
query76	1880	1155	1076	1076
query77	489	284	279	279
query78	10525	9379	9359	9359
query79	1703	614	605	605
query80	947	460	446	446
query81	497	244	237	237
query82	1286	123	126	123
query83	258	146	144	144
query84	291	75	71	71
query85	877	293	293	293
query86	348	318	293	293
query87	4770	4746	4760	4746
query88	3586	2210	2188	2188
query89	500	301	294	294
query90	2013	190	186	186
query91	139	111	105	105
query92	67	51	50	50
query93	1992	556	553	553
query94	840	294	295	294
query95	350	255	249	249
query96	616	272	279	272
query97	2901	2732	2735	2732
query98	233	192	205	192
query99	1616	1371	1320	1320
Total cold run time: 320018 ms
Total hot run time: 199281 ms

doris-robot · 2024-11-29T10:03:08Z

ClickBench: Total hot run time: 32.79 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 124fde4adb7f8706648139da4a078d5e1f3fcdc5, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.11	0.11
query5	0.43	0.42	0.41
query6	1.14	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.51
query10	0.55	0.57	0.55
query11	0.15	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.60	0.59
query14	2.82	2.84	2.76
query15	0.91	0.83	0.82
query16	0.37	0.39	0.38
query17	0.96	1.01	1.02
query18	0.22	0.21	0.21
query19	1.80	1.86	1.98
query20	0.02	0.02	0.01
query21	15.38	0.59	0.59
query22	3.21	3.43	1.36
query23	17.27	1.09	0.83
query24	2.95	1.69	1.48
query25	0.18	0.16	0.24
query26	0.44	0.13	0.15
query27	0.05	0.04	0.04
query28	10.15	1.08	1.08
query29	12.60	3.26	3.21
query30	0.25	0.06	0.07
query31	2.86	0.41	0.38
query32	3.27	0.47	0.46
query33	3.00	3.05	3.07
query34	17.17	4.42	4.51
query35	4.53	4.52	4.48
query36	0.67	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 107.13 s
Total hot run time: 32.79 s

dqz123 · 2024-12-02T02:08:43Z

run buildall

doris-robot · 2024-12-02T03:11:54Z

TPC-H: Total hot run time: 39989 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f8be410b4c1c8cd47616921a9cc9bc57a689c68, data reload: false

------ Round 1 ----------------------------------
q1	17583	8326	7300	7300
q2	2042	177	168	168
q3	10608	1102	1207	1102
q4	10572	723	712	712
q5	7598	2773	2751	2751
q6	241	146	146	146
q7	1005	642	603	603
q8	9252	1834	1945	1834
q9	6686	6454	6434	6434
q10	6998	2275	2351	2275
q11	465	262	262	262
q12	437	225	224	224
q13	17786	3061	3023	3023
q14	257	200	214	200
q15	590	530	522	522
q16	668	599	607	599
q17	990	540	553	540
q18	7391	6640	6679	6640
q19	1318	1043	1018	1018
q20	461	179	177	177
q21	4043	3151	3249	3151
q22	381	312	308	308
Total cold run time: 107372 ms
Total hot run time: 39989 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7288	7281	7226	7226
q2	330	233	238	233
q3	2932	2914	2978	2914
q4	2056	1835	1920	1835
q5	5698	5696	5712	5696
q6	227	139	142	139
q7	2249	1820	1843	1820
q8	3412	3582	3557	3557
q9	8959	9060	9092	9060
q10	3634	3550	3568	3550
q11	610	500	492	492
q12	838	570	681	570
q13	12102	3252	3249	3249
q14	307	273	274	273
q15	588	535	535	535
q16	693	648	633	633
q17	1864	1653	1641	1641
q18	8455	7956	7805	7805
q19	1723	1697	1627	1627
q20	2133	1876	1879	1876
q21	5706	5554	5610	5554
q22	638	573	562	562
Total cold run time: 72442 ms
Total hot run time: 60847 ms

doris-robot · 2024-12-02T03:24:35Z

TPC-DS: Total hot run time: 197476 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f8be410b4c1c8cd47616921a9cc9bc57a689c68, data reload: false

query1	1234	949	973	949
query2	6234	2095	2060	2060
query3	10986	4534	4560	4534
query4	66763	28855	23461	23461
query5	4908	467	472	467
query6	408	184	180	180
query7	5484	306	296	296
query8	328	245	236	236
query9	8605	2739	2753	2739
query10	429	250	257	250
query11	17036	15504	15884	15504
query12	157	106	108	106
query13	1477	399	413	399
query14	10617	7195	8198	7195
query15	218	195	206	195
query16	6669	485	486	485
query17	1099	565	561	561
query18	1045	303	312	303
query19	218	161	169	161
query20	122	127	116	116
query21	204	106	109	106
query22	4799	4591	4408	4408
query23	35090	34535	34546	34535
query24	5596	2740	2440	2440
query25	530	388	386	386
query26	687	148	159	148
query27	2072	287	286	286
query28	4542	2524	2494	2494
query29	670	426	431	426
query30	209	148	154	148
query31	1008	830	854	830
query32	65	58	54	54
query33	406	307	298	298
query34	934	530	532	530
query35	890	749	754	749
query36	1152	963	998	963
query37	121	75	76	75
query38	4595	4340	4366	4340
query39	1522	1486	1500	1486
query40	203	100	100	100
query41	45	41	46	41
query42	110	97	101	97
query43	521	502	501	501
query44	1171	843	823	823
query45	192	168	181	168
query46	1198	714	733	714
query47	2048	1954	1902	1902
query48	410	307	300	300
query49	736	395	392	392
query50	828	390	384	384
query51	7433	7319	7096	7096
query52	99	85	90	85
query53	249	174	176	174
query54	508	388	396	388
query55	76	71	73	71
query56	254	242	232	232
query57	1270	1124	1107	1107
query58	215	208	204	204
query59	3056	3084	2983	2983
query60	261	239	248	239
query61	106	107	104	104
query62	792	664	681	664
query63	215	188	202	188
query64	1820	668	630	630
query65	3325	3281	3217	3217
query66	721	302	306	302
query67	15927	15780	15523	15523
query68	3797	583	552	552
query69	419	254	257	254
query70	1183	1164	1171	1164
query71	400	238	244	238
query72	6363	4040	4131	4040
query73	768	357	365	357
query74	10101	8945	9034	8945
query75	3406	2679	2622	2622
query76	1934	1086	1108	1086
query77	493	272	278	272
query78	10519	9605	9434	9434
query79	2131	601	609	601
query80	1363	430	447	430
query81	520	234	226	226
query82	1270	120	123	120
query83	253	147	140	140
query84	282	69	73	69
query85	1070	287	308	287
query86	421	335	303	303
query87	4688	4618	4498	4498
query88	3739	2217	2179	2179
query89	424	293	294	293
query90	1973	184	184	184
query91	133	102	102	102
query92	67	48	47	47
query93	2811	536	538	536
query94	789	286	284	284
query95	342	249	251	249
query96	640	279	283	279
query97	2917	2678	2658	2658
query98	215	190	198	190
query99	1685	1412	1309	1309
Total cold run time: 320034 ms
Total hot run time: 197476 ms

doris-robot · 2024-12-02T03:29:06Z

TeamCity be ut coverage result:
Function Coverage: 38.39% (9991/26023)
Line Coverage: 29.45% (83695/284168)
Region Coverage: 28.56% (43043/150693)
Branch Coverage: 25.19% (21881/86860)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2f8be410b4c1c8cd47616921a9cc9bc57a689c68_2f8be410b4c1c8cd47616921a9cc9bc57a689c68/report/index.html

doris-robot · 2024-12-02T03:30:05Z

ClickBench: Total hot run time: 33.06 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2f8be410b4c1c8cd47616921a9cc9bc57a689c68, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.11
query5	0.42	0.39	0.39
query6	1.17	0.66	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.56	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.61	0.59
query14	2.72	2.71	2.85
query15	0.90	0.83	0.84
query16	0.40	0.38	0.38
query17	1.07	1.05	1.04
query18	0.22	0.20	0.21
query19	1.93	1.88	2.01
query20	0.02	0.00	0.02
query21	15.43	0.58	0.58
query22	2.70	2.63	2.18
query23	17.21	0.77	0.88
query24	3.03	1.34	1.22
query25	0.31	0.18	0.09
query26	0.49	0.15	0.14
query27	0.04	0.04	0.04
query28	10.36	1.11	1.07
query29	12.56	3.22	3.17
query30	0.24	0.06	0.07
query31	2.86	0.38	0.40
query32	3.27	0.46	0.47
query33	2.96	3.03	3.02
query34	16.98	4.41	4.44
query35	4.51	4.50	4.51
query36	0.68	0.48	0.47
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.98 s
Total hot run time: 33.06 s

dqz123 · 2024-12-02T12:38:26Z

run buildall

doris-robot · 2024-12-02T13:51:23Z

TeamCity be ut coverage result:
Function Coverage: 38.39% (9978/25994)
Line Coverage: 29.45% (83688/284137)
Region Coverage: 28.58% (43055/150652)
Branch Coverage: 25.20% (21892/86858)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2ae4cac049ebb748982a09b30a632d92d9fa1daf_2ae4cac049ebb748982a09b30a632d92d9fa1daf/report/index.html

doris-robot · 2024-12-02T14:16:49Z

TPC-H: Total hot run time: 40206 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ae4cac049ebb748982a09b30a632d92d9fa1daf, data reload: false

------ Round 1 ----------------------------------
q1	17637	7657	7308	7308
q2	2047	177	171	171
q3	10725	1064	1171	1064
q4	10563	792	719	719
q5	7608	2741	2701	2701
q6	241	149	154	149
q7	1022	613	606	606
q8	9246	1899	1960	1899
q9	6787	6462	6531	6462
q10	7001	2307	2299	2299
q11	474	265	270	265
q12	428	222	215	215
q13	17769	3022	3021	3021
q14	262	225	208	208
q15	568	547	514	514
q16	666	590	605	590
q17	982	559	543	543
q18	7470	6717	6718	6717
q19	1359	1031	978	978
q20	461	183	181	181
q21	4002	3311	3279	3279
q22	379	317	317	317
Total cold run time: 107697 ms
Total hot run time: 40206 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7266	7260	7297	7260
q2	337	230	230	230
q3	2933	2880	2979	2880
q4	2086	1808	1840	1808
q5	5676	5705	5666	5666
q6	235	147	144	144
q7	2239	1843	1789	1789
q8	3452	3562	3529	3529
q9	9026	9144	9152	9144
q10	3616	3595	3560	3560
q11	592	493	499	493
q12	821	582	627	582
q13	10512	3252	3249	3249
q14	324	274	272	272
q15	572	512	533	512
q16	712	670	659	659
q17	1873	1634	1631	1631
q18	8286	7971	7617	7617
q19	1717	1390	1600	1390
q20	2119	1854	1893	1854
q21	5689	5591	5551	5551
q22	665	605	568	568
Total cold run time: 70748 ms
Total hot run time: 60388 ms

doris-robot · 2024-12-02T14:29:29Z

TPC-DS: Total hot run time: 197854 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ae4cac049ebb748982a09b30a632d92d9fa1daf, data reload: false

query1	1252	969	940	940
query2	6220	2063	2076	2063
query3	10972	4512	4411	4411
query4	67350	28899	23642	23642
query5	4930	468	453	453
query6	423	188	179	179
query7	5539	304	310	304
query8	331	250	236	236
query9	8526	2689	2689	2689
query10	430	253	237	237
query11	17381	15381	16022	15381
query12	157	102	102	102
query13	1455	414	413	413
query14	10116	7251	7806	7251
query15	214	185	192	185
query16	7095	456	471	456
query17	1302	602	595	595
query18	1798	322	336	322
query19	203	164	158	158
query20	125	117	113	113
query21	206	109	111	109
query22	4773	4558	4433	4433
query23	35045	35437	34453	34453
query24	5468	2536	2508	2508
query25	480	387	388	387
query26	640	154	152	152
query27	1812	286	298	286
query28	4265	2501	2499	2499
query29	688	403	425	403
query30	206	145	157	145
query31	1005	867	842	842
query32	64	53	55	53
query33	466	293	294	293
query34	956	513	529	513
query35	910	757	767	757
query36	1078	989	980	980
query37	127	72	85	72
query38	4433	4525	4493	4493
query39	1508	1490	1459	1459
query40	206	103	98	98
query41	45	44	51	44
query42	112	101	99	99
query43	540	511	498	498
query44	1197	839	869	839
query45	189	172	168	168
query46	1185	740	726	726
query47	2035	1943	1925	1925
query48	432	317	312	312
query49	740	406	412	406
query50	848	404	398	398
query51	7354	7258	7130	7130
query52	96	89	97	89
query53	247	176	179	176
query54	515	402	401	401
query55	77	74	84	74
query56	245	224	239	224
query57	1242	1134	1118	1118
query58	214	209	213	209
query59	3144	2880	2935	2880
query60	263	235	242	235
query61	140	107	114	107
query62	793	678	667	667
query63	209	182	194	182
query64	1385	671	646	646
query65	3303	3221	3244	3221
query66	714	299	302	299
query67	16034	15753	15655	15655
query68	4086	571	562	562
query69	426	250	247	247
query70	1206	1161	1148	1148
query71	376	248	288	248
query72	6140	4070	4066	4066
query73	754	360	363	360
query74	10379	9091	8984	8984
query75	3390	2689	2668	2668
query76	1887	1133	1097	1097
query77	478	264	270	264
query78	10497	9400	9540	9400
query79	1562	603	598	598
query80	884	477	422	422
query81	478	238	223	223
query82	1289	122	122	122
query83	245	140	146	140
query84	281	73	70	70
query85	882	311	304	304
query86	330	298	315	298
query87	4698	4544	4553	4544
query88	3619	2210	2181	2181
query89	416	292	294	292
query90	2037	193	184	184
query91	137	110	105	105
query92	67	47	49	47
query93	1945	545	546	545
query94	804	272	300	272
query95	349	245	250	245
query96	637	284	277	277
query97	2894	2651	2712	2651
query98	215	193	190	190
query99	1611	1326	1307	1307
Total cold run time: 318452 ms
Total hot run time: 197854 ms

doris-robot · 2024-12-02T14:34:58Z

ClickBench: Total hot run time: 32.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ae4cac049ebb748982a09b30a632d92d9fa1daf, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.07
query4	1.64	0.10	0.10
query5	0.43	0.40	0.41
query6	1.16	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.58	0.52	0.52
query10	0.55	0.54	0.56
query11	0.13	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.73	2.74	2.74
query15	0.92	0.82	0.83
query16	0.38	0.39	0.38
query17	1.07	1.09	1.05
query18	0.22	0.20	0.20
query19	1.97	1.87	2.03
query20	0.02	0.01	0.01
query21	15.36	0.61	0.60
query22	2.64	1.99	1.58
query23	16.96	1.19	0.74
query24	2.51	0.96	1.47
query25	0.20	0.14	0.11
query26	0.49	0.14	0.13
query27	0.04	0.04	0.04
query28	10.78	1.10	1.08
query29	12.52	3.22	3.18
query30	0.24	0.06	0.06
query31	2.87	0.38	0.38
query32	3.27	0.47	0.47
query33	2.99	3.03	3.10
query34	17.07	4.43	4.49
query35	4.51	4.55	4.48
query36	0.68	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.07	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.56 s
Total hot run time: 32.32 s

dqz123 · 2024-12-04T06:37:55Z

run buildall

doris-robot · 2024-12-04T08:22:13Z

TPC-H: Total hot run time: 40116 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f94bfa9ed414a11b4c330722180e95972c3820c8, data reload: false

------ Round 1 ----------------------------------
q1	17602	7557	7340	7340
q2	2042	173	166	166
q3	10634	1075	1180	1075
q4	10571	740	832	740
q5	7622	2730	2780	2730
q6	240	152	147	147
q7	995	634	597	597
q8	9244	1878	1968	1878
q9	6768	6475	6542	6475
q10	7007	2312	2325	2312
q11	467	258	254	254
q12	431	223	218	218
q13	17779	3032	3048	3032
q14	261	218	219	218
q15	572	550	511	511
q16	612	577	587	577
q17	997	569	511	511
q18	7356	6645	6610	6610
q19	1335	1052	985	985
q20	475	179	184	179
q21	4019	3301	3244	3244
q22	370	324	317	317
Total cold run time: 107399 ms
Total hot run time: 40116 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7475	7326	7275	7275
q2	325	239	235	235
q3	2927	2879	2919	2879
q4	2079	1826	1858	1826
q5	5718	5713	5729	5713
q6	233	139	149	139
q7	2270	1805	1796	1796
q8	3451	3609	3537	3537
q9	8964	9088	9076	9076
q10	3640	3544	3554	3544
q11	592	488	503	488
q12	840	617	604	604
q13	13597	3260	3211	3211
q14	317	288	281	281
q15	584	539	529	529
q16	691	634	643	634
q17	1897	1644	1640	1640
q18	8335	7858	7503	7503
q19	1708	1512	1678	1512
q20	2139	1900	1870	1870
q21	5629	5411	5403	5403
q22	660	584	575	575
Total cold run time: 74071 ms
Total hot run time: 60270 ms

doris-robot · 2024-12-04T08:34:51Z

TPC-DS: Total hot run time: 197608 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f94bfa9ed414a11b4c330722180e95972c3820c8, data reload: false

query1	1265	978	940	940
query2	6224	2065	2012	2012
query3	10971	4586	4463	4463
query4	66796	29216	23763	23763
query5	5057	497	439	439
query6	421	185	173	173
query7	5700	300	290	290
query8	336	243	239	239
query9	9333	2650	2638	2638
query10	478	249	249	249
query11	17415	15440	15915	15440
query12	156	106	104	104
query13	1558	428	409	409
query14	10123	7228	6886	6886
query15	219	196	195	195
query16	7237	515	497	497
query17	1194	607	566	566
query18	1845	308	296	296
query19	220	154	152	152
query20	121	139	109	109
query21	213	105	105	105
query22	4758	4475	4415	4415
query23	35092	34478	35137	34478
query24	5459	2569	2554	2554
query25	483	376	381	376
query26	634	155	156	155
query27	1756	277	288	277
query28	4419	2512	2451	2451
query29	699	425	432	425
query30	215	152	156	152
query31	1002	860	865	860
query32	66	55	57	55
query33	421	307	274	274
query34	955	530	548	530
query35	885	763	781	763
query36	1103	981	973	973
query37	119	78	72	72
query38	4595	4450	4518	4450
query39	1546	1500	1503	1500
query40	196	103	105	103
query41	45	42	43	42
query42	112	96	99	96
query43	554	487	506	487
query44	1195	842	860	842
query45	198	180	178	178
query46	1177	734	727	727
query47	2073	1955	1947	1947
query48	426	318	325	318
query49	764	406	404	404
query50	856	396	394	394
query51	7519	7324	7076	7076
query52	96	87	87	87
query53	260	183	178	178
query54	520	406	389	389
query55	80	76	83	76
query56	247	233	238	233
query57	1250	1118	1092	1092
query58	209	209	211	209
query59	3162	3115	2984	2984
query60	270	239	267	239
query61	108	101	109	101
query62	773	688	672	672
query63	212	187	195	187
query64	1352	695	631	631
query65	3272	3266	3207	3207
query66	705	299	303	299
query67	15797	15602	15519	15519
query68	4227	567	583	567
query69	420	255	243	243
query70	1161	1130	1137	1130
query71	332	255	254	254
query72	6318	4109	4050	4050
query73	768	357	359	357
query74	10316	9001	9020	9001
query75	3421	2657	2658	2657
query76	1883	1150	1137	1137
query77	503	266	274	266
query78	10466	9425	9478	9425
query79	1121	591	608	591
query80	834	470	433	433
query81	502	226	237	226
query82	224	124	117	117
query83	169	209	138	138
query84	274	70	74	70
query85	858	297	293	293
query86	335	297	295	295
query87	4764	4602	4654	4602
query88	3526	2226	2195	2195
query89	415	299	291	291
query90	1980	185	183	183
query91	141	102	104	102
query92	63	49	50	49
query93	1186	537	536	536
query94	791	296	283	283
query95	354	263	246	246
query96	614	289	283	283
query97	2852	2650	2706	2650
query98	214	196	187	187
query99	1607	1337	1309	1309
Total cold run time: 317156 ms
Total hot run time: 197608 ms

doris-robot · 2024-12-04T08:40:21Z

ClickBench: Total hot run time: 32.83 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f94bfa9ed414a11b4c330722180e95972c3820c8, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.64	0.11	0.10
query5	0.41	0.42	0.38
query6	1.16	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.60	0.51	0.50
query10	0.57	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.61	0.59
query14	2.69	2.82	2.74
query15	0.91	0.83	0.82
query16	0.39	0.37	0.39
query17	1.04	0.99	1.04
query18	0.22	0.21	0.22
query19	1.94	1.87	2.02
query20	0.01	0.01	0.02
query21	15.37	0.61	0.57
query22	2.87	2.91	1.92
query23	16.99	1.01	0.76
query24	3.03	1.22	1.92
query25	0.27	0.11	0.05
query26	0.57	0.14	0.13
query27	0.04	0.04	0.04
query28	9.72	1.10	1.07
query29	12.58	3.18	3.19
query30	0.25	0.06	0.07
query31	2.85	0.38	0.38
query32	3.29	0.47	0.46
query33	2.99	3.07	3.03
query34	16.82	4.48	4.56
query35	4.52	4.53	4.54
query36	0.67	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.18 s
Total hot run time: 32.83 s

doris-robot · 2024-12-04T08:40:24Z

TeamCity be ut coverage result:
Function Coverage: 38.47% (10008/26014)
Line Coverage: 29.52% (83930/284279)
Region Coverage: 28.62% (43124/150666)
Branch Coverage: 25.22% (21919/86922)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f94bfa9ed414a11b4c330722180e95972c3820c8_f94bfa9ed414a11b4c330722180e95972c3820c8/report/index.html

dqz123 · 2024-12-06T07:38:48Z

run buildall

doris-robot · 2024-12-06T08:51:02Z

TeamCity be ut coverage result:
Function Coverage: 38.47% (10007/26012)
Line Coverage: 29.49% (83890/284512)
Region Coverage: 28.59% (43120/150805)
Branch Coverage: 25.19% (21921/87022)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6eeaa9f355e4a3db7073fe2394c38274a3b7d854_6eeaa9f355e4a3db7073fe2394c38274a3b7d854/report/index.html

doris-robot · 2024-12-06T08:53:15Z

TPC-H: Total hot run time: 40160 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6eeaa9f355e4a3db7073fe2394c38274a3b7d854, data reload: false

------ Round 1 ----------------------------------
q1	17607	7575	7311	7311
q2	2051	194	173	173
q3	10871	1091	1224	1091
q4	10556	791	689	689
q5	7631	2776	2755	2755
q6	236	146	146	146
q7	990	655	600	600
q8	9236	1949	1919	1919
q9	6691	6524	6496	6496
q10	7061	2343	2321	2321
q11	465	268	262	262
q12	415	224	226	224
q13	17770	3066	3033	3033
q14	249	208	220	208
q15	590	544	531	531
q16	671	591	593	591
q17	999	531	517	517
q18	7241	6644	6735	6644
q19	1387	1027	968	968
q20	468	179	183	179
q21	4012	3289	3183	3183
q22	394	325	319	319
Total cold run time: 107591 ms
Total hot run time: 40160 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7344	7246	7226	7226
q2	346	225	232	225
q3	2940	2922	2954	2922
q4	2070	1810	1857	1810
q5	5662	5680	5697	5680
q6	225	144	139	139
q7	2208	1826	1823	1823
q8	3363	3527	3512	3512
q9	9006	9097	9057	9057
q10	3572	3557	3557	3557
q11	601	528	516	516
q12	819	579	624	579
q13	11167	3320	3243	3243
q14	298	274	279	274
q15	562	525	527	525
q16	694	650	626	626
q17	1876	1656	1643	1643
q18	8372	7877	7497	7497
q19	1694	1512	1477	1477
q20	2115	1874	1865	1865
q21	5493	5492	5458	5458
q22	680	611	572	572
Total cold run time: 71107 ms
Total hot run time: 60226 ms

doris-robot · 2024-12-06T09:05:32Z

TPC-DS: Total hot run time: 197556 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6eeaa9f355e4a3db7073fe2394c38274a3b7d854, data reload: false

query1	1495	971	984	971
query2	6235	2203	2089	2089
query3	10939	4375	4500	4375
query4	32952	23587	23628	23587
query5	3959	472	475	472
query6	268	174	165	165
query7	4002	306	293	293
query8	293	230	227	227
query9	9368	2733	2723	2723
query10	473	247	251	247
query11	17897	15370	15257	15257
query12	154	105	100	100
query13	1569	422	421	421
query14	8984	7572	7080	7080
query15	268	208	200	200
query16	8068	476	516	476
query17	1740	593	567	567
query18	2116	324	303	303
query19	351	150	155	150
query20	117	115	109	109
query21	203	104	103	103
query22	4995	4689	4315	4315
query23	36506	34810	34635	34635
query24	9915	2534	2602	2534
query25	516	377	389	377
query26	696	153	153	153
query27	2297	303	296	296
query28	6502	2491	2469	2469
query29	659	425	407	407
query30	235	153	147	147
query31	1041	853	834	834
query32	95	55	55	55
query33	688	296	302	296
query34	972	531	538	531
query35	928	805	773	773
query36	1121	975	970	970
query37	124	79	79	79
query38	4809	4364	4498	4364
query39	1497	1442	1507	1442
query40	211	110	107	107
query41	47	48	46	46
query42	111	102	101	101
query43	556	507	509	507
query44	1226	819	810	810
query45	192	170	173	170
query46	1186	764	722	722
query47	2094	1975	1929	1929
query48	419	328	326	326
query49	889	404	393	393
query50	833	399	394	394
query51	7407	7324	7140	7140
query52	101	90	96	90
query53	257	184	177	177
query54	1134	434	409	409
query55	81	78	79	78
query56	260	261	232	232
query57	1289	1138	1096	1096
query58	233	224	201	201
query59	3388	3240	3110	3110
query60	268	241	240	240
query61	107	105	103	103
query62	875	694	673	673
query63	218	195	182	182
query64	3561	658	632	632
query65	3290	3213	3175	3175
query66	816	300	299	299
query67	16020	15624	15673	15624
query68	4422	561	558	558
query69	417	248	251	248
query70	1198	1156	1135	1135
query71	391	244	254	244
query72	6238	4054	3979	3979
query73	765	373	362	362
query74	10425	9024	9007	9007
query75	3528	2667	2648	2648
query76	2719	1275	1076	1076
query77	371	268	272	268
query78	10476	9549	9399	9399
query79	2520	598	621	598
query80	1079	426	429	426
query81	542	236	234	234
query82	688	121	109	109
query83	234	146	141	141
query84	321	71	67	67
query85	1590	311	297	297
query86	483	315	298	298
query87	4789	4619	4543	4543
query88	4211	2220	2205	2205
query89	426	298	294	294
query90	2012	187	185	185
query91	135	104	102	102
query92	61	54	48	48
query93	2333	546	549	546
query94	915	280	287	280
query95	362	245	250	245
query96	614	280	282	280
query97	2865	2692	2668	2668
query98	217	194	208	194
query99	1547	1323	1314	1314
Total cold run time: 301010 ms
Total hot run time: 197556 ms

doris-robot · 2024-12-06T09:11:03Z

ClickBench: Total hot run time: 33.55 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6eeaa9f355e4a3db7073fe2394c38274a3b7d854, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.45	0.41	0.42
query6	1.16	0.64	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.54	0.53	0.51
query10	0.59	0.55	0.56
query11	0.14	0.10	0.11
query12	0.15	0.12	0.11
query13	0.62	0.61	0.61
query14	2.72	2.76	2.76
query15	0.91	0.83	0.83
query16	0.37	0.36	0.39
query17	1.02	1.01	0.97
query18	0.23	0.21	0.21
query19	1.94	1.85	1.97
query20	0.02	0.01	0.01
query21	15.36	0.62	0.59
query22	2.56	2.53	1.93
query23	17.10	0.84	0.83
query24	3.29	1.86	2.01
query25	0.29	0.19	0.08
query26	0.57	0.14	0.13
query27	0.05	0.04	0.05
query28	9.46	1.12	1.08
query29	12.60	3.22	3.25
query30	0.26	0.06	0.05
query31	2.86	0.38	0.38
query32	3.26	0.47	0.46
query33	3.00	3.00	3.05
query34	17.05	4.52	4.49
query35	4.53	4.43	4.49
query36	0.67	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.15	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.03
Total cold run time: 106.27 s
Total hot run time: 33.55 s

dqz123 · 2024-12-06T13:22:33Z

run p0

…#44365) ```sql CREATE TABLE `array_test` ( `id` int(11) NULL COMMENT "", `c_array` ARRAY<int(11)> NULL COMMENT "" ) ENGINE=OLAP DUPLICATE KEY(`id`) COMMENT "OLAP" DISTRIBUTED BY HASH(`id`) BUCKETS 1 PROPERTIES ( "replication_allocation" = "tag.location.default: 1", "in_memory" = "false", "storage_format" = "V2" ); ``` this sql is ok ```sql select *, array_map(x -> x+1, c_array) from array_test ``` but lambda expressions in `array_map` do not support field arguments for example: ```sql select *, array_map(x -> x+id, c_array) from array_test ``` This pr solves two problems 1. support column arg in lambda expression 2. prevent high memory usage due to large block data Related PR: #xxx let lambda expression support refer outer slot Co-authored-by: garenshi <garenshi@tencent.com>

…slot #44365 (#49394)

zhangstar333 reviewed Nov 25, 2024

View reviewed changes

be/src/vec/exprs/lambda_function/varray_map_function.cpp Outdated Show resolved Hide resolved

morrySnow reviewed Nov 26, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java Show resolved Hide resolved

qzsee force-pushed the fix-lambda-slot-bind branch 2 times, most recently from c12ee00 to fb6e74e Compare November 29, 2024 03:30

morrySnow reviewed Nov 29, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/SubExprAnalyzer.java Show resolved Hide resolved

zhangstar333 reviewed Nov 29, 2024

View reviewed changes

be/src/vec/core/block.cpp Outdated Show resolved Hide resolved

qzsee force-pushed the fix-lambda-slot-bind branch from 2ae4cac to f94bfa9 Compare December 4, 2024 06:36

morrySnow closed this Dec 6, 2024

morrySnow reopened this Dec 6, 2024

1

6eeaa9f

qzsee dismissed stale reviews from morrySnow and zhangstar333 via 6eeaa9f December 6, 2024 07:37

morrySnow closed this Dec 9, 2024

morrySnow reopened this Dec 9, 2024

morrySnow approved these changes Dec 9, 2024

View reviewed changes

morrySnow requested a review from zhangstar333 December 9, 2024 02:54

zhangstar333 approved these changes Dec 9, 2024

View reviewed changes

zhangstar333 merged commit a25d457 into apache:master Dec 9, 2024
16 of 17 checks passed

qzsee mentioned this pull request Dec 9, 2024

[opt](lambda) let lambda expression support refer outer slot #45186

Merged

16 tasks

yiguolei added the dev/2.1.8-merged label Dec 11, 2024

wm1581066 added the usercase Important user case type label label Feb 6, 2025

morrySnow mentioned this pull request Mar 24, 2025

[Improve] lambda expression can support simple nesting #49378

Closed

16 tasks

morrySnow added the dev/3.0.x label Mar 24, 2025

github-actions bot added the dev/3.0.x-conflict label Mar 24, 2025

qzsee mentioned this pull request Mar 24, 2025

branch-3.0: [opt](lambda) let lambda expression support refer outer slot #44365 #49394

Merged

16 tasks

dataroaring pushed a commit that referenced this pull request May 7, 2025

branch-3.0: [opt](lambda) let lambda expression support refer outer …

d87bf7a

…slot #44365 (#49394)

dataroaring added dev/3.0.6-merged and removed dev/3.0.x dev/3.0.x-conflict labels May 7, 2025

gavinchou mentioned this pull request Jun 11, 2025

Release Note 3.0.6 #51585

Closed

[opt](lambda) let lambda expression support refer outer slot #44365

[opt](lambda) let lambda expression support refer outer slot #44365

Uh oh!

Conversation

qzsee commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

doris-robot commented Nov 21, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhangstar333 commented Nov 29, 2024

Uh oh!

zhangstar333 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

qzsee commented Nov 29, 2024

Uh oh!

shuke987 commented Nov 29, 2024

Uh oh!

doris-robot commented Nov 29, 2024

Uh oh!

doris-robot commented Nov 29, 2024

Uh oh!

doris-robot commented Nov 29, 2024

Uh oh!

doris-robot commented Nov 29, 2024

Uh oh!

dqz123 commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

dqz123 commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

doris-robot commented Dec 2, 2024

Uh oh!

dqz123 commented Dec 4, 2024

Uh oh!

doris-robot commented Dec 4, 2024

Uh oh!

doris-robot commented Dec 4, 2024

Uh oh!

doris-robot commented Dec 4, 2024

Uh oh!

doris-robot commented Dec 4, 2024

Uh oh!

dqz123 commented Dec 6, 2024

Uh oh!

doris-robot commented Dec 6, 2024

Uh oh!

doris-robot commented Dec 6, 2024

Uh oh!

doris-robot commented Dec 6, 2024

Uh oh!

doris-robot commented Dec 6, 2024

Uh oh!

dqz123 commented Dec 6, 2024

Uh oh!

Uh oh!

Reviewers

qzsee commented Nov 21, 2024 •

edited

Loading

zhangstar333 left a comment •

edited

Loading