Skip to content

branch-3.0: [opt](lambda) let lambda expression support refer outer slot #44365 #49394

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 3 commits into from
May 7, 2025
Merged

branch-3.0: [opt](lambda) let lambda expression support refer outer slot #44365 #49394

dataroaring merged 3 commits into from
May 7, 2025

Conversation

@qzsee
Copy link
Contributor

@qzsee qzsee commented Mar 24, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

pick from : #44365

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@qzsee
[opt](lambda) let lambda expression support refer outer slot (apache…
8ca4c81 
…#44365)

```sql
CREATE TABLE `array_test` (
  `id` int(11) NULL COMMENT "",
  `c_array` ARRAY<int(11)> NULL COMMENT ""
) ENGINE=OLAP
DUPLICATE KEY(`id`)
COMMENT "OLAP"
DISTRIBUTED BY HASH(`id`) BUCKETS 1
PROPERTIES (
"replication_allocation" = "tag.location.default: 1",
"in_memory" = "false",
"storage_format" = "V2"
);
```
this sql is ok
```sql
select *, array_map(x -> x+1, c_array) from array_test
```
but lambda expressions in `array_map` do not support field arguments

for example:
```sql
select *, array_map(x -> x+id, c_array) from array_test
```

This pr solves two problems
1. support column arg in lambda expression
2. prevent high memory usage due to large block data

Related PR: #xxx
let lambda expression support refer outer slot
Co-authored-by: garenshi <garenshi@tencent.com>
@qzsee qzsee requested a review from dataroaring as a code owner March 24, 2025 07:01
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333
Copy link
Contributor

zhangstar333 commented Apr 22, 2025

run buildall

@morrySnow morrySnow changed the title [opt](lambda) let lambda expression support refer outer slot (#44365) branch-3.0: [opt](lambda) let lambda expression support refer outer slot #44365 Apr 22, 2025
zhangstar333
zhangstar333 reviewed Apr 22, 2025
View reviewed changes
morrySnow
morrySnow previously approved these changes Apr 22, 2025
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Apr 22, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 22, 2025

PR approved by anyone and no changes requested.

@dqz123
Copy link

dqz123 commented Apr 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

TPC-H: Total hot run time: 40103 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44e53dfcceb4f3dc1c665565d1f2fa86f1557230, data reload: false

------ Round 1 ----------------------------------
q1	17585	6801	6615	6615
q2	2084	185	176	176
q3	10620	1080	1192	1080
q4	10532	783	762	762
q5	7761	2856	2815	2815
q6	216	134	136	134
q7	977	616	604	604
q8	9366	1962	2076	1962
q9	6589	6389	6407	6389
q10	7051	2257	2322	2257
q11	468	261	257	257
q12	405	212	212	212
q13	17819	2996	2982	2982
q14	248	215	209	209
q15	506	459	464	459
q16	664	600	596	596
q17	990	597	530	530
q18	7383	6692	6616	6616
q19	1397	1050	1034	1034
q20	482	201	206	201
q21	4014	3389	3232	3232
q22	1127	990	981	981
Total cold run time: 108284 ms
Total hot run time: 40103 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6602	6560	6567	6560
q2	341	239	231	231
q3	2908	2788	2946	2788
q4	2046	1776	1725	1725
q5	5790	5779	5753	5753
q6	209	128	128	128
q7	2214	1838	1791	1791
q8	3385	3587	3524	3524
q9	8917	8838	8852	8838
q10	3552	3521	3532	3521
q11	582	499	495	495
q12	805	620	654	620
q13	9456	3148	3147	3147
q14	290	268	271	268
q15	511	472	453	453
q16	692	641	634	634
q17	1884	1615	1622	1615
q18	8376	7885	7813	7813
q19	1674	1594	1572	1572
q20	2049	1794	1870	1794
q21	5528	5342	5591	5342
q22	1135	1043	1032	1032
Total cold run time: 68946 ms
Total hot run time: 59644 ms

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

TPC-DS: Total hot run time: 196787 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44e53dfcceb4f3dc1c665565d1f2fa86f1557230, data reload: false

query1	1298	907	890	890
query2	6258	2125	2067	2067
query3	10874	4200	4072	4072
query4	61629	29289	23266	23266
query5	5202	476	458	458
query6	415	179	181	179
query7	5421	314	304	304
query8	314	233	225	225
query9	8368	2620	2573	2573
query10	452	274	258	258
query11	17556	15200	15711	15200
query12	167	105	103	103
query13	1439	442	438	438
query14	10551	7107	7319	7107
query15	200	180	173	173
query16	7058	476	476	476
query17	1139	582	609	582
query18	1856	324	319	319
query19	211	172	159	159
query20	111	108	105	105
query21	202	101	105	101
query22	4709	4570	4745	4570
query23	34234	34215	33657	33657
query24	6189	2926	2877	2877
query25	521	397	399	397
query26	684	166	162	162
query27	2264	348	361	348
query28	4241	2443	2430	2430
query29	688	455	449	449
query30	235	166	158	158
query31	983	839	851	839
query32	63	60	54	54
query33	387	301	285	285
query34	909	510	514	510
query35	851	731	710	710
query36	1054	947	939	939
query37	114	68	68	68
query38	4046	3981	3885	3885
query39	1684	1485	1425	1425
query40	209	97	97	97
query41	52	50	49	49
query42	113	100	105	100
query43	523	496	482	482
query44	1142	813	833	813
query45	184	167	170	167
query46	1154	728	740	728
query47	2096	1963	1955	1955
query48	484	387	390	387
query49	735	372	394	372
query50	852	430	420	420
query51	7151	7374	7320	7320
query52	97	91	91	91
query53	259	195	184	184
query54	579	456	463	456
query55	77	78	80	78
query56	256	260	257	257
query57	1260	1181	1177	1177
query58	223	214	239	214
query59	3350	2991	3077	2991
query60	280	268	260	260
query61	107	105	104	104
query62	791	691	699	691
query63	234	196	187	187
query64	1645	688	668	668
query65	3299	3196	3159	3159
query66	701	300	300	300
query67	15786	15654	15597	15597
query68	4071	588	577	577
query69	416	259	260	259
query70	1189	1132	1083	1083
query71	351	263	243	243
query72	6295	4054	4007	4007
query73	748	342	360	342
query74	10088	9176	9327	9176
query75	3352	2644	2659	2644
query76	2077	1005	1133	1005
query77	491	278	273	273
query78	10481	9690	9611	9611
query79	1547	618	603	603
query80	910	488	436	436
query81	523	241	240	240
query82	1263	92	87	87
query83	269	152	148	148
query84	292	86	85	85
query85	983	303	283	283
query86	333	300	285	285
query87	4435	4271	4283	4271
query88	3861	2382	2357	2357
query89	411	294	299	294
query90	1987	183	186	183
query91	182	150	148	148
query92	60	49	49	49
query93	1975	559	553	553
query94	752	299	296	296
query95	352	261	251	251
query96	618	294	283	283
query97	3305	3123	3111	3111
query98	209	207	205	205
query99	1536	1299	1277	1277
Total cold run time: 313578 ms
Total hot run time: 196787 ms

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

ClickBench: Total hot run time: 31.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44e53dfcceb4f3dc1c665565d1f2fa86f1557230, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.66	0.10	0.10
query5	0.50	0.52	0.52
query6	1.14	0.72	0.74
query7	0.03	0.02	0.02
query8	0.06	0.03	0.03
query9	0.57	0.51	0.50
query10	0.54	0.54	0.58
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.72	2.77	2.71
query15	0.88	0.83	0.81
query16	0.38	0.37	0.37
query17	1.04	1.02	0.96
query18	0.23	0.21	0.22
query19	1.85	1.78	2.03
query20	0.02	0.01	0.01
query21	15.36	0.59	0.59
query22	2.59	2.68	1.69
query23	16.96	1.35	0.79
query24	3.27	2.02	0.66
query25	0.17	0.15	0.18
query26	0.47	0.14	0.13
query27	0.05	0.04	0.05
query28	9.54	0.48	0.52
query29	12.57	3.23	3.27
query30	0.25	0.06	0.07
query31	2.86	0.40	0.40
query32	3.23	0.45	0.46
query33	2.95	3.02	3.00
query34	17.15	4.46	4.48
query35	4.54	4.57	4.55
query36	0.69	0.49	0.48
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.96 s
Total hot run time: 31.67 s

@doris-robot
Copy link

doris-robot commented Apr 23, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/155) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 40.20% (10555/26255)
Line Coverage 30.98% (89316/288341)
Region Coverage 30.10% (46056/153028)
Branch Coverage 26.60% (23543/88502)

@dataroaring dataroaring closed this May 7, 2025
@dataroaring dataroaring reopened this May 7, 2025
dataroaring
dataroaring approved these changes May 7, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit d87bf7a into apache:branch-3.0 May 7, 2025
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zhangstar333 zhangstar333 zhangstar333 left review comments

@dataroaring dataroaring dataroaring approved these changes

@morrySnow morrySnow morrySnow left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@qzsee @hello-stephen @zhangstar333 @dqz123 @doris-robot @dataroaring @morrySnow