Skip to content

[fix](planner) remove and retain input slot for aggregate slot which is not materialized #33033

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
nextdreamblue merged 1 commit into from
Mar 29, 2024
Merged

[fix](planner) remove and retain input slot for aggregate slot which is not materialized #33033

nextdreamblue merged 1 commit into from
Mar 29, 2024

Conversation

@nextdreamblue
Copy link
Contributor

@nextdreamblue nextdreamblue commented Mar 29, 2024

Proposed changes

fix bug introduced by #32092。do not remove input slot which need retain

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@nextdreamblue
[fix](planner) remove and retain input slot for aggregate slot which …
379db69 
…is not materialized

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@doris-robot
Copy link

doris-robot commented Mar 29, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@nextdreamblue
Copy link
Contributor Author

nextdreamblue commented Mar 29, 2024

run buildall

cambyzju
cambyzju approved these changes Mar 29, 2024
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 29, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 29, 2024

PR approved by anyone and no changes requested.

lide-reed
lide-reed approved these changes Mar 29, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Mar 29, 2024

TPC-H: Total hot run time: 37982 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 379db69f4432f7d74d25ba1c35206e7baa4282b6, data reload: false

------ Round 1 ----------------------------------
q1	17607	4383	4197	4197
q2	2083	194	182	182
q3	10468	1114	1125	1114
q4	10203	782	731	731
q5	7433	2759	2782	2759
q6	213	131	131	131
q7	1016	590	594	590
q8	9220	2008	2151	2008
q9	7416	6524	6481	6481
q10	8495	3466	3511	3466
q11	443	243	227	227
q12	432	217	209	209
q13	18947	2934	2931	2931
q14	261	227	243	227
q15	526	484	464	464
q16	509	385	376	376
q17	949	602	562	562
q18	7270	6601	6638	6601
q19	4425	1445	1438	1438
q20	651	306	289	289
q21	3355	2695	2874	2695
q22	367	305	304	304
Total cold run time: 112289 ms
Total hot run time: 37982 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4335	4181	4155	4155
q2	378	258	275	258
q3	2965	2645	2711	2645
q4	1844	1513	1493	1493
q5	5279	5214	5224	5214
q6	210	123	124	123
q7	2243	1859	1910	1859
q8	3207	3278	3292	3278
q9	8479	8494	8509	8494
q10	3963	3781	3798	3781
q11	618	521	497	497
q12	833	687	674	674
q13	17895	3248	3117	3117
q14	354	289	289	289
q15	524	478	490	478
q16	502	488	466	466
q17	1815	1457	1452	1452
q18	8275	8046	7740	7740
q19	1615	1537	1588	1537
q20	2033	1839	1818	1818
q21	10654	4842	4861	4842
q22	541	475	451	451
Total cold run time: 78562 ms
Total hot run time: 54661 ms

@doris-robot
Copy link

doris-robot commented Mar 29, 2024

TPC-DS: Total hot run time: 182118 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 379db69f4432f7d74d25ba1c35206e7baa4282b6, data reload: false

query1	896	363	1126	363
query2	6821	2068	1940	1940
query3	6652	217	219	217
query4	32822	21692	21410	21410
query5	4113	388	401	388
query6	228	181	170	170
query7	3983	290	287	287
query8	217	173	175	173
query9	5622	2269	2236	2236
query10	365	233	239	233
query11	14669	14238	14281	14238
query12	141	88	87	87
query13	955	371	376	371
query14	9023	6961	7020	6961
query15	213	175	190	175
query16	7322	268	273	268
query17	1477	594	586	586
query18	1894	293	277	277
query19	195	156	159	156
query20	92	90	91	90
query21	197	131	139	131
query22	5071	4879	4880	4879
query23	33301	32850	32803	32803
query24	8367	2897	2932	2897
query25	580	430	409	409
query26	727	166	162	162
query27	2976	381	358	358
query28	5804	1921	1933	1921
query29	895	651	647	647
query30	301	162	161	161
query31	933	761	747	747
query32	69	58	57	57
query33	499	258	257	257
query34	903	483	514	483
query35	805	708	724	708
query36	1057	908	928	908
query37	107	74	71	71
query38	3740	3666	3630	3630
query39	1649	1622	1624	1622
query40	180	126	126	126
query41	48	47	45	45
query42	110	97	102	97
query43	505	455	443	443
query44	1114	733	756	733
query45	280	286	264	264
query46	1110	724	747	724
query47	2045	1967	2046	1967
query48	395	308	308	308
query49	864	358	357	357
query50	794	403	395	395
query51	6859	6835	6764	6764
query52	100	91	93	91
query53	336	273	278	273
query54	247	221	217	217
query55	79	76	75	75
query56	247	226	226	226
query57	1260	1133	1133	1133
query58	223	199	201	199
query59	2827	2613	2574	2574
query60	247	225	232	225
query61	94	92	89	89
query62	576	446	438	438
query63	303	273	277	273
query64	4796	4153	4096	4096
query65	3074	2989	3013	2989
query66	762	339	334	334
query67	15706	14900	15076	14900
query68	7815	538	536	536
query69	586	324	311	311
query70	1204	1209	1101	1101
query71	494	268	282	268
query72	6617	2634	2451	2451
query73	832	323	321	321
query74	7243	6349	6426	6349
query75	3388	2220	2382	2220
query76	4535	920	1053	920
query77	600	258	257	257
query78	10888	10243	10143	10143
query79	9837	517	524	517
query80	1944	412	424	412
query81	530	219	217	217
query82	788	91	87	87
query83	209	167	165	165
query84	251	84	82	82
query85	971	271	268	268
query86	404	272	286	272
query87	3733	3546	3558	3546
query88	5752	2406	2393	2393
query89	505	373	374	373
query90	1992	175	171	171
query91	122	100	98	98
query92	61	44	46	44
query93	5673	510	502	502
query94	1246	185	178	178
query95	408	325	311	311
query96	579	274	264	264
query97	2677	2459	2529	2459
query98	237	222	218	218
query99	1183	832	846	832
Total cold run time: 294166 ms
Total hot run time: 182118 ms

@starocean999
Copy link
Contributor

starocean999 commented Mar 29, 2024

LGTM

@doris-robot
Copy link

doris-robot commented Mar 29, 2024

ClickBench: Total hot run time: 29.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 379db69f4432f7d74d25ba1c35206e7baa4282b6, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.10	0.10
query5	0.52	0.48	0.49
query6	1.12	0.65	0.65
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.55	0.55	0.55
query11	0.15	0.09	0.11
query12	0.14	0.12	0.10
query13	0.59	0.58	0.57
query14	0.78	0.76	0.80
query15	0.84	0.81	0.81
query16	0.37	0.36	0.37
query17	1.01	0.99	1.03
query18	0.22	0.25	0.23
query19	1.85	1.65	1.68
query20	0.02	0.01	0.01
query21	15.74	0.66	0.66
query22	3.22	3.04	1.87
query23	16.95	0.94	0.88
query24	1.14	0.23	0.23
query25	0.08	0.07	0.09
query26	0.24	0.17	0.17
query27	0.08	0.08	0.07
query28	14.10	0.94	0.92
query29	12.56	3.12	3.22
query30	0.28	0.10	0.08
query31	2.79	0.39	0.37
query32	3.29	0.45	0.45
query33	2.77	2.82	2.81
query34	16.75	4.41	4.41
query35	4.52	4.48	4.46
query36	0.59	0.47	0.47
query37	0.19	0.16	0.16
query38	0.15	0.13	0.14
query39	0.05	0.03	0.03
query40	0.19	0.14	0.14
query41	0.09	0.04	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 106.64 s
Total hot run time: 29.64 s

@doris-robot
Copy link

doris-robot commented Mar 29, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 379db69f4432f7d74d25ba1c35206e7baa4282b6 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@nextdreamblue nextdreamblue mentioned this pull request Mar 29, 2024
Merged
@nextdreamblue nextdreamblue merged commit aeeb1cb into apache:master Mar 29, 2024
@nextdreamblue nextdreamblue mentioned this pull request Mar 30, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 1, 2024
@nextdreamblue @Doris-Extras
[fix](planner) remove and retain input slot for aggregate slot which …
e3066dd 
…is not materialized (#33033)

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
yiguolei pushed a commit that referenced this pull request Apr 10, 2024
@nextdreamblue @Doris-Extras
[fix](planner) remove and retain input slot for aggregate slot which …
c5ab7ca 
…is not materialized (#33033)

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

@cambyzju cambyzju cambyzju approved these changes

@starocean999 starocean999 Awaiting requested review from starocean999

@morrySnow morrySnow Awaiting requested review from morrySnow

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.8-merged need_more_review reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@nextdreamblue @doris-robot @starocean999 @lide-reed @cambyzju @xiaokang @morrySnow