Skip to content

[minor](Nereids): remove useless stream filter() in Translator #33758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jackwener merged 2 commits into from
Apr 17, 2024
Merged

[minor](Nereids): remove useless stream filter() in Translator #33758

jackwener merged 2 commits into from
Apr 17, 2024

Conversation

@jackwener
Copy link
Member

@jackwener jackwener commented Apr 17, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 17, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 17, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 17, 2024

PR approved by anyone and no changes requested.

@jackwener
Copy link
Member Author

jackwener commented Apr 17, 2024

run buildall

@jackwener
Copy link
Member Author

jackwener commented Apr 17, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

TPC-H: Total hot run time: 38753 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1be132c489c1d26429c5c3f2fa9c6c63cfcf3366, data reload: false

------ Round 1 ----------------------------------
q1	17904	5473	4270	4270
q2	2015	193	183	183
q3	10528	1124	1184	1124
q4	10234	807	736	736
q5	7497	2748	2678	2678
q6	221	129	133	129
q7	1026	594	583	583
q8	9221	2087	2074	2074
q9	7402	6630	6542	6542
q10	8594	3543	3541	3541
q11	436	237	221	221
q12	468	227	220	220
q13	17776	2914	2958	2914
q14	276	225	243	225
q15	531	486	477	477
q16	522	391	383	383
q17	988	703	665	665
q18	7502	6901	6795	6795
q19	5503	1544	1543	1543
q20	629	329	301	301
q21	3613	2843	2898	2843
q22	360	306	309	306
Total cold run time: 113246 ms
Total hot run time: 38753 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4300	4196	4237	4196
q2	379	265	267	265
q3	3005	2698	2728	2698
q4	1904	1606	1596	1596
q5	5263	5298	5339	5298
q6	213	127	125	125
q7	2277	1863	1874	1863
q8	3221	3410	3386	3386
q9	8632	8596	8711	8596
q10	4044	3841	4038	3841
q11	602	514	495	495
q12	829	651	633	633
q13	16217	3206	3269	3206
q14	324	305	300	300
q15	536	493	495	493
q16	511	446	451	446
q17	1856	1544	1495	1495
q18	8299	8045	7952	7952
q19	1669	1574	1572	1572
q20	2052	1868	1818	1818
q21	7767	4956	4902	4902
q22	590	466	457	457
Total cold run time: 74490 ms
Total hot run time: 55633 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

TPC-DS: Total hot run time: 186569 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1be132c489c1d26429c5c3f2fa9c6c63cfcf3366, data reload: false

query1	905	373	362	362
query2	6650	2584	2416	2416
query3	6649	210	207	207
query4	23535	21478	21204	21204
query5	4177	410	405	405
query6	263	190	183	183
query7	4593	288	283	283
query8	229	183	178	178
query9	8511	2313	2309	2309
query10	426	246	268	246
query11	14705	14246	14214	14214
query12	137	93	85	85
query13	1640	373	365	365
query14	9954	7950	7959	7950
query15	256	187	178	178
query16	8121	272	264	264
query17	1914	592	602	592
query18	2071	278	263	263
query19	192	147	153	147
query20	89	82	84	82
query21	194	125	127	125
query22	5088	4822	4844	4822
query23	33482	33231	33323	33231
query24	6552	3023	3014	3014
query25	538	389	403	389
query26	692	156	152	152
query27	1975	356	367	356
query28	3872	2070	2060	2060
query29	858	632	631	631
query30	238	178	180	178
query31	975	761	782	761
query32	82	51	53	51
query33	586	243	264	243
query34	881	474	495	474
query35	811	709	698	698
query36	1062	929	925	925
query37	108	69	68	68
query38	3582	3323	3282	3282
query39	1579	1590	1582	1582
query40	165	127	127	127
query41	50	46	43	43
query42	106	101	98	98
query43	609	568	543	543
query44	1066	748	744	744
query45	291	269	263	263
query46	1099	747	728	728
query47	2033	1915	1942	1915
query48	385	301	305	301
query49	725	378	379	378
query50	804	404	397	397
query51	7042	6729	6764	6729
query52	103	91	91	91
query53	346	279	278	278
query54	280	233	226	226
query55	79	74	71	71
query56	242	225	213	213
query57	1233	1119	1107	1107
query58	214	194	214	194
query59	3360	3569	3316	3316
query60	253	243	238	238
query61	91	87	87	87
query62	519	434	436	434
query63	310	284	282	282
query64	4625	3821	3826	3821
query65	3044	3019	3043	3019
query66	752	345	328	328
query67	15275	14900	14809	14809
query68	6708	543	545	543
query69	553	307	303	303
query70	1294	1201	1210	1201
query71	1455	1267	1265	1265
query72	6664	2722	2542	2542
query73	718	315	318	315
query74	6935	6435	6395	6395
query75	3557	2660	2645	2645
query76	3783	991	899	899
query77	603	271	371	271
query78	11007	10233	10174	10174
query79	11071	542	509	509
query80	1786	433	417	417
query81	498	245	240	240
query82	827	90	91	90
query83	198	163	159	159
query84	257	85	78	78
query85	870	316	249	249
query86	418	303	318	303
query87	3484	3315	3249	3249
query88	5452	2305	2306	2305
query89	519	370	355	355
query90	2152	175	181	175
query91	119	93	95	93
query92	56	47	47	47
query93	7095	509	491	491
query94	1057	178	177	177
query95	1098	1087	1091	1087
query96	619	260	268	260
query97	3102	2910	2904	2904
query98	253	209	217	209
query99	1209	873	864	864
Total cold run time: 288690 ms
Total hot run time: 186569 ms

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

ClickBench: Total hot run time: 30.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1be132c489c1d26429c5c3f2fa9c6c63cfcf3366, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.09
query5	0.50	0.49	0.50
query6	1.46	0.72	0.74
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.50	0.48
query10	0.55	0.56	0.55
query11	0.16	0.11	0.12
query12	0.14	0.11	0.12
query13	0.59	0.58	0.58
query14	0.77	0.77	0.78
query15	0.82	0.80	0.80
query16	0.37	0.36	0.34
query17	0.98	1.02	0.97
query18	0.22	0.24	0.23
query19	1.87	1.74	1.69
query20	0.01	0.02	0.01
query21	15.42	0.65	0.65
query22	4.01	6.10	2.65
query23	18.24	1.37	1.23
query24	2.06	0.22	0.22
query25	0.15	0.07	0.07
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.30	1.00	0.98
query29	12.59	3.29	3.25
query30	0.26	0.07	0.06
query31	2.86	0.37	0.37
query32	3.30	0.47	0.45
query33	2.82	2.89	2.80
query34	16.99	4.40	4.38
query35	4.48	4.50	4.50
query36	0.66	0.46	0.47
query37	0.18	0.17	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.26 s
Total hot run time: 30.98 s

@doris-robot
Copy link

doris-robot commented Apr 17, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1be132c489c1d26429c5c3f2fa9c6c63cfcf3366 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@jackwener jackwener merged commit 5cd4901 into apache:master Apr 17, 2024
@jackwener jackwener deleted the translator branch April 17, 2024 10:35
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@jackwener @Doris-Extras
[minor](Nereids): remove useless stream filter() in Translator (#33758)
e9eccb0
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@jackwener @Doris-Extras
[minor](Nereids): remove useless stream filter() in Translator (#33758)
461561f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@keanji-x keanji-x keanji-x approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@jackwener @doris-robot @keanji-x @starocean999 @morrySnow