Skip to content

[branch-2.0][Improve](tvf)avro-jni add projection push down #29113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
DongLiang-0 wants to merge 4 commits into from
Closed

[branch-2.0][Improve](tvf)avro-jni add projection push down #29113

DongLiang-0 wants to merge 4 commits into from

Conversation

@DongLiang-0
Copy link
Contributor

@DongLiang-0 DongLiang-0 commented Dec 27, 2023

Proposed changes

Related pr:
#26885
#27933
#28709

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@DongLiang-0
Copy link
Contributor Author

DongLiang-0 commented Dec 27, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Dec 27, 2023

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Dec 27, 2023

TeamCity be ut coverage result:
Function Coverage: 37.79% (8005/21182)
Line Coverage: 29.48% (65070/220736)
Region Coverage: 28.94% (33455/115616)
Branch Coverage: 24.80% (17165/69202)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2c637a95659f62230718d8501677dd729e646d5d_2c637a95659f62230718d8501677dd729e646d5d/report/index.html

@doris-robot
Copy link

doris-robot commented Dec 27, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 51.9 seconds
stream load tsv: 571 seconds loaded 74807831229 Bytes, about 124 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.6 seconds inserted 10000000 Rows, about 337K ops/s
storage size: 17162255460 Bytes

@xiaokang xiaokang requested a review from morningman December 28, 2023 02:24
@DongLiang-0
Copy link
Contributor Author

DongLiang-0 commented Jan 30, 2024

run buildall

@xiaokang
Copy link
Contributor

xiaokang commented Jan 31, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jan 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jan 31, 2024

TPC-H: Total hot run time: 50227 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cded9218bd5fd4a22a72fed2df2a19d422718aab, data reload: false

------ Round 1 ----------------------------------
q1	17925	4362	4316	4316
q2	2027	153	139	139
q3	10348	1900	1943	1900
q4	10096	1247	1336	1247
q5	8680	3979	4008	3979
q6	232	124	121	121
q7	2006	1590	1609	1590
q8	9327	2717	2727	2717
q9	10926	10729	10540	10540
q10	8655	3533	3548	3533
q11	415	227	240	227
q12	468	293	299	293
q13	18362	4011	4071	4011
q14	352	314	321	314
q15	504	461	460	460
q16	694	597	595	595
q17	1134	989	1023	989
q18	7298	7040	6947	6947
q19	1668	1545	1540	1540
q20	546	293	304	293
q21	4484	4134	4093	4093
q22	501	399	383	383
Total cold run time: 116648 ms
Total hot run time: 50227 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4329	4316	4316
q2	321	218	221	218
q3	4197	4174	4165	4165
q4	2755	2741	2750	2741
q5	7327	7146	7279	7146
q6	234	119	115	115
q7	3208	2869	2857	2857
q8	4359	4486	4474	4474
q9	17309	17174	17133	17133
q10	4260	4277	4252	4252
q11	775	685	698	685
q12	1041	857	839	839
q13	6737	3750	3728	3728
q14	448	417	420	417
q15	507	454	453	453
q16	747	693	700	693
q17	3918	3863	3848	3848
q18	8838	8718	8935	8718
q19	1729	1716	1650	1650
q20	2391	2170	2147	2147
q21	8574	8557	8535	8535
q22	1054	934	935	934
Total cold run time: 85037 ms
Total hot run time: 80064 ms

@doris-robot
Copy link

doris-robot commented Jan 31, 2024

TPC-DS: Total hot run time: 241175 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cded9218bd5fd4a22a72fed2df2a19d422718aab, data reload: false

query1	926	398	382	382
query2	6536	2291	2043	2043
query3	6915	209	202	202
query4	20100	18041	17944	17944
query5	19738	6274	6218	6218
query6	277	217	218	217
query7	4155	291	294	291
query8	265	255	258	255
query9	3071	2662	2596	2596
query10	396	288	313	288
query11	11340	10718	10817	10718
query12	121	72	76	72
query13	5652	638	618	618
query14	17491	13238	13345	13238
query15	393	238	225	225
query16	6462	260	260	260
query17	3199	867	868	867
query18	2258	397	404	397
query19	202	143	146	143
query20	78	73	77	73
query21	189	97	89	89
query22	5241	5139	5176	5139
query23	32484	31810	32172	31810
query24	7010	6515	6468	6468
query25	520	440	429	429
query26	522	157	161	157
query27	1870	297	294	294
query28	6135	2233	2199	2199
query29	2958	2727	2768	2727
query30	237	160	161	160
query31	900	751	734	734
query32	61	58	56	56
query33	391	258	253	253
query34	847	467	476	467
query35	1108	936	908	908
query36	1781	1427	1676	1427
query37	95	66	55	55
query38	3105	2935	2870	2870
query39	1363	1326	1313	1313
query40	205	90	94	90
query41	36	31	31	31
query42	95	81	83	81
query43	653	622	603	603
query44	1093	714	713	713
query45	245	228	224	224
query46	1226	979	974	974
query47	1751	1630	1639	1630
query48	947	677	661	661
query49	598	368	352	352
query50	861	620	615	615
query51	5569	5512	5468	5468
query52	82	82	75	75
query53	439	324	317	317
query54	2408	2250	2263	2250
query55	86	78	82	78
query56	223	209	186	186
query57	1189	1092	1199	1092
query58	203	201	197	197
query59	3244	3298	3478	3298
query60	198	198	188	188
query61	86	87	86	86
query62	871	475	494	475
query63	468	334	337	334
query64	2004	1320	1311	1311
query65	3708	3610	3622	3610
query66	800	362	363	362
query67	18106	17377	16381	16381
query68	8130	617	678	617
query69	547	347	325	325
query70	2085	1630	1759	1630
query71	366	297	297	297
query72	4486	3289	3343	3289
query73	728	321	317	317
query74	6282	5816	5870	5816
query75	4206	3486	3513	3486
query76	4611	1177	1189	1177
query77	574	252	252	252
query78	32169	50021	50004	50004
query79	15415	624	656	624
query80	4574	371	380	371
query81	564	234	227	227
query82	1188	95	94	94
query83	320	135	136	135
query84	258	68	70	68
query85	2306	272	276	272
query86	455	368	370	368
query87	3234	2971	3017	2971
query88	7240	2362	2334	2334
query89	437	285	280	280
query90	2530	214	195	195
query91	160	116	119	116
query92	66	52	52	52
query93	6492	566	603	566
query94	1665	207	204	204
query95	1106	1057	1055	1055
query96	637	338	329	329
query97	6510	6347	6402	6347
query98	183	167	168	167
query99	3652	930	906	906
Total cold run time: 346962 ms
Total hot run time: 241175 ms

@doris-robot
Copy link

doris-robot commented Jan 31, 2024

ClickBench: Total hot run time: 30.94 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cded9218bd5fd4a22a72fed2df2a19d422718aab, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.81	0.07	0.06
query5	0.53	0.53	0.53
query6	1.24	0.61	0.63
query7	0.02	0.01	0.00
query8	0.03	0.02	0.02
query9	0.53	0.51	0.48
query10	0.56	0.54	0.53
query11	0.11	0.09	0.08
query12	0.10	0.09	0.09
query13	0.63	0.62	0.61
query14	0.78	0.79	0.79
query15	0.78	0.78	0.76
query16	0.36	0.36	0.37
query17	1.01	1.01	1.01
query18	0.22	0.26	0.23
query19	1.92	1.90	1.75
query20	0.01	0.01	0.01
query21	15.50	0.55	0.56
query22	2.28	1.68	1.46
query23	17.23	0.95	1.07
query24	3.86	3.19	1.23
query25	1.63	0.11	0.11
query26	0.25	0.14	0.14
query27	0.11	0.11	0.10
query28	6.75	0.72	0.71
query29	12.67	2.35	2.39
query30	0.54	0.53	0.52
query31	2.80	0.38	0.39
query32	3.39	0.52	0.50
query33	3.09	3.11	3.03
query34	15.26	4.79	4.78
query35	4.86	4.85	4.87
query36	1.07	1.01	1.02
query37	0.05	0.05	0.04
query38	0.03	0.02	0.02
query39	0.01	0.02	0.02
query40	0.15	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 102.6 s
Total hot run time: 30.94 s

@doris-robot
Copy link

doris-robot commented Jan 31, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cded9218bd5fd4a22a72fed2df2a19d422718aab with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@doris-robot
Copy link

doris-robot commented Jan 31, 2024

TeamCity be ut coverage result:
Function Coverage: 37.84% (8034/21232)
Line Coverage: 29.55% (65480/221616)
Region Coverage: 28.96% (33629/116106)
Branch Coverage: 24.83% (17256/69510)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cded9218bd5fd4a22a72fed2df2a19d422718aab_cded9218bd5fd4a22a72fed2df2a19d422718aab/report/index.html

@morningman morningman self-assigned this Mar 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman Awaiting requested review from morningman

Assignees

@morningman morningman

Labels

dev/2.0.5 kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@DongLiang-0 @doris-robot @xiaokang @morningman