Skip to content

Conversation

@xinyiZzz
Copy link
Contributor

Proposed changes

arrow 13.0.0 -> 15.0.2

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@xinyiZzz xinyiZzz force-pushed the 20240325_fix_arrow_flight_be branch from 5f266d4 to eb7a214 Compare March 25, 2024 16:37
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.26% (8737/24777)
Line Coverage: 27.05% (71546/264506)
Region Coverage: 26.28% (37105/141175)
Branch Coverage: 23.17% (18971/81872)
Coverage Report: http://coverage.selectdb-in.cc/coverage/eb7a2140976f6ee22a58a832c3c3620fa22c8883_eb7a2140976f6ee22a58a832c3c3620fa22c8883/report/index.html

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.26% (8736/24777)
Line Coverage: 27.04% (71534/264518)
Region Coverage: 26.28% (37112/141193)
Branch Coverage: 23.17% (18973/81874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bb300df7ec9b8edc2c41528c7daf1b86c3bf267d_bb300df7ec9b8edc2c41528c7daf1b86c3bf267d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 37736 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb300df7ec9b8edc2c41528c7daf1b86c3bf267d, data reload: false

------ Round 1 ----------------------------------
q1	17643	4201	4095	4095
q2	2107	158	151	151
q3	10586	1099	1200	1099
q4	10227	772	775	772
q5	7447	2997	2959	2959
q6	205	125	122	122
q7	1009	572	571	571
q8	9321	1989	1988	1988
q9	7198	6570	6536	6536
q10	8427	3480	3547	3480
q11	430	229	213	213
q12	375	202	193	193
q13	17789	2851	2831	2831
q14	239	205	214	205
q15	502	467	453	453
q16	497	376	375	375
q17	945	533	612	533
q18	7015	6421	6381	6381
q19	2143	1437	1497	1437
q20	547	270	241	241
q21	3542	2912	2807	2807
q22	340	294	306	294
Total cold run time: 108534 ms
Total hot run time: 37736 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4121	4098	4058	4058
q2	319	228	226	226
q3	2973	2830	2827	2827
q4	1842	1560	1582	1560
q5	5282	5291	5310	5291
q6	192	113	115	113
q7	2223	1867	1882	1867
q8	3117	3280	3259	3259
q9	8669	8661	8657	8657
q10	3774	3782	3772	3772
q11	550	435	459	435
q12	696	577	541	541
q13	16935	2832	2800	2800
q14	276	252	250	250
q15	493	464	447	447
q16	481	431	425	425
q17	1723	1488	1461	1461
q18	7433	7177	6968	6968
q19	1613	1530	1550	1530
q20	1904	1744	1695	1695
q21	4744	4636	4674	4636
q22	505	454	458	454
Total cold run time: 69865 ms
Total hot run time: 53272 ms

@xinyiZzz xinyiZzz force-pushed the 20240325_fix_arrow_flight_be branch from bb300df to 48803d7 Compare March 26, 2024 07:57
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38602 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48803d794a8f5e94ec49830065d970311a6110ef, data reload: false

------ Round 1 ----------------------------------
q1	17803	4408	4201	4201
q2	2434	166	157	157
q3	11901	1168	1239	1168
q4	10756	820	844	820
q5	7669	3129	3050	3050
q6	206	126	126	126
q7	1072	596	592	592
q8	9442	2068	2035	2035
q9	7385	6707	6730	6707
q10	9531	3553	3584	3553
q11	434	222	224	222
q12	374	207	199	199
q13	17798	2895	2879	2879
q14	226	208	212	208
q15	511	460	466	460
q16	507	373	381	373
q17	963	541	617	541
q18	7365	6517	6424	6424
q19	1584	1498	1408	1408
q20	531	253	253	253
q21	3536	3023	2926	2926
q22	354	300	304	300
Total cold run time: 112382 ms
Total hot run time: 38602 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4143	4063	4168	4063
q2	329	257	236	236
q3	3002	2869	2853	2853
q4	1829	1507	1527	1507
q5	5291	5350	5347	5347
q6	199	119	117	117
q7	2259	1889	1865	1865
q8	3186	3307	3271	3271
q9	8705	8638	8633	8633
q10	3804	3780	3764	3764
q11	544	437	441	437
q12	727	541	574	541
q13	16888	2873	2887	2873
q14	271	255	253	253
q15	494	453	447	447
q16	468	426	431	426
q17	1747	1501	1455	1455
q18	7479	7291	7124	7124
q19	1608	1576	1529	1529
q20	1900	1735	1718	1718
q21	4804	4616	4699	4616
q22	555	428	458	428
Total cold run time: 70232 ms
Total hot run time: 53503 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.26% (8737/24777)
Line Coverage: 27.05% (71553/264518)
Region Coverage: 26.28% (37110/141193)
Branch Coverage: 23.18% (18978/81874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/48803d794a8f5e94ec49830065d970311a6110ef_48803d794a8f5e94ec49830065d970311a6110ef/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 181977 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48803d794a8f5e94ec49830065d970311a6110ef, data reload: false

query1	950	390	373	373
query2	6535	1941	1833	1833
query3	6716	217	223	217
query4	31940	21447	21384	21384
query5	4266	397	400	397
query6	263	179	182	179
query7	4628	311	303	303
query8	229	174	179	174
query9	9335	2364	2373	2364
query10	571	255	258	255
query11	15562	14255	14272	14255
query12	140	97	91	91
query13	1637	418	414	414
query14	10176	7603	7987	7603
query15	290	202	208	202
query16	7766	266	271	266
query17	1925	580	543	543
query18	2005	295	288	288
query19	348	153	159	153
query20	96	91	86	86
query21	209	138	134	134
query22	5018	4857	4741	4741
query23	33425	32913	32444	32444
query24	11908	2829	2849	2829
query25	618	370	364	364
query26	1738	155	160	155
query27	3021	358	352	352
query28	7719	1867	1885	1867
query29	995	672	620	620
query30	312	164	157	157
query31	1006	727	719	719
query32	96	63	60	60
query33	774	254	263	254
query34	998	489	509	489
query35	846	637	635	635
query36	989	876	884	876
query37	133	66	68	66
query38	3540	3467	3464	3464
query39	1485	1461	1432	1432
query40	298	116	116	116
query41	51	48	48	48
query42	104	100	102	100
query43	471	446	445	445
query44	1209	730	715	715
query45	279	268	240	240
query46	1121	696	701	696
query47	1925	1833	1831	1831
query48	447	354	357	354
query49	1178	347	347	347
query50	774	384	383	383
query51	6987	6787	6869	6787
query52	107	90	96	90
query53	349	275	282	275
query54	325	246	240	240
query55	88	83	82	82
query56	261	232	227	227
query57	1239	1146	1129	1129
query58	243	217	206	206
query59	2897	2723	2576	2576
query60	267	254	252	252
query61	115	112	132	112
query62	663	458	451	451
query63	318	285	280	280
query64	6986	4209	4082	4082
query65	3200	3093	3074	3074
query66	1470	386	387	386
query67	15246	14801	14654	14654
query68	6945	524	517	517
query69	638	381	396	381
query70	1294	1207	1190	1190
query71	518	281	273	273
query72	6512	2718	2524	2524
query73	740	319	334	319
query74	7845	6460	6404	6404
query75	3390	2216	2205	2205
query76	4540	937	881	881
query77	651	263	253	253
query78	11312	10418	10279	10279
query79	9987	520	521	520
query80	1546	382	381	381
query81	516	239	222	222
query82	834	86	85	85
query83	210	150	145	145
query84	284	80	80	80
query85	1465	315	311	311
query86	463	312	291	291
query87	3790	3523	3523	3523
query88	5051	2375	2379	2375
query89	508	372	382	372
query90	1967	177	175	175
query91	173	131	137	131
query92	71	54	50	50
query93	6999	501	483	483
query94	1128	177	181	177
query95	426	329	336	329
query96	632	277	268	268
query97	2673	2482	2472	2472
query98	224	218	212	212
query99	1254	933	934	933
Total cold run time: 313412 ms
Total hot run time: 181977 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 48803d794a8f5e94ec49830065d970311a6110ef with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.9 seconds inserted 10000000 Rows, about 478K ops/s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 26, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Apr 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38641 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 38afe355c42535d7f8e549ff9e1fddfd9886283a, data reload: false

------ Round 1 ----------------------------------
q1	17605	4110	4036	4036
q2	2012	187	178	178
q3	10484	1289	1424	1289
q4	10215	916	1037	916
q5	7481	2977	2935	2935
q6	217	132	128	128
q7	1093	620	604	604
q8	9402	2067	2047	2047
q9	6711	6187	6133	6133
q10	8467	3498	3506	3498
q11	429	242	242	242
q12	384	217	211	211
q13	17779	2884	2909	2884
q14	272	241	242	241
q15	537	475	479	475
q16	498	395	382	382
q17	961	889	899	889
q18	7436	6546	6357	6357
q19	1618	1540	1528	1528
q20	606	313	301	301
q21	3566	3066	3093	3066
q22	355	301	316	301
Total cold run time: 108128 ms
Total hot run time: 38641 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4074	4024	4053	4024
q2	327	221	219	219
q3	2934	2962	2951	2951
q4	1846	1830	1820	1820
q5	5241	5202	5206	5202
q6	208	125	124	124
q7	2235	1836	1813	1813
q8	3230	3294	3286	3286
q9	8462	8430	8446	8430
q10	3721	3956	4025	3956
q11	563	475	479	475
q12	753	606	590	590
q13	14942	3087	3074	3074
q14	322	280	285	280
q15	531	505	489	489
q16	523	464	462	462
q17	1785	1726	1704	1704
q18	8246	7712	7785	7712
q19	1735	1698	1683	1683
q20	1994	1823	1856	1823
q21	5196	4954	4927	4927
q22	517	423	425	423
Total cold run time: 69385 ms
Total hot run time: 55467 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181711 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 38afe355c42535d7f8e549ff9e1fddfd9886283a, data reload: false

query1	881	1122	354	354
query2	6336	1967	1954	1954
query3	6650	215	215	215
query4	24781	21633	21578	21578
query5	4154	401	401	401
query6	293	184	180	180
query7	4609	312	296	296
query8	231	176	165	165
query9	8460	2281	2273	2273
query10	455	263	267	263
query11	15060	14482	14448	14448
query12	149	95	94	94
query13	1631	391	386	386
query14	8488	7062	6904	6904
query15	222	175	180	175
query16	6851	288	273	273
query17	981	590	570	570
query18	1864	304	277	277
query19	211	166	166	166
query20	99	91	96	91
query21	206	127	133	127
query22	4934	4754	4776	4754
query23	33421	32704	32809	32704
query24	12562	3218	3202	3202
query25	743	429	435	429
query26	1692	175	164	164
query27	3148	396	384	384
query28	6914	1942	1896	1896
query29	1210	630	612	612
query30	309	158	159	158
query31	1021	768	785	768
query32	110	69	67	67
query33	702	251	254	251
query34	1090	508	542	508
query35	863	747	729	729
query36	1012	844	863	844
query37	142	85	82	82
query38	3774	3528	3608	3528
query39	1650	1594	1608	1594
query40	234	149	153	149
query41	53	52	50	50
query42	112	103	110	103
query43	454	423	416	416
query44	1144	724	724	724
query45	285	271	254	254
query46	1101	838	853	838
query47	1994	1908	1911	1908
query48	383	317	321	317
query49	955	382	356	356
query50	816	421	412	412
query51	6931	6763	6875	6763
query52	115	101	98	98
query53	364	290	297	290
query54	296	237	239	237
query55	89	80	77	77
query56	252	224	231	224
query57	1255	1174	1171	1171
query58	260	227	230	227
query59	2693	2334	2515	2334
query60	262	238	243	238
query61	93	94	103	94
query62	657	470	451	451
query63	304	284	286	284
query64	5915	3384	3186	3186
query65	3023	3013	3032	3013
query66	1041	330	320	320
query67	15368	15075	14925	14925
query68	7482	569	582	569
query69	576	342	339	339
query70	1233	1105	1105	1105
query71	519	273	273	273
query72	6463	2594	2383	2383
query73	790	337	331	331
query74	6629	6286	6382	6286
query75	3564	2323	2295	2295
query76	5213	1084	1196	1084
query77	567	251	248	248
query78	10865	10068	10170	10068
query79	8422	525	525	525
query80	1209	423	416	416
query81	483	219	221	219
query82	780	106	102	102
query83	194	161	163	161
query84	274	89	88	88
query85	1384	286	286	286
query86	450	284	311	284
query87	3667	3466	3454	3454
query88	4144	2367	2374	2367
query89	559	367	372	367
query90	1923	181	182	181
query91	133	103	104	103
query92	62	49	54	49
query93	7004	528	539	528
query94	1002	198	193	193
query95	428	341	332	332
query96	635	272	272	272
query97	2663	2470	2454	2454
query98	227	215	209	209
query99	1276	817	820	817
Total cold run time: 295178 ms
Total hot run time: 181711 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 38afe355c42535d7f8e549ff9e1fddfd9886283a, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.05	0.05
query4	1.66	0.06	0.06
query5	0.47	0.48	0.49
query6	1.12	0.65	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.50
query10	0.58	0.57	0.55
query11	0.15	0.11	0.10
query12	0.14	0.11	0.12
query13	0.62	0.60	0.59
query14	0.76	0.79	0.80
query15	0.85	0.85	0.84
query16	0.34	0.34	0.34
query17	0.95	0.97	1.00
query18	0.24	0.26	0.27
query19	1.76	1.67	1.73
query20	0.02	0.01	0.01
query21	15.40	0.75	0.73
query22	2.92	5.64	1.73
query23	17.70	1.33	1.02
query24	1.36	0.43	0.22
query25	0.13	0.09	0.09
query26	0.29	0.18	0.18
query27	0.09	0.08	0.08
query28	13.46	0.94	0.94
query29	12.60	3.22	3.22
query30	0.27	0.06	0.05
query31	2.86	0.38	0.39
query32	3.29	0.47	0.47
query33	2.80	2.89	2.86
query34	15.49	4.32	4.31
query35	4.34	4.33	4.38
query36	0.67	0.47	0.47
query37	0.22	0.18	0.17
query38	0.18	0.16	0.16
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 105.11 s
Total hot run time: 29.83 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 38afe355c42535d7f8e549ff9e1fddfd9886283a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.0 seconds inserted 10000000 Rows, about 625K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.57% (8849/24877)
Line Coverage: 27.28% (72529/265838)
Region Coverage: 26.48% (37540/141752)
Branch Coverage: 23.28% (19136/82188)
Coverage Report: http://coverage.selectdb-in.cc/coverage/38afe355c42535d7f8e549ff9e1fddfd9886283a_38afe355c42535d7f8e549ff9e1fddfd9886283a/report/index.html

Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit fde3fbf into apache:master Apr 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants