Skip to content

[fix](arrow-flight-sql) Fix arrow flight result sink #36827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xinyiZzz merged 2 commits into from
Jun 26, 2024
Merged

[fix](arrow-flight-sql) Fix arrow flight result sink #36827

xinyiZzz merged 2 commits into from
Jun 26, 2024

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Jun 25, 2024 •
edited
Loading

Proposed changes

  1. get arrow flight result schema use query id instead of instance id.
  2. get arrow flight result is a sync method, need wait for data ready and return result, introduced by [Improvement](sink) optimization for parallel result sink #36667.
    TODO, waiting for data will block pipeline, so use a request pool to save requests waiting for data.

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Jun 25, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@xinyiZzz xinyiZzz force-pushed the 20240620_arrow_flight_regressiontest branch from 1c8e198 to 22fb9da Compare June 25, 2024 12:50
@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Jun 25, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jun 25, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

TPC-H: Total hot run time: 39672 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 22fb9da16a3efc8a0ff259faeb1629c472c51d25, data reload: false

------ Round 1 ----------------------------------
q1	17601	4609	4259	4259
q2	2003	188	186	186
q3	10464	1210	1074	1074
q4	10179	774	790	774
q5	7543	2658	2579	2579
q6	217	131	131	131
q7	935	588	597	588
q8	9225	2041	2040	2040
q9	9117	6452	6479	6452
q10	8958	3698	3746	3698
q11	476	231	232	231
q12	504	220	221	220
q13	17775	2972	2955	2955
q14	253	217	224	217
q15	511	490	474	474
q16	484	382	376	376
q17	961	694	680	680
q18	8003	7384	7357	7357
q19	4578	1571	1513	1513
q20	667	322	325	322
q21	4960	3211	3948	3211
q22	390	335	335	335
Total cold run time: 115804 ms
Total hot run time: 39672 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4421	4267	4257	4257
q2	383	270	251	251
q3	2915	2739	2870	2739
q4	1936	1785	1692	1692
q5	5583	5496	5513	5496
q6	213	131	136	131
q7	2127	1800	1893	1800
q8	3271	3423	3387	3387
q9	9115	9083	9267	9083
q10	4066	3855	3680	3680
q11	598	496	510	496
q12	835	651	603	603
q13	17094	3145	3133	3133
q14	295	268	277	268
q15	530	477	500	477
q16	469	434	441	434
q17	1815	1551	1527	1527
q18	8045	8000	7746	7746
q19	1804	1506	1700	1506
q20	3024	1888	1879	1879
q21	5238	4953	4763	4763
q22	631	549	547	547
Total cold run time: 74408 ms
Total hot run time: 55895 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

TPC-DS: Total hot run time: 174373 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 22fb9da16a3efc8a0ff259faeb1629c472c51d25, data reload: false

query1	895	391	359	359
query2	6424	2337	2343	2337
query3	6628	201	198	198
query4	19343	17249	17305	17249
query5	3599	459	464	459
query6	250	168	172	168
query7	4586	306	293	293
query8	314	297	296	296
query9	8448	2434	2370	2370
query10	553	289	270	270
query11	10575	9981	9850	9850
query12	108	79	83	79
query13	1637	382	376	376
query14	10453	7766	7789	7766
query15	242	200	188	188
query16	7753	276	274	274
query17	1901	546	540	540
query18	1969	277	283	277
query19	201	195	148	148
query20	83	82	81	81
query21	202	130	128	128
query22	4193	4126	3866	3866
query23	33893	33684	33910	33684
query24	10973	2874	2919	2874
query25	628	379	370	370
query26	1035	156	158	156
query27	2314	338	330	330
query28	6487	2163	2150	2150
query29	885	669	639	639
query30	254	160	156	156
query31	984	764	744	744
query32	90	56	56	56
query33	793	286	280	280
query34	1033	487	478	478
query35	740	621	638	621
query36	1093	979	988	979
query37	140	74	74	74
query38	2941	2812	2836	2812
query39	921	827	850	827
query40	212	139	129	129
query41	59	51	48	48
query42	113	94	103	94
query43	582	557	552	552
query44	1205	752	723	723
query45	190	165	164	164
query46	1061	714	743	714
query47	1867	1760	1752	1752
query48	378	310	300	300
query49	842	413	408	408
query50	789	393	399	393
query51	6916	6869	6801	6801
query52	100	91	91	91
query53	362	285	292	285
query54	921	450	440	440
query55	76	75	79	75
query56	290	260	260	260
query57	1113	1037	1032	1032
query58	268	229	238	229
query59	3294	3259	3128	3128
query60	295	298	269	269
query61	90	94	88	88
query62	596	452	445	445
query63	322	293	290	290
query64	8742	2249	1754	1754
query65	3164	3076	3085	3076
query66	732	323	336	323
query67	15410	15045	15051	15045
query68	5555	545	556	545
query69	647	445	385	385
query70	1131	1119	1047	1047
query71	481	270	268	268
query72	7724	5575	5860	5575
query73	786	328	326	326
query74	5903	5530	5384	5384
query75	3869	2645	2671	2645
query76	3637	962	907	907
query77	665	309	301	301
query78	10326	9942	9675	9675
query79	2200	510	542	510
query80	2029	466	465	465
query81	560	222	220	220
query82	1402	109	108	108
query83	279	176	180	176
query84	267	87	84	84
query85	1302	292	277	277
query86	460	316	322	316
query87	3242	3059	3082	3059
query88	3812	2372	2349	2349
query89	473	384	398	384
query90	1784	190	190	190
query91	133	100	101	100
query92	59	49	53	49
query93	2201	521	515	515
query94	1036	188	191	188
query95	409	319	315	315
query96	606	268	262	262
query97	3216	3080	3024	3024
query98	231	202	196	196
query99	1273	830	818	818
Total cold run time: 273248 ms
Total hot run time: 174373 ms

@doris-robot
Copy link

doris-robot commented Jun 25, 2024

ClickBench: Total hot run time: 30.6 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 22fb9da16a3efc8a0ff259faeb1629c472c51d25, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.04	0.03
query3	0.22	0.05	0.04
query4	1.69	0.07	0.07
query5	0.51	0.52	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.54	0.51	0.48
query10	0.54	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.11	0.11
query13	0.60	0.59	0.59
query14	0.75	0.77	0.80
query15	0.82	0.82	0.81
query16	0.36	0.37	0.39
query17	1.01	1.03	1.02
query18	0.23	0.22	0.25
query19	1.90	1.72	1.68
query20	0.01	0.01	0.01
query21	15.46	0.65	0.65
query22	4.00	7.41	2.13
query23	18.29	1.35	1.20
query24	2.06	0.21	0.22
query25	0.16	0.09	0.08
query26	0.26	0.18	0.18
query27	0.07	0.08	0.09
query28	13.28	1.01	0.99
query29	12.64	3.34	3.26
query30	0.25	0.06	0.05
query31	2.86	0.38	0.38
query32	3.28	0.47	0.47
query33	2.90	2.85	2.95
query34	17.25	4.43	4.42
query35	4.48	4.51	4.44
query36	0.66	0.47	0.48
query37	0.18	0.15	0.15
query38	0.14	0.14	0.14
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.04	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.38 s
Total hot run time: 30.6 s

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Jun 26, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-H: Total hot run time: 39794 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8e900beda248701f9b7d7ec38d3f07063a6fd832, data reload: false

------ Round 1 ----------------------------------
q1	17601	4403	4250	4250
q2	2003	186	188	186
q3	10482	1151	1055	1055
q4	10198	803	834	803
q5	7455	2670	2623	2623
q6	220	132	131	131
q7	944	589	598	589
q8	9224	2051	2039	2039
q9	8654	6538	6446	6446
q10	8936	3719	3717	3717
q11	455	230	237	230
q12	444	226	223	223
q13	19089	2995	2995	2995
q14	271	218	203	203
q15	529	496	477	477
q16	522	379	371	371
q17	958	696	724	696
q18	7943	7537	7378	7378
q19	8097	1525	1513	1513
q20	688	319	320	319
q21	4837	3218	3933	3218
q22	399	332	335	332
Total cold run time: 119949 ms
Total hot run time: 39794 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4270	4224	4224
q2	372	269	265	265
q3	3080	2840	2946	2840
q4	1972	1770	1741	1741
q5	5479	5471	5430	5430
q6	220	128	138	128
q7	2244	1863	1837	1837
q8	3248	3436	3389	3389
q9	8661	8707	8741	8707
q10	4109	3630	3787	3630
q11	601	501	488	488
q12	807	631	645	631
q13	16292	3143	3174	3143
q14	298	262	275	262
q15	536	489	480	480
q16	486	429	431	429
q17	1805	1511	1511	1511
q18	8198	7833	7757	7757
q19	2596	1761	1614	1614
q20	2809	1875	1866	1866
q21	5167	4791	4982	4791
q22	641	566	557	557
Total cold run time: 74026 ms
Total hot run time: 55720 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

TPC-DS: Total hot run time: 173035 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8e900beda248701f9b7d7ec38d3f07063a6fd832, data reload: false

query1	931	379	373	373
query2	6365	2514	2252	2252
query3	6627	204	202	202
query4	19404	17490	17139	17139
query5	3577	474	465	465
query6	234	156	155	155
query7	4578	290	293	290
query8	294	290	272	272
query9	8378	2386	2369	2369
query10	561	285	287	285
query11	10604	10025	10038	10025
query12	109	86	84	84
query13	1625	377	390	377
query14	9640	7600	6323	6323
query15	226	195	180	180
query16	7706	271	273	271
query17	1894	524	503	503
query18	1905	270	267	267
query19	188	150	153	150
query20	89	81	81	81
query21	206	126	124	124
query22	4300	4027	4107	4027
query23	33751	33627	33606	33606
query24	10839	2926	2909	2909
query25	593	413	429	413
query26	744	155	159	155
query27	2355	325	326	325
query28	6124	2167	2168	2167
query29	900	628	631	628
query30	254	155	154	154
query31	1013	755	740	740
query32	90	52	56	52
query33	640	309	287	287
query34	903	485	501	485
query35	756	653	653	653
query36	1132	964	964	964
query37	157	76	74	74
query38	2922	2874	2842	2842
query39	907	860	858	858
query40	218	154	132	132
query41	55	51	51	51
query42	114	100	100	100
query43	597	556	535	535
query44	1055	744	729	729
query45	192	168	166	166
query46	1082	741	699	699
query47	1878	1784	1793	1784
query48	359	303	307	303
query49	836	406	411	406
query50	767	398	397	397
query51	6954	6808	6740	6740
query52	97	93	85	85
query53	355	300	297	297
query54	875	436	439	436
query55	76	72	72	72
query56	273	273	276	273
query57	1158	1072	1084	1072
query58	240	240	248	240
query59	3409	3062	3112	3062
query60	290	263	277	263
query61	96	124	100	100
query62	616	438	436	436
query63	322	292	296	292
query64	8921	2243	1725	1725
query65	3253	3096	3126	3096
query66	791	322	327	322
query67	15302	14894	15085	14894
query68	4494	541	546	541
query69	574	464	391	391
query70	1204	1136	1148	1136
query71	352	287	277	277
query72	7250	5530	5322	5322
query73	761	324	320	320
query74	5868	5553	5542	5542
query75	3361	2660	2680	2660
query76	2152	993	940	940
query77	611	316	319	316
query78	10315	9847	9719	9719
query79	2227	514	511	511
query80	1599	522	460	460
query81	585	219	217	217
query82	833	106	100	100
query83	281	168	165	165
query84	253	83	86	83
query85	1726	286	278	278
query86	458	321	331	321
query87	3283	3059	3095	3059
query88	4107	2355	2354	2354
query89	487	394	382	382
query90	1730	187	185	185
query91	129	101	99	99
query92	60	50	48	48
query93	2437	523	516	516
query94	1102	187	191	187
query95	400	315	321	315
query96	589	267	268	267
query97	3238	3047	3073	3047
query98	221	196	195	195
query99	1200	849	849	849
Total cold run time: 267589 ms
Total hot run time: 173035 ms

@doris-robot
Copy link

doris-robot commented Jun 26, 2024

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8e900beda248701f9b7d7ec38d3f07063a6fd832, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.22	0.06	0.05
query4	1.67	0.08	0.08
query5	0.48	0.48	0.46
query6	1.15	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.54	0.54	0.52
query11	0.14	0.12	0.11
query12	0.15	0.11	0.12
query13	0.59	0.58	0.59
query14	0.76	0.77	0.79
query15	0.82	0.82	0.80
query16	0.35	0.38	0.37
query17	0.98	0.94	1.00
query18	0.23	0.23	0.25
query19	1.90	1.66	1.70
query20	0.01	0.00	0.01
query21	15.44	0.65	0.65
query22	4.03	7.27	2.11
query23	18.23	1.33	1.31
query24	2.14	0.21	0.22
query25	0.16	0.08	0.09
query26	0.27	0.18	0.18
query27	0.07	0.08	0.08
query28	13.31	1.02	1.00
query29	12.58	3.26	3.27
query30	0.26	0.05	0.05
query31	2.89	0.38	0.38
query32	3.29	0.47	0.48
query33	2.90	2.92	2.91
query34	17.10	4.42	4.42
query35	4.49	4.47	4.56
query36	0.65	0.46	0.48
query37	0.19	0.16	0.16
query38	0.16	0.14	0.15
query39	0.06	0.03	0.03
query40	0.18	0.16	0.14
query41	0.09	0.04	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.29 s
Total hot run time: 30.63 s

HappenLee
HappenLee approved these changes Jun 26, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 26, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 26, 2024

PR approved by anyone and no changes requested.

wangbo
wangbo approved these changes Jun 26, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit cb7bed8 into apache:master Jun 26, 2024
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@xinyiZzz @dataroaring
[fix](arrow-flight-sql) Fix arrow flight result sink (#36827)
5087f7d 
1. get arrow flight result schema use query id instead of instance id.
2. get arrow flight result is a sync method, need wait for data ready
and return result, introduced by #36035 36667.
TODO, waiting for data will block pipeline, so use a request pool to
save requests waiting for data.
xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Jul 12, 2024
@xinyiZzz
[fix](arrow-flight-sql) Fix arrow flight result sink (apache#36827)
f4176c5 
1. get arrow flight result schema use query id instead of instance id.
2. get arrow flight result is a sync method, need wait for data ready
and return result, introduced by apache#36035 36667.
TODO, waiting for data will block pipeline, so use a request pool to
save requests waiting for data.
@xinyiZzz xinyiZzz mentioned this pull request Jul 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@HappenLee HappenLee HappenLee approved these changes

@wangbo wangbo wangbo approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@xinyiZzz @doris-robot @HappenLee @wangbo @dataroaring