Skip to content

[feature](Nereids) parallel export into file#31623

Merged
morrySnow merged 1 commit intoapache:masterfrom
morrySnow:parallel_outfile
Mar 4, 2024
Merged

[feature](Nereids) parallel export into file#31623
morrySnow merged 1 commit intoapache:masterfrom
morrySnow:parallel_outfile

Conversation

@morrySnow
Copy link
Contributor

legacy planner impl PR: #6539

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@morrySnow
Copy link
Contributor Author

run buildall

@morrySnow morrySnow changed the title [feature](Nereids) parallel output file [feature](Nereids) parallel output export Feb 29, 2024
@morrySnow morrySnow changed the title [feature](Nereids) parallel output export [feature](Nereids) parallel export into file Feb 29, 2024
@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37826 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0621a422c00325810a28ffb1d05191d9974bb882, data reload: false

------ Round 1 ----------------------------------
q1	17673	4002	4005	4002
q2	2048	148	141	141
q3	10563	938	934	934
q4	4664	937	939	937
q5	7645	2893	3012	2893
q6	183	126	129	126
q7	1309	821	809	809
q8	9478	2016	2087	2016
q9	7264	6520	6467	6467
q10	8261	2595	2599	2595
q11	422	219	217	217
q12	717	313	312	312
q13	17991	2945	3004	2945
q14	310	249	261	249
q15	474	449	434	434
q16	468	390	390	390
q17	938	885	852	852
q18	6692	5919	5774	5774
q19	1572	1532	1516	1516
q20	533	280	275	275
q21	7606	3641	3771	3641
q22	794	307	301	301
Total cold run time: 107605 ms
Total hot run time: 37826 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3987	3988	3990	3988
q2	311	219	222	219
q3	2939	2910	2917	2910
q4	1841	1798	1815	1798
q5	5221	5208	5216	5208
q6	202	119	119	119
q7	2242	1830	1841	1830
q8	3182	3254	3260	3254
q9	8532	8574	8503	8503
q10	6172	3685	3723	3685
q11	531	441	445	441
q12	697	554	543	543
q13	8784	2798	2779	2779
q14	281	254	253	253
q15	473	439	436	436
q16	445	394	431	394
q17	1694	1667	1625	1625
q18	7819	7473	7378	7378
q19	3202	1614	1613	1613
q20	1935	1742	1760	1742
q21	4885	4829	4762	4762
q22	528	467	491	467
Total cold run time: 65903 ms
Total hot run time: 53947 ms

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38043 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7be5a8b1480a1a26d1704e2241f563c3c0e5220e, data reload: false

------ Round 1 ----------------------------------
q1	17659	4062	4026	4026
q2	2048	149	145	145
q3	10555	961	953	953
q4	4665	933	948	933
q5	7648	2902	3076	2902
q6	179	124	129	124
q7	1301	836	821	821
q8	9504	2017	2091	2017
q9	7364	6503	6488	6488
q10	8196	2588	2581	2581
q11	425	229	219	219
q12	767	320	314	314
q13	17982	2950	2992	2950
q14	276	264	279	264
q15	473	433	437	433
q16	458	390	394	390
q17	945	853	845	845
q18	6737	5890	5947	5890
q19	1589	1510	1524	1510
q20	550	286	276	276
q21	7571	3679	3667	3667
q22	803	295	300	295
Total cold run time: 107695 ms
Total hot run time: 38043 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3977	3974	4096	3974
q2	320	220	219	219
q3	2945	2944	2912	2912
q4	1858	1837	1812	1812
q5	5258	5224	5239	5224
q6	199	119	118	118
q7	2257	1814	1854	1814
q8	3242	3281	3287	3281
q9	8557	8504	8536	8504
q10	6159	3716	3737	3716
q11	519	452	447	447
q12	694	511	526	511
q13	12406	2757	2778	2757
q14	283	245	249	245
q15	470	441	435	435
q16	460	412	410	410
q17	1699	1660	1647	1647
q18	7981	7465	7374	7374
q19	1627	1618	1620	1618
q20	2001	1717	1726	1717
q21	4983	4756	4840	4756
q22	481	474	475	474
Total cold run time: 68376 ms
Total hot run time: 53965 ms

Copy link
Contributor

@BePPPower BePPPower left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Mar 1, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 4, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 4, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit 5048c6e into apache:master Mar 4, 2024
@morrySnow morrySnow deleted the parallel_outfile branch March 4, 2024 04:14
@morrySnow morrySnow added not-merge/2.0 do not merge into 2.0 branch and removed dev/2.0.x labels Mar 5, 2024
yiguolei pushed a commit that referenced this pull request Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. not-merge/2.0 do not merge into 2.0 branch reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants