Skip to content

Conversation

@yiguolei
Copy link
Contributor

@yiguolei yiguolei commented Mar 3, 2025

What problem does this PR solve?

Reduce these two pool size to allow BE start normally.

  1. spill io threads could started as needed
  2. arrow flight thread pool is async thread, should not use too much.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei yiguolei added usercase Important user case type label p0_c dev/2.1.x dev/3.0.x labels Mar 3, 2025
@yiguolei
Copy link
Contributor Author

yiguolei commented Mar 3, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31576 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ee3783276ff4f133dfa99dd5e64e41fe1cdf4fc, data reload: false

------ Round 1 ----------------------------------
q1	17617	5114	5060	5060
q2	2046	322	188	188
q3	10366	1314	701	701
q4	10273	1022	515	515
q5	8276	2436	2297	2297
q6	195	174	129	129
q7	898	752	603	603
q8	9297	1305	1101	1101
q9	5061	4906	4716	4716
q10	6814	2303	1871	1871
q11	464	277	253	253
q12	350	352	218	218
q13	17771	3684	3106	3106
q14	224	217	221	217
q15	512	467	466	466
q16	630	614	580	580
q17	598	865	337	337
q18	6603	6384	6276	6276
q19	1521	953	536	536
q20	332	315	190	190
q21	2729	2222	1902	1902
q22	378	337	314	314
Total cold run time: 102955 ms
Total hot run time: 31576 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5122	5074	5156	5074
q2	232	325	238	238
q3	2152	2705	2269	2269
q4	1454	1812	1373	1373
q5	4268	4171	4165	4165
q6	216	164	122	122
q7	1874	1833	1787	1787
q8	2678	2613	2558	2558
q9	7272	7168	7108	7108
q10	2981	3205	2742	2742
q11	577	511	494	494
q12	696	795	640	640
q13	3561	3907	3236	3236
q14	291	308	273	273
q15	488	457	468	457
q16	640	702	632	632
q17	1128	1588	1338	1338
q18	7738	7424	7179	7179
q19	809	849	1002	849
q20	1943	2040	1857	1857
q21	5566	5044	4811	4811
q22	638	594	539	539
Total cold run time: 52324 ms
Total hot run time: 49741 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184402 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0ee3783276ff4f133dfa99dd5e64e41fe1cdf4fc, data reload: false

query1	985	410	384	384
query2	6545	1902	1871	1871
query3	6808	214	211	211
query4	26165	23753	23556	23556
query5	4324	677	480	480
query6	294	206	209	206
query7	4611	526	304	304
query8	303	251	244	244
query9	8637	2520	2543	2520
query10	509	324	263	263
query11	15413	15167	14975	14975
query12	160	114	110	110
query13	1668	541	404	404
query14	9177	6372	6437	6372
query15	212	184	172	172
query16	7335	650	483	483
query17	1189	708	538	538
query18	1970	410	305	305
query19	197	184	152	152
query20	116	111	109	109
query21	206	127	101	101
query22	4040	4183	4133	4133
query23	33958	33362	33077	33077
query24	7725	2391	2370	2370
query25	497	456	371	371
query26	1090	250	151	151
query27	2274	526	328	328
query28	4001	2429	2374	2374
query29	579	507	419	419
query30	238	194	158	158
query31	950	871	773	773
query32	74	70	64	64
query33	569	351	302	302
query34	803	899	494	494
query35	798	793	736	736
query36	942	977	875	875
query37	147	100	77	77
query38	4207	4115	4082	4082
query39	1459	1423	1555	1423
query40	207	118	102	102
query41	55	50	49	49
query42	128	110	104	104
query43	513	505	470	470
query44	1286	801	787	787
query45	177	168	159	159
query46	870	1052	641	641
query47	1741	1793	1701	1701
query48	383	423	298	298
query49	787	535	450	450
query50	725	745	420	420
query51	4222	4142	4178	4142
query52	112	105	97	97
query53	232	261	185	185
query54	497	495	411	411
query55	85	79	80	79
query56	279	272	263	263
query57	1124	1114	1040	1040
query58	246	241	259	241
query59	2656	2771	2670	2670
query60	271	278	254	254
query61	119	120	123	120
query62	794	749	679	679
query63	244	196	191	191
query64	4402	1134	723	723
query65	3252	3143	3128	3128
query66	1132	411	299	299
query67	15800	15445	15205	15205
query68	7130	883	495	495
query69	488	316	268	268
query70	1164	1116	1088	1088
query71	403	298	270	270
query72	5558	3676	3889	3676
query73	764	805	349	349
query74	9088	8908	8760	8760
query75	3205	3148	2678	2678
query76	3286	1212	750	750
query77	468	359	273	273
query78	9881	10158	9295	9295
query79	1871	918	597	597
query80	1482	545	452	452
query81	572	283	238	238
query82	421	131	104	104
query83	189	180	160	160
query84	243	98	85	85
query85	786	355	311	311
query86	454	298	274	274
query87	4509	4401	4324	4324
query88	2971	2240	2225	2225
query89	410	337	285	285
query90	1906	201	194	194
query91	141	143	112	112
query92	78	66	64	64
query93	1710	1086	563	563
query94	741	431	307	307
query95	362	269	264	264
query96	503	604	268	268
query97	3288	3372	3254	3254
query98	222	206	209	206
query99	1331	1420	1338	1338
Total cold run time: 269139 ms
Total hot run time: 184402 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0ee3783276ff4f133dfa99dd5e64e41fe1cdf4fc, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.56	0.54	0.56
query6	1.17	0.71	0.72
query7	0.03	0.02	0.01
query8	0.04	0.04	0.03
query9	0.59	0.53	0.52
query10	0.56	0.57	0.57
query11	0.16	0.11	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.69	2.82	2.84
query15	0.93	0.86	0.84
query16	0.39	0.39	0.37
query17	1.01	1.01	1.03
query18	0.21	0.21	0.20
query19	1.94	1.83	2.00
query20	0.02	0.01	0.02
query21	15.38	0.90	0.55
query22	0.75	1.15	0.64
query23	14.99	1.39	0.60
query24	7.11	0.87	1.52
query25	0.54	0.13	0.16
query26	0.62	0.17	0.15
query27	0.05	0.04	0.05
query28	8.99	0.84	0.42
query29	12.56	4.01	3.32
query30	0.25	0.09	0.07
query31	2.82	0.59	0.40
query32	3.23	0.56	0.49
query33	2.97	2.98	3.06
query34	15.81	5.11	4.56
query35	4.48	4.51	4.52
query36	0.67	0.52	0.50
query37	0.08	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 104.7 s
Total hot run time: 31.05 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 44.96% (12001/26690)
Line Coverage 34.46% (100842/292623)
Region Coverage 33.65% (51655/153522)
Branch Coverage 29.41% (26134/88856)

@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2025

PR approved by anyone and no changes requested.

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Mar 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 3, 2025
@yiguolei yiguolei merged commit b06b3c5 into apache:master Mar 3, 2025
28 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Mar 3, 2025
…ll io threads (#48530)

### What problem does this PR solve?

Reduce these two pool size to allow BE start normally.

1. spill io threads could started as needed
2. arrow flight thread pool is async thread, should not use too much.



### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
github-actions bot pushed a commit that referenced this pull request Mar 3, 2025
…ll io threads (#48530)

### What problem does this PR solve?

Reduce these two pool size to allow BE start normally.

1. spill io threads could started as needed
2. arrow flight thread pool is async thread, should not use too much.



### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Copy link
Contributor

@xinyiZzz xinyiZzz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

yiguolei added a commit that referenced this pull request Mar 3, 2025
…ight and spill io threads #48530 (#48556)

Cherry-picked from #48530

Co-authored-by: yiguolei <guolei@selectdb.com>
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

yiguolei added a commit that referenced this pull request Mar 10, 2025
…ight and spill io threads #48530 (#48555)

Cherry-picked from #48530

Co-authored-by: yiguolei <guolei@selectdb.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
…ll io threads (apache#48530)

### What problem does this PR solve?

Reduce these two pool size to allow BE start normally.

1. spill io threads could started as needed
2. arrow flight thread pool is async thread, should not use too much.



### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged p0_c reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants