Skip to content

branch-3.1: [fix](cloud) fix correct pipeline task number calculation in cloud mode #57129 #57261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Oct 30, 2025
Merged

branch-3.1: [fix](cloud) fix correct pipeline task number calculation in cloud mode #57129 #57261

morrySnow merged 1 commit into from
Oct 30, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Oct 23, 2025

Cherry-picked from #57129

@liaoxin01
[fix](cloud) fix correct pipeline task number calculation in cloud mo…
91a6ae6 
…de (#57129)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This fixes incorrect resource calculation that could lead to improper
pipeline task scheduling in cloud deployments.
Fix pipeline task calculation to use current compute group backends
instead of all repository backends in cloud mode. Use
getBackendsByCurrentCluster() instead of getAllBackendsByAllCluster().


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot requested a review from morrySnow as a code owner October 23, 2025 06:23
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Oct 23, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Oct 23, 2025

TPC-H: Total hot run time: 33233 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91a6ae6224baff54a5b0e870c007f741323afcaa, data reload: false

------ Round 1 ----------------------------------
q1	17589	5534	5358	5358
q2	2032	422	285	285
q3	12209	1279	776	776
q4	10544	881	477	477
q5	9379	2425	2213	2213
q6	193	163	132	132
q7	913	746	627	627
q8	9328	1456	1233	1233
q9	5309	4993	4962	4962
q10	6760	2320	1852	1852
q11	491	290	273	273
q12	349	355	219	219
q13	17809	3610	3044	3044
q14	231	234	214	214
q15	534	460	473	460
q16	452	435	378	378
q17	683	873	405	405
q18	6823	6527	6443	6443
q19	1206	978	555	555
q20	348	360	230	230
q21	3265	2281	2090	2090
q22	1082	1034	1007	1007
Total cold run time: 107529 ms
Total hot run time: 33233 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5541	5512	5499	5499
q2	242	338	241	241
q3	2296	2633	2373	2373
q4	1398	1796	1389	1389
q5	4452	5144	5028	5028
q6	175	162	134	134
q7	2126	1996	1779	1779
q8	2680	2822	2752	2752
q9	7349	7351	7349	7349
q10	3086	3220	2802	2802
q11	592	517	500	500
q12	715	774	609	609
q13	3509	3775	3271	3271
q14	282	300	286	286
q15	533	476	470	470
q16	448	478	440	440
q17	1275	1770	1268	1268
q18	7701	7415	7338	7338
q19	875	1211	1131	1131
q20	2059	2083	1947	1947
q21	5495	5074	4495	4495
q22	1117	1128	1068	1068
Total cold run time: 53946 ms
Total hot run time: 52169 ms

@doris-robot
Copy link

doris-robot commented Oct 23, 2025

TPC-DS: Total hot run time: 195814 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91a6ae6224baff54a5b0e870c007f741323afcaa, data reload: false

query1	965	410	400	400
query2	6315	2100	2024	2024
query3	8696	217	210	210
query4	33901	24297	23538	23538
query5	3664	612	465	465
query6	290	202	182	182
query7	4205	503	327	327
query8	313	249	264	249
query9	9397	2853	2848	2848
query10	496	328	271	271
query11	18105	15538	15313	15313
query12	158	115	110	110
query13	1572	569	444	444
query14	9021	8024	7297	7297
query15	238	197	183	183
query16	7796	670	487	487
query17	1586	794	614	614
query18	1462	426	353	353
query19	269	206	173	173
query20	130	129	127	127
query21	212	139	114	114
query22	4643	4817	4585	4585
query23	35274	34952	34276	34276
query24	7581	2736	2762	2736
query25	569	521	449	449
query26	892	294	184	184
query27	1975	486	378	378
query28	5802	2445	2403	2403
query29	710	618	498	498
query30	242	193	181	181
query31	973	922	856	856
query32	94	64	62	62
query33	492	371	333	333
query34	797	869	537	537
query35	819	835	756	756
query36	1049	1158	1027	1027
query37	115	99	71	71
query38	4018	4117	3952	3952
query39	1550	1526	1845	1526
query40	212	127	108	108
query41	51	53	49	49
query42	132	116	116	116
query43	561	555	515	515
query44	1373	881	892	881
query45	188	182	195	182
query46	935	1076	711	711
query47	2014	2028	1943	1943
query48	437	448	376	376
query49	719	500	410	410
query50	709	704	443	443
query51	7475	7408	7389	7389
query52	120	104	100	100
query53	250	278	202	202
query54	587	594	515	515
query55	93	83	79	79
query56	302	284	274	274
query57	1308	1291	1254	1254
query58	252	232	239	232
query59	3323	3250	3154	3154
query60	307	296	283	283
query61	120	116	130	116
query62	805	766	700	700
query63	251	213	205	205
query64	3710	1067	679	679
query65	3483	3408	3339	3339
query66	939	409	324	324
query67	16727	16090	15965	15965
query68	7569	861	585	585
query69	510	315	280	280
query70	1311	1196	1216	1196
query71	371	304	278	278
query72	5724	3755	3975	3755
query73	682	782	382	382
query74	9683	9143	8824	8824
query75	3281	3171	2684	2684
query76	3066	1190	800	800
query77	568	386	306	306
query78	10440	10472	9648	9648
query79	3409	925	624	624
query80	752	533	447	447
query81	508	256	226	226
query82	571	126	93	93
query83	181	169	150	150
query84	295	105	94	94
query85	801	366	313	313
query86	357	336	308	308
query87	4372	4333	4292	4292
query88	5152	2635	2623	2623
query89	405	343	304	304
query90	1854	190	192	190
query91	147	150	113	113
query92	61	60	54	54
query93	1831	946	580	580
query94	678	389	304	304
query95	357	284	285	284
query96	536	624	299	299
query97	3223	3309	3203	3203
query98	223	220	202	202
query99	1510	1429	1290	1290
Total cold run time: 294197 ms
Total hot run time: 195814 ms

@doris-robot
Copy link

doris-robot commented Oct 23, 2025

ClickBench: Total hot run time: 28.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91a6ae6224baff54a5b0e870c007f741323afcaa, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.52	0.52	0.52
query6	1.16	0.73	0.74
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.61	0.52	0.50
query10	0.56	0.55	0.56
query11	0.15	0.11	0.10
query12	0.14	0.12	0.11
query13	0.63	0.60	0.59
query14	0.79	0.82	0.79
query15	0.85	0.85	0.84
query16	0.41	0.42	0.39
query17	1.06	1.03	1.08
query18	0.24	0.22	0.23
query19	1.98	1.87	1.86
query20	0.02	0.02	0.01
query21	15.36	0.95	0.59
query22	0.77	0.87	0.65
query23	15.00	1.44	0.56
query24	3.58	0.55	1.79
query25	0.23	0.21	0.10
query26	0.37	0.14	0.15
query27	0.06	0.05	0.05
query28	12.97	1.02	0.45
query29	12.54	4.09	3.36
query30	0.27	0.10	0.07
query31	2.84	0.62	0.40
query32	3.23	0.54	0.46
query33	3.04	3.01	3.09
query34	16.71	5.19	4.56
query35	4.60	4.58	4.65
query36	0.64	0.50	0.46
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.02	0.03
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.86 s
Total hot run time: 28.82 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Oct 23, 2025

FE UT Coverage Report

Increment line coverage 92.31% (12/13) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit ad3cf0f into branch-3.1 Oct 30, 2025
23 checks passed
@morrySnow morrySnow deleted the auto-pick-57129-branch-3.1 branch October 30, 2025 06:39
@morrySnow morrySnow mentioned this pull request Nov 13, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@hello-stephen @doris-robot @morrySnow @dataroaring @liaoxin01