Skip to content

[fix](coordinator) fix legacy coordinator use 1 instance for shuffle fragment when children has multiple instances #59295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
924060929 merged 1 commit into from
Dec 24, 2025
Merged

[fix](coordinator) fix legacy coordinator use 1 instance for shuffle fragment when children has multiple instances #59295

924060929 merged 1 commit into from
Dec 24, 2025
+109 −5

Conversation

@924060929
Copy link
Contributor

@924060929 924060929 commented Dec 23, 2025 •
edited
Loading

What problem does this PR solve?

fix legacy coordinator use 1 instance for shuffle fragment when children has multiple instances, introduced by #39999

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 23, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@924060929
Copy link
Contributor Author

924060929 commented Dec 23, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 23, 2025

TPC-H: Total hot run time: 36128 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 11847660a4f99b274ad18efa2a7709651511fc3d, data reload: false

------ Round 1 ----------------------------------
q1	17644	4293	4070	4070
q2	2010	343	233	233
q3	10201	1301	746	746
q4	10209	832	320	320
q5	7528	2131	1904	1904
q6	185	170	135	135
q7	997	864	709	709
q8	9360	1385	1128	1128
q9	6954	5300	5394	5300
q10	6862	2388	1969	1969
q11	520	342	289	289
q12	693	727	581	581
q13	17818	3633	3136	3136
q14	303	280	280	280
q15	592	521	504	504
q16	697	675	643	643
q17	697	827	537	537
q18	7456	7481	7817	7481
q19	1348	994	644	644
q20	416	375	263	263
q21	4527	4257	4269	4257
q22	1145	1089	999	999
Total cold run time: 108162 ms
Total hot run time: 36128 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4356	4374	4307	4307
q2	344	417	333	333
q3	2424	2855	2544	2544
q4	1458	1893	1495	1495
q5	4423	4680	4428	4428
q6	205	161	123	123
q7	2047	1975	1827	1827
q8	2894	2577	2593	2577
q9	7446	7574	7076	7076
q10	2931	3093	2664	2664
q11	609	533	508	508
q12	633	696	561	561
q13	3265	3616	3058	3058
q14	282	282	257	257
q15	540	494	489	489
q16	616	668	599	599
q17	1141	1435	1364	1364
q18	7368	7220	6980	6980
q19	872	835	848	835
q20	1889	1950	1791	1791
q21	4674	4351	4253	4253
q22	1096	1063	1013	1013
Total cold run time: 51513 ms
Total hot run time: 49082 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 23, 2025

FE UT Coverage Report

Increment line coverage 100.00% (5/5) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Dec 23, 2025

TPC-DS: Total hot run time: 179441 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 11847660a4f99b274ad18efa2a7709651511fc3d, data reload: false

query5	5454	600	442	442
query6	327	243	214	214
query7	4220	464	279	279
query8	317	249	234	234
query9	8773	2525	2532	2525
query10	554	371	315	315
query11	15646	14887	15102	14887
query12	179	120	111	111
query13	1261	482	379	379
query14	6378	3024	2769	2769
query14_1	2676	2647	2676	2647
query15	203	206	178	178
query16	904	467	435	435
query17	1135	731	622	622
query18	2681	442	347	347
query19	241	239	205	205
query20	128	119	117	117
query21	231	145	122	122
query22	4127	4195	4073	4073
query23	16554	16306	15888	15888
query23_1	16049	16156	16246	16156
query24	7398	1667	1241	1241
query24_1	1242	1250	1271	1250
query25	585	511	460	460
query26	1302	276	157	157
query27	2751	475	306	306
query28	4462	2130	2104	2104
query29	796	544	446	446
query30	312	243	213	213
query31	809	707	625	625
query32	76	74	65	65
query33	549	342	289	289
query34	890	900	544	544
query35	768	801	708	708
query36	859	906	821	821
query37	126	91	80	80
query38	2954	3014	3070	3014
query39	758	748	729	729
query39_1	714	717	696	696
query40	244	139	127	127
query41	68	66	65	65
query42	108	111	111	111
query43	454	447	417	417
query44	1363	773	764	764
query45	192	207	186	186
query46	900	1020	631	631
query47	1709	1778	1669	1669
query48	335	339	260	260
query49	656	443	392	392
query50	705	323	244	244
query51	3956	3876	3814	3814
query52	106	115	102	102
query53	327	374	292	292
query54	290	258	246	246
query55	82	73	74	73
query56	287	300	295	295
query57	1160	1187	1103	1103
query58	265	263	259	259
query59	2357	2511	2363	2363
query60	315	314	292	292
query61	180	153	153	153
query62	748	743	656	656
query63	326	296	297	296
query64	4951	1297	994	994
query65	4018	3962	3967	3962
query66	1372	449	323	323
query67	15415	14953	15025	14953
query68	4946	1010	737	737
query69	496	354	317	317
query70	1103	1009	961	961
query71	334	309	281	281
query72	6022	4951	5202	4951
query73	693	598	304	304
query74	8880	8859	8743	8743
query75	3182	3170	2815	2815
query76	3831	1151	751	751
query77	533	411	294	294
query78	9634	9704	8975	8975
query79	1695	921	619	619
query80	1589	654	554	554
query81	559	265	240	240
query82	406	136	102	102
query83	362	257	244	244
query84	262	130	104	104
query85	954	514	466	466
query86	412	278	279	278
query87	3242	3219	3069	3069
query88	3227	2278	2242	2242
query89	470	424	387	387
query90	1939	163	156	156
query91	177	181	152	152
query92	71	69	67	67
query93	1159	945	578	578
query94	555	288	279	279
query95	549	328	369	328
query96	589	465	212	212
query97	2314	2311	2234	2234
query98	211	205	196	196
query99	1334	1376	1319	1319
Total cold run time: 260034 ms
Total hot run time: 179441 ms

@doris-robot
Copy link

doris-robot commented Dec 23, 2025

ClickBench: Total hot run time: 27.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 11847660a4f99b274ad18efa2a7709651511fc3d, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.11
query5	0.27	0.26	0.25
query6	1.17	0.64	0.63
query7	0.03	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.52	0.50
query10	0.54	0.56	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.11
query13	0.61	0.60	0.60
query14	0.99	0.98	0.99
query15	0.80	0.81	0.79
query16	0.40	0.41	0.43
query17	1.04	1.06	0.97
query18	0.23	0.22	0.21
query19	1.95	1.88	1.82
query20	0.02	0.01	0.01
query21	15.43	0.29	0.15
query22	5.01	0.05	0.05
query23	15.95	0.28	0.10
query24	1.15	0.25	0.17
query25	0.13	0.06	0.06
query26	0.14	0.12	0.14
query27	0.07	0.08	0.05
query28	3.09	1.21	1.03
query29	12.66	3.93	3.31
query30	0.28	0.14	0.12
query31	2.81	0.62	0.40
query32	3.23	0.55	0.45
query33	3.03	3.00	3.04
query34	16.87	5.17	4.51
query35	4.61	4.56	4.55
query36	0.64	0.51	0.49
query37	0.10	0.06	0.07
query38	0.08	0.05	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.64 s
Total hot run time: 27.19 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 24, 2025

PR approved by anyone and no changes requested.

@924060929 924060929 merged commit ae2ba6f into apache:master Dec 24, 2025
32 of 34 checks passed
@924060929 924060929 deleted the fix-old-coordinator-one-instance branch December 24, 2025 10:52
github-actions bot pushed a commit that referenced this pull request Dec 24, 2025
@924060929
[fix](coordinator) fix legacy coordinator use 1 instance for shuffle …
098e06f 
…fragment when children has multiple instances (#59295)

fix legacy coordinator use 1 instance for shuffle fragment when children
has multiple instances, introduced by #39999
@github-actions github-actions bot mentioned this pull request Dec 24, 2025
Closed
github-actions bot pushed a commit that referenced this pull request Dec 24, 2025
@924060929
[fix](coordinator) fix legacy coordinator use 1 instance for shuffle …
1dadcb0 
…fragment when children has multiple instances (#59295)

fix legacy coordinator use 1 instance for shuffle fragment when children
has multiple instances, introduced by #39999
@github-actions github-actions bot mentioned this pull request Dec 24, 2025
Merged
924060929 added a commit that referenced this pull request Dec 25, 2025
@924060929
[fix](coordinator) fix legacy coordinator use 1 instance for shuffle …
3d5d0ed 
…fragment when children has multiple instances (#59295)

fix legacy coordinator use 1 instance for shuffle fragment when children
has multiple instances, introduced by #39999

(cherry picked from commit ae2ba6f)
@924060929 924060929 mentioned this pull request Dec 25, 2025
Merged
@924060929 924060929 added the usercase Important user case type label label Dec 25, 2025
morrySnow pushed a commit that referenced this pull request Dec 25, 2025
@924060929
branch-3.1: [fix](coordinator) fix legacy coordinator use 1 instance …
041e9b4 
…for shuffle fragment when children has multiple instances #59295 (#59349)

cherry pick from #59295
yiguolei pushed a commit that referenced this pull request Dec 26, 2025
@github-actions @924060929
branch-4.0: [fix](coordinator) fix legacy coordinator use 1 instance …
b546cee 
…for shuffle fragment when children has multiple instances #59295 (#59334)

Cherry-picked from #59295

Co-authored-by: 924060929 <lanhuajian@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.4-merged dev/4.0.3-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@924060929 @Thearas @doris-robot @hello-stephen @starocean999 @morrySnow @yiguolei