Skip to content

branch-3.0: [improve](routine load) ensure abnormal jobs do not interfere with normal task scheduling #47530 #47847

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Feb 24, 2025
Merged

branch-3.0: [improve](routine load) ensure abnormal jobs do not interfere with normal task scheduling #47530 #47847

dataroaring merged 1 commit into from
Feb 24, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Feb 13, 2025

Cherry-picked from #47530

@sollhui
[improve](routine load) ensure abnormal jobs do not interfere with no…
8dc8508 
…rmal task scheduling (#47530)

When a routine load job is created and successfully acquires all
partition information, but encounters persistent Kafka IP timeout issues
(e.g., due to network routing configuration changes), it may impact the
scheduling of all routine load tasks.
@Thearas
Copy link
Contributor

Thearas commented Feb 13, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 13, 2025
@Thearas
Copy link
Contributor

Thearas commented Feb 13, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 13, 2025

TPC-H: Total hot run time: 40908 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8dc85084cd248e5a402e06a7ed926bed9ceb50c2, data reload: false

------ Round 1 ----------------------------------
q1	17631	7377	7228	7228
q2	2068	177	166	166
q3	10883	1082	1140	1082
q4	10546	761	734	734
q5	7751	2839	2826	2826
q6	237	148	146	146
q7	971	617	608	608
q8	9344	1975	2020	1975
q9	6534	6385	6353	6353
q10	7031	2296	2358	2296
q11	465	265	262	262
q12	399	217	208	208
q13	17792	2980	3032	2980
q14	232	213	219	213
q15	565	511	538	511
q16	655	579	584	579
q17	976	530	584	530
q18	7166	6684	6726	6684
q19	1387	1038	1019	1019
q20	496	217	215	215
q21	3998	3331	3286	3286
q22	1076	1007	1016	1007
Total cold run time: 108203 ms
Total hot run time: 40908 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7403	7319	7267	7267
q2	340	241	237	237
q3	3043	3008	2992	2992
q4	2091	1813	1878	1813
q5	5842	5762	5698	5698
q6	224	141	146	141
q7	2250	1806	1809	1806
q8	3336	3507	3469	3469
q9	8934	8824	8794	8794
q10	3614	3566	3594	3566
q11	604	511	494	494
q12	788	620	622	620
q13	8603	3189	3182	3182
q14	308	292	265	265
q15	577	526	529	526
q16	690	673	628	628
q17	1827	1628	1583	1583
q18	8240	7887	7509	7509
q19	1645	1637	1620	1620
q20	2112	1866	1904	1866
q21	5419	5306	5228	5228
q22	1155	1078	1010	1010
Total cold run time: 69045 ms
Total hot run time: 60314 ms

@doris-robot
Copy link

doris-robot commented Feb 13, 2025

TPC-DS: Total hot run time: 198554 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8dc85084cd248e5a402e06a7ed926bed9ceb50c2, data reload: false

query1	1331	931	908	908
query2	6245	2122	2143	2122
query3	10841	4448	4375	4375
query4	66285	34076	23842	23842
query5	5002	457	461	457
query6	304	178	178	178
query7	4951	319	312	312
query8	295	235	235	235
query9	6191	2705	2681	2681
query10	413	268	268	268
query11	15815	15243	15734	15243
query12	163	106	103	103
query13	1053	452	426	426
query14	11049	7917	7522	7522
query15	207	188	185	185
query16	7305	488	502	488
query17	1091	589	604	589
query18	2154	335	332	332
query19	229	167	163	163
query20	121	112	118	112
query21	209	108	106	106
query22	4533	4260	4410	4260
query23	34711	34327	34242	34242
query24	6182	2913	2954	2913
query25	536	433	418	418
query26	659	173	181	173
query27	1837	355	360	355
query28	4203	2453	2438	2438
query29	708	512	426	426
query30	248	167	181	167
query31	996	854	853	853
query32	66	61	57	57
query33	446	292	311	292
query34	923	511	522	511
query35	840	731	732	731
query36	1081	983	981	981
query37	125	79	77	77
query38	4140	4187	4094	4094
query39	1573	1486	1478	1478
query40	207	104	106	104
query41	49	49	47	47
query42	121	105	104	104
query43	558	498	511	498
query44	1187	822	830	822
query45	195	175	167	167
query46	1185	738	727	727
query47	2010	1918	1924	1918
query48	498	391	392	391
query49	737	417	387	387
query50	861	448	426	426
query51	7301	7393	6967	6967
query52	106	90	88	88
query53	263	184	190	184
query54	562	457	450	450
query55	75	83	77	77
query56	273	259	256	256
query57	1236	1115	1066	1066
query58	216	211	235	211
query59	3343	3048	2981	2981
query60	296	272	259	259
query61	113	108	105	105
query62	843	755	729	729
query63	232	194	198	194
query64	1396	687	663	663
query65	3276	3193	3225	3193
query66	712	298	309	298
query67	15910	15526	15587	15526
query68	3641	595	577	577
query69	430	266	273	266
query70	1175	1087	1146	1087
query71	352	254	271	254
query72	6432	4200	4039	4039
query73	763	349	353	349
query74	10165	8989	9305	8989
query75	3357	2691	2655	2655
query76	1635	959	1052	959
query77	518	280	278	278
query78	10630	9636	9514	9514
query79	1598	612	593	593
query80	882	437	435	435
query81	515	248	247	247
query82	1290	123	122	122
query83	176	156	155	155
query84	284	82	76	76
query85	1019	306	297	297
query86	357	289	279	279
query87	4409	4270	4336	4270
query88	4120	2362	2351	2351
query89	427	307	301	301
query90	2048	189	193	189
query91	191	150	151	150
query92	69	52	53	52
query93	2278	543	551	543
query94	848	300	297	297
query95	364	263	254	254
query96	616	277	293	277
query97	3335	3185	3205	3185
query98	219	214	193	193
query99	1757	1370	1440	1370
Total cold run time: 314722 ms
Total hot run time: 198554 ms

@doris-robot
Copy link

doris-robot commented Feb 13, 2025

ClickBench: Total hot run time: 32.89 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8dc85084cd248e5a402e06a7ed926bed9ceb50c2, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.51	0.52	0.50
query6	1.16	0.73	0.73
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.50
query10	0.55	0.54	0.56
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.61
query14	2.75	2.83	2.73
query15	0.90	0.84	0.83
query16	0.36	0.38	0.38
query17	1.05	1.05	1.03
query18	0.24	0.21	0.22
query19	1.97	1.79	1.98
query20	0.02	0.01	0.00
query21	15.36	0.57	0.56
query22	2.20	2.07	1.94
query23	16.92	1.19	0.74
query24	3.05	1.16	0.99
query25	0.11	0.13	0.23
query26	0.47	0.15	0.13
query27	0.04	0.04	0.05
query28	10.61	1.10	1.06
query29	12.64	3.31	3.29
query30	0.24	0.06	0.07
query31	2.84	0.39	0.39
query32	3.24	0.47	0.46
query33	3.01	3.04	3.02
query34	17.00	4.50	4.45
query35	4.47	4.57	4.52
query36	0.66	0.49	0.51
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.31 s
Total hot run time: 32.89 s

dataroaring
dataroaring approved these changes Feb 22, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor Author

github-actions bot commented Feb 22, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 22, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Feb 22, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit be1737b into branch-3.0 Feb 24, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-47530-branch-3.0 branch February 24, 2025 03:04
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Thearas @doris-robot @dataroaring @sollhui