Skip to content

[opt](cloud) optimize create warmup job logic #57752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liutang123 merged 1 commit into from
Nov 10, 2025
Merged

[opt](cloud) optimize create warmup job logic #57752

liutang123 merged 1 commit into from
Nov 10, 2025

Conversation

@liutang123
Copy link
Contributor

@liutang123 liutang123 commented Nov 6, 2025

What problem does this PR solve?

When executing the warm up command for multiple tables, the speed is too slow.

Clipboard_Screenshot_1762401437 Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@liutang123
Copy link
Contributor Author

liutang123 commented Nov 6, 2025

@freemandealer Hi, do you have time to see this PR?

@liutang123
Copy link
Contributor Author

liutang123 commented Nov 6, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 6, 2025

TPC-DS: Total hot run time: 188578 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c2e8e3773ec887b8ccd8cdcb67ecb70ecf6f6db6, data reload: false

query1	1055	422	379	379
query2	6549	1688	1667	1667
query3	6755	222	214	214
query4	26242	24184	23060	23060
query5	5339	637	447	447
query6	328	242	213	213
query7	4655	496	295	295
query8	294	253	242	242
query9	8715	2596	2607	2596
query10	541	352	285	285
query11	15971	15057	14804	14804
query12	181	113	111	111
query13	1667	539	432	432
query14	11116	9163	9142	9142
query15	221	187	174	174
query16	7559	685	492	492
query17	1186	756	627	627
query18	2034	499	377	377
query19	236	230	197	197
query20	145	142	167	142
query21	212	138	117	117
query22	4715	4875	4530	4530
query23	34588	33601	33842	33601
query24	8570	2519	2564	2519
query25	607	523	519	519
query26	1714	277	156	156
query27	2798	531	377	377
query28	4802	2211	2200	2200
query29	844	685	512	512
query30	303	230	200	200
query31	930	837	764	764
query32	85	74	70	70
query33	604	404	348	348
query34	841	891	552	552
query35	854	876	745	745
query36	983	977	958	958
query37	123	102	78	78
query38	3667	3628	3712	3628
query39	1578	1445	1452	1445
query40	218	133	113	113
query41	59	60	54	54
query42	130	107	115	107
query43	490	509	479	479
query44	1240	778	735	735
query45	182	174	175	174
query46	886	993	638	638
query47	1740	1827	1723	1723
query48	399	423	310	310
query49	754	479	414	414
query50	635	681	401	401
query51	3966	4006	3906	3906
query52	106	114	95	95
query53	232	256	191	191
query54	298	306	262	262
query55	86	83	82	82
query56	306	320	301	301
query57	1182	1171	1131	1131
query58	276	264	268	264
query59	2563	2627	2530	2530
query60	340	336	330	330
query61	164	161	155	155
query62	780	726	641	641
query63	219	184	190	184
query64	4442	1151	844	844
query65	4034	3976	3956	3956
query66	1075	423	321	321
query67	15205	15147	14938	14938
query68	7989	867	590	590
query69	484	317	283	283
query70	1326	1255	1240	1240
query71	442	326	297	297
query72	5846	4925	4904	4904
query73	654	571	352	352
query74	9097	9030	8763	8763
query75	3296	3361	2855	2855
query76	3279	1160	707	707
query77	499	398	310	310
query78	9640	9781	8922	8922
query79	1054	807	588	588
query80	652	554	514	514
query81	496	258	224	224
query82	345	158	130	130
query83	263	264	242	242
query84	250	118	97	97
query85	869	469	427	427
query86	320	323	284	284
query87	3675	3703	3536	3536
query88	2819	2252	2208	2208
query89	374	321	297	297
query90	1851	216	205	205
query91	166	163	134	134
query92	67	68	63	63
query93	1106	963	638	638
query94	651	441	339	339
query95	397	319	301	301
query96	484	574	275	275
query97	2868	2978	2873	2873
query98	237	217	212	212
query99	1313	1393	1274	1274
Total cold run time: 274423 ms
Total hot run time: 188578 ms

@doris-robot
Copy link

doris-robot commented Nov 6, 2025

ClickBench: Total hot run time: 27.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c2e8e3773ec887b8ccd8cdcb67ecb70ecf6f6db6, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.04	0.05
query3	0.25	0.08	0.08
query4	1.61	0.12	0.11
query5	0.27	0.26	0.26
query6	1.19	0.65	0.65
query7	0.03	0.04	0.03
query8	0.06	0.04	0.04
query9	0.61	0.53	0.51
query10	0.58	0.57	0.59
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.60
query14	1.01	1.01	1.01
query15	0.84	0.83	0.83
query16	0.40	0.41	0.41
query17	1.00	1.06	1.05
query18	0.22	0.20	0.20
query19	1.97	1.80	1.85
query20	0.02	0.01	0.01
query21	15.46	0.19	0.14
query22	5.04	0.07	0.05
query23	15.66	0.25	0.10
query24	2.80	0.74	0.50
query25	0.08	0.06	0.06
query26	0.14	0.13	0.13
query27	0.07	0.06	0.06
query28	4.50	1.14	0.93
query29	12.63	4.02	3.28
query30	0.27	0.15	0.12
query31	2.81	0.62	0.38
query32	3.24	0.55	0.46
query33	2.97	3.02	3.15
query34	15.90	5.16	4.54
query35	4.58	4.62	4.57
query36	0.66	0.50	0.49
query37	0.09	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.10	0.03	0.03
query42	0.04	0.04	0.03
query43	0.04	0.04	0.04
Total cold run time: 98.49 s
Total hot run time: 27.69 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 6, 2025

FE UT Coverage Report

Increment line coverage 100.00% (7/7) 🎉
Increment coverage report
Complete coverage report

@liutang123
Copy link
Contributor Author

liutang123 commented Nov 6, 2025

run p0

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 6, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

@liutang123
Copy link
Contributor Author

liutang123 commented Nov 6, 2025

run buildall

dataroaring
dataroaring approved these changes Nov 7, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 7, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 7, 2025

PR approved by anyone and no changes requested.

@liutang123
Copy link
Contributor Author

liutang123 commented Nov 7, 2025

run p0

@liutang123
Copy link
Contributor Author

liutang123 commented Nov 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 10, 2025

TPC-H: Total hot run time: 34162 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a9a472729ac22812169ce3e80c8a04fc442b0080, data reload: false

------ Round 1 ----------------------------------
q1	17646	5194	5056	5056
q2	2087	306	203	203
q3	10287	1285	719	719
q4	10234	933	377	377
q5	7489	2373	2334	2334
q6	189	170	134	134
q7	896	766	625	625
q8	9333	1299	1054	1054
q9	6962	5177	5141	5141
q10	6868	2228	1848	1848
q11	496	306	293	293
q12	461	367	232	232
q13	17809	3642	3020	3020
q14	230	228	211	211
q15	585	497	493	493
q16	1029	1000	946	946
q17	594	872	372	372
q18	7801	7131	7038	7038
q19	1356	938	560	560
q20	359	326	226	226
q21	3796	3223	2306	2306
q22	1072	1028	974	974
Total cold run time: 107579 ms
Total hot run time: 34162 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5195	5108	5102	5102
q2	257	320	229	229
q3	2172	2641	2282	2282
q4	1336	1728	1345	1345
q5	4180	4638	4489	4489
q6	244	183	138	138
q7	2022	1932	1819	1819
q8	2645	2623	2485	2485
q9	7322	7443	7289	7289
q10	3123	3213	2810	2810
q11	606	526	497	497
q12	676	734	591	591
q13	3445	3832	3456	3456
q14	293	306	284	284
q15	557	514	500	500
q16	1082	1081	1016	1016
q17	1230	1611	1409	1409
q18	7818	7558	7675	7558
q19	822	849	1015	849
q20	2027	2081	1927	1927
q21	4836	4392	4340	4340
q22	1084	1063	1003	1003
Total cold run time: 52972 ms
Total hot run time: 51418 ms

@doris-robot
Copy link

doris-robot commented Nov 10, 2025

TPC-DS: Total hot run time: 187624 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a9a472729ac22812169ce3e80c8a04fc442b0080, data reload: false

query1	1615	420	420	420
query2	6566	1655	1722	1655
query3	6838	225	221	221
query4	26984	23425	23696	23425
query5	4448	637	448	448
query6	346	231	217	217
query7	4642	492	299	299
query8	317	259	240	240
query9	8704	2552	2581	2552
query10	468	344	294	294
query11	16015	15190	14882	14882
query12	180	120	114	114
query13	1678	563	449	449
query14	11004	9166	9230	9166
query15	203	189	168	168
query16	7300	666	541	541
query17	1266	780	637	637
query18	2057	457	315	315
query19	201	196	171	171
query20	130	125	121	121
query21	458	133	114	114
query22	4096	4043	3877	3877
query23	33666	32979	32900	32900
query24	8503	2389	2403	2389
query25	583	510	454	454
query26	734	280	157	157
query27	2704	506	347	347
query28	4284	2190	2182	2182
query29	757	606	478	478
query30	302	223	198	198
query31	927	821	729	729
query32	85	71	76	71
query33	600	391	329	329
query34	788	836	514	514
query35	829	830	759	759
query36	931	989	893	893
query37	177	110	82	82
query38	3499	3581	3428	3428
query39	1495	1398	1439	1398
query40	230	128	121	121
query41	61	59	62	59
query42	127	113	113	113
query43	498	499	456	456
query44	1247	732	733	732
query45	190	183	166	166
query46	875	972	636	636
query47	1774	1832	1701	1701
query48	398	445	316	316
query49	725	528	416	416
query50	636	682	411	411
query51	3878	3948	3970	3948
query52	118	116	109	109
query53	240	275	207	207
query54	310	309	274	274
query55	93	88	86	86
query56	335	316	303	303
query57	1178	1199	1098	1098
query58	287	277	273	273
query59	2509	2693	2538	2538
query60	346	361	336	336
query61	163	157	162	157
query62	809	739	669	669
query63	232	192	192	192
query64	3171	1180	866	866
query65	4009	3953	3960	3953
query66	1032	436	348	348
query67	15296	15373	15130	15130
query68	7333	871	598	598
query69	571	318	279	279
query70	1371	1287	1261	1261
query71	436	345	313	313
query72	5582	4765	4854	4765
query73	600	570	361	361
query74	8988	9128	8963	8963
query75	3324	3373	2903	2903
query76	3216	1162	734	734
query77	520	410	317	317
query78	9417	9574	8793	8793
query79	1544	837	593	593
query80	632	585	512	512
query81	513	260	234	234
query82	212	159	134	134
query83	280	266	253	253
query84	262	114	88	88
query85	911	474	428	428
query86	380	317	298	298
query87	3703	3669	3659	3659
query88	2844	2248	2222	2222
query89	392	325	310	310
query90	1898	220	220	220
query91	169	169	132	132
query92	78	69	68	68
query93	1930	977	642	642
query94	725	458	343	343
query95	389	324	303	303
query96	489	568	282	282
query97	2903	2957	2835	2835
query98	240	210	208	208
query99	1307	1401	1325	1325
Total cold run time: 270306 ms
Total hot run time: 187624 ms

@doris-robot
Copy link

doris-robot commented Nov 10, 2025

ClickBench: Total hot run time: 27.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a9a472729ac22812169ce3e80c8a04fc442b0080, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.04	0.05
query3	0.26	0.09	0.08
query4	1.61	0.12	0.12
query5	0.26	0.26	0.25
query6	1.16	0.64	0.64
query7	0.03	0.03	0.03
query8	0.06	0.05	0.04
query9	0.60	0.54	0.51
query10	0.58	0.58	0.57
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.62
query14	1.00	1.00	0.99
query15	0.85	0.83	0.84
query16	0.40	0.39	0.40
query17	1.01	1.02	1.05
query18	0.21	0.20	0.20
query19	1.89	1.74	1.76
query20	0.02	0.01	0.02
query21	15.46	0.24	0.14
query22	4.93	0.08	0.05
query23	15.63	0.26	0.10
query24	2.46	0.77	1.23
query25	0.09	0.08	0.08
query26	0.15	0.13	0.12
query27	0.06	0.06	0.05
query28	5.72	1.13	0.94
query29	12.57	3.86	3.23
query30	0.30	0.14	0.12
query31	2.82	0.60	0.38
query32	3.23	0.58	0.48
query33	3.03	3.06	3.09
query34	15.67	5.14	4.54
query35	4.59	4.57	4.60
query36	0.67	0.50	0.49
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 98.94 s
Total hot run time: 27.82 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 10, 2025

FE Regression Coverage Report

Increment line coverage 0.00% (0/7) 🎉
Increment coverage report
Complete coverage report

lide-reed
lide-reed approved these changes Nov 10, 2025
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liutang123 liutang123 merged commit 009c3b5 into apache:master Nov 10, 2025
26 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 10, 2025
@liutang123
[opt](cloud) optimize the logic of creating warmup job(#57752)
b36aea3 
Retrieve the tablet set only once when creating a warm up job

Co-authored-by: liulijia <liulijia1029@gmail.com>
@github-actions github-actions bot mentioned this pull request Nov 10, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Nov 10, 2025
@liutang123
[opt](cloud) optimize the logic of creating warmup job(#57752)
ad85b84 
Retrieve the tablet set only once when creating a warm up job

Co-authored-by: liulijia <liulijia1029@gmail.com>
@github-actions github-actions bot mentioned this pull request Nov 10, 2025
Merged
liutang123 added a commit to liutang123/doris that referenced this pull request Nov 11, 2025
@liutang123
[opt](cloud) optimize the logic of creating warmup job(apache#57752)
c5a4f06 
Retrieve the tablet set only once when creating a warm up job

Co-authored-by: liulijia <liulijia1029@gmail.com>
@liutang123 liutang123 mentioned this pull request Nov 11, 2025
Merged
yiguolei pushed a commit that referenced this pull request Nov 11, 2025
@github-actions @liutang123
branch-4.0: [opt](cloud) optimize create warmup job logic #57752 (#57866
bf22655 
)

Cherry-picked from #57752

Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: liulijia <liulijia1029@gmail.com>
liutang123 added a commit that referenced this pull request Nov 11, 2025
@liutang123
[opt](cloud) optimize the logic of creating warmup job(#57752) (#57896)
36344b8 
Cherry-picked from #57752

Co-authored-by: liulijia <liulijia1029@gmail.com>
morrySnow pushed a commit that referenced this pull request Nov 11, 2025
@github-actions @liutang123
branch-3.1: [opt](cloud) optimize create warmup job logic #57752 (#57865
ab04c2a 
)

Cherry-picked from #57752

Co-authored-by: Lijia Liu <liutang123@yeah.net>
Co-authored-by: liulijia <liulijia1029@gmail.com>
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 13, 2025
@liutang123 @wyxxxcat
[opt](cloud) optimize the logic of creating warmup job(apache#57752)
1347814 
Retrieve the tablet set only once when creating a warm up job

Co-authored-by: liulijia <liulijia1029@gmail.com>
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Nov 18, 2025
@liutang123 @wyxxxcat
[opt](cloud) optimize the logic of creating warmup job(apache#57752)
8e82941 
Retrieve the tablet set only once when creating a warm up job

Co-authored-by: liulijia <liulijia1029@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

@w41ter w41ter Awaiting requested review from w41ter w41ter is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.x dev/3.1.3-merged dev/4.0.2-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@liutang123 @doris-robot @hello-stephen @lide-reed @dataroaring @yiguolei @morrySnow