Skip to content

branch-3.0: [fix](cloud) fix corner case when warm up data larger than cache capacity #49050 #49675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 31, 2025
Merged

branch-3.0: [fix](cloud) fix corner case when warm up data larger than cache capacity #49050 #49675

dataroaring merged 1 commit into from
Mar 31, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Mar 31, 2025

Cherry-picked from #49050

@freemandealer
[fix](cloud) fix corner case when warm up data larger than cache capa…
6813e0b 
…city (#49050)

when warm up target table's first partition is larger cache capacity of
the cluster/compute group, the deleted code will break and leave
warmUpTotalFileCache zero, resulting no error&warning but 0 batch size.

this commit will warn user this situation and allow user use FORCE warm
up to load partial data into the file cache.
@github-actions github-actions bot requested a review from dataroaring as a code owner March 31, 2025 02:43
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 31, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Mar 31, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 31, 2025

TPC-H: Total hot run time: 40092 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6813e0b2f197e382e559eb7aa09d850d89bd923b, data reload: false

------ Round 1 ----------------------------------
q1	17581	6770	6658	6658
q2	2060	199	160	160
q3	10601	1071	1166	1071
q4	10559	707	735	707
q5	7722	2897	2802	2802
q6	217	133	129	129
q7	973	618	613	613
q8	9350	1917	1981	1917
q9	6516	6360	6426	6360
q10	7047	2308	2285	2285
q11	451	261	261	261
q12	393	215	210	210
q13	17771	2968	2970	2968
q14	245	207	214	207
q15	493	472	461	461
q16	681	575	565	565
q17	979	688	630	630
q18	7166	6825	6657	6657
q19	1403	1095	1070	1070
q20	469	202	202	202
q21	4269	3282	3180	3180
q22	1085	980	979	979
Total cold run time: 108031 ms
Total hot run time: 40092 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6666	6547	6565	6547
q2	326	238	226	226
q3	2861	2768	2873	2768
q4	2027	1757	1782	1757
q5	5722	5747	5756	5747
q6	208	131	126	126
q7	2212	1836	1781	1781
q8	3397	3567	3536	3536
q9	8806	8960	9014	8960
q10	3587	3529	3559	3529
q11	592	490	496	490
q12	810	626	610	610
q13	8099	3277	3212	3212
q14	303	274	274	274
q15	503	473	470	470
q16	711	655	687	655
q17	1869	1683	1761	1683
q18	8221	7827	7531	7531
q19	1645	1518	1608	1518
q20	2119	1872	1898	1872
q21	5525	5329	5352	5329
q22	1121	1049	1049	1049
Total cold run time: 67330 ms
Total hot run time: 59670 ms

@doris-robot
Copy link

doris-robot commented Mar 31, 2025

TPC-DS: Total hot run time: 196427 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6813e0b2f197e382e559eb7aa09d850d89bd923b, data reload: false

query1	1320	927	957	927
query2	6240	2082	2083	2082
query3	10837	4253	4198	4198
query4	65349	28674	23367	23367
query5	5031	470	482	470
query6	425	189	197	189
query7	5567	305	308	305
query8	307	222	229	222
query9	8905	2600	2587	2587
query10	456	267	256	256
query11	17244	15237	15820	15237
query12	153	102	112	102
query13	1496	458	440	440
query14	10440	6660	7026	6660
query15	202	183	191	183
query16	7132	471	474	471
query17	1087	568	584	568
query18	1894	321	334	321
query19	210	168	173	168
query20	122	113	111	111
query21	214	106	106	106
query22	4727	4352	4214	4214
query23	34523	34017	33996	33996
query24	6131	2987	2895	2895
query25	532	449	427	427
query26	663	171	168	168
query27	2003	363	373	363
query28	3953	2467	2420	2420
query29	715	474	462	462
query30	241	173	163	163
query31	994	826	846	826
query32	74	58	59	58
query33	423	303	303	303
query34	894	502	515	502
query35	842	736	730	730
query36	1070	994	1000	994
query37	119	70	68	68
query38	4056	4034	4054	4034
query39	1532	1481	1473	1473
query40	200	96	108	96
query41	49	45	46	45
query42	115	105	99	99
query43	547	489	497	489
query44	1172	814	801	801
query45	194	177	169	169
query46	1185	716	706	706
query47	1997	1930	1927	1927
query48	461	375	379	375
query49	738	411	403	403
query50	864	426	425	425
query51	7260	7146	7012	7012
query52	97	91	89	89
query53	257	185	181	181
query54	557	455	449	449
query55	81	77	80	77
query56	249	238	253	238
query57	1226	1114	1118	1114
query58	218	205	202	202
query59	3144	2921	2808	2808
query60	281	261	249	249
query61	111	118	108	108
query62	779	683	668	668
query63	221	186	193	186
query64	1360	694	632	632
query65	3281	3158	3156	3156
query66	693	311	321	311
query67	16028	15577	15666	15577
query68	3954	584	567	567
query69	426	264	274	264
query70	1176	1122	1074	1074
query71	356	257	247	247
query72	6359	4042	4185	4042
query73	741	343	340	340
query74	10273	9228	9218	9218
query75	3328	2639	2622	2622
query76	1899	1085	1116	1085
query77	504	285	270	270
query78	10461	9643	9642	9642
query79	1351	600	589	589
query80	995	417	426	417
query81	515	239	240	239
query82	184	87	87	87
query83	159	157	146	146
query84	280	80	78	78
query85	993	295	284	284
query86	376	295	292	292
query87	4425	4270	4415	4270
query88	4043	2365	2327	2327
query89	402	289	296	289
query90	2039	187	187	187
query91	179	149	149	149
query92	65	49	50	49
query93	2154	562	565	562
query94	849	292	280	280
query95	359	256	257	256
query96	605	278	280	278
query97	3310	3173	3147	3147
query98	212	204	201	201
query99	1613	1310	1271	1271
Total cold run time: 316073 ms
Total hot run time: 196427 ms

@freemandealer
Copy link
Contributor

freemandealer commented Mar 31, 2025

run cloud_p0

@doris-robot
Copy link

doris-robot commented Mar 31, 2025

ClickBench: Total hot run time: 32.52 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6813e0b2f197e382e559eb7aa09d850d89bd923b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.10
query5	0.53	0.52	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.04	0.05	0.03
query9	0.57	0.51	0.49
query10	0.56	0.56	0.56
query11	0.15	0.10	0.10
query12	0.13	0.11	0.11
query13	0.61	0.59	0.59
query14	2.73	2.90	2.88
query15	0.91	0.82	0.82
query16	0.38	0.38	0.40
query17	1.05	1.06	0.99
query18	0.23	0.22	0.20
query19	1.96	1.90	2.02
query20	0.01	0.01	0.01
query21	15.39	0.61	0.59
query22	3.46	2.35	1.97
query23	16.82	1.02	0.85
query24	3.79	1.10	1.88
query25	0.22	0.28	0.04
query26	0.50	0.15	0.15
query27	0.03	0.04	0.04
query28	8.99	0.54	0.46
query29	12.56	3.23	3.18
query30	0.24	0.05	0.06
query31	2.86	0.39	0.38
query32	3.24	0.45	0.46
query33	3.00	2.99	3.00
query34	17.14	4.48	4.52
query35	4.56	4.50	4.46
query36	0.65	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 106.9 s
Total hot run time: 32.52 s

dataroaring
dataroaring approved these changes Mar 31, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2025
@github-actions
Copy link
Contributor Author

github-actions bot commented Mar 31, 2025

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 086d09f into branch-3.0 Mar 31, 2025
23 of 24 checks passed
@github-actions
Copy link
Contributor Author

github-actions bot commented Mar 31, 2025

PR approved by anyone and no changes requested.

@github-actions github-actions bot deleted the auto-pick-49050-branch-3.0 branch March 31, 2025 06:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@hello-stephen @doris-robot @freemandealer @dataroaring