Skip to content

[chore](file-cache) Enable file cache for cloud mode by force #41357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
hello-stephen merged 1 commit into from
Sep 29, 2024
Merged

[chore](file-cache) Enable file cache for cloud mode by force #41357

hello-stephen merged 1 commit into from
Sep 29, 2024

Conversation

@Hastyshell
Copy link
Collaborator

@Hastyshell Hastyshell commented Sep 26, 2024

Proposed changes

Temp local rowset writer for external sorting replies on file cache.

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 26, 2024

run buildall

dataroaring
dataroaring previously approved these changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 26, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 26, 2024

PR approved by anyone and no changes requested.

@Hastyshell Hastyshell force-pushed the fix-cloud-sc-filecache branch from c303196 to 5f68ebe Compare September 26, 2024 10:05
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 26, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TPC-H: Total hot run time: 40821 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f68ebe93d37d08ee4ecac84a1ca5e73e8a9619d, data reload: false

------ Round 1 ----------------------------------
q1	17851	7446	7358	7358
q2	2364	163	152	152
q3	11026	1139	1162	1139
q4	10226	743	694	694
q5	7730	2861	2762	2762
q6	240	151	148	148
q7	965	618	598	598
q8	9341	1928	1955	1928
q9	6554	6410	6391	6391
q10	6940	2282	2355	2282
q11	439	250	250	250
q12	407	221	213	213
q13	17801	3007	2976	2976
q14	247	207	219	207
q15	575	531	512	512
q16	643	592	585	585
q17	988	523	563	523
q18	7241	6624	6769	6624
q19	1396	1042	1105	1042
q20	480	205	199	199
q21	4006	3258	3293	3258
q22	1086	980	1004	980
Total cold run time: 108546 ms
Total hot run time: 40821 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7232	7211	7244	7211
q2	335	231	224	224
q3	2922	2753	2739	2739
q4	1933	1652	1667	1652
q5	5399	5422	5484	5422
q6	226	143	142	142
q7	2097	1714	1756	1714
q8	3200	3392	3371	3371
q9	8552	8446	8487	8446
q10	3483	3457	3420	3420
q11	574	490	474	474
q12	788	565	603	565
q13	5544	2999	3002	2999
q14	287	260	274	260
q15	560	528	510	510
q16	696	639	632	632
q17	1798	1577	1548	1548
q18	7775	7480	7460	7460
q19	1654	1475	1584	1475
q20	2043	1795	1831	1795
q21	5449	5127	5223	5127
q22	1088	994	1017	994
Total cold run time: 63635 ms
Total hot run time: 58180 ms

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TeamCity be ut coverage result:
Function Coverage: 37.31% (9635/25823)
Line Coverage: 28.72% (79749/277663)
Region Coverage: 28.13% (41219/146511)
Branch Coverage: 24.76% (20988/84768)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5f68ebe93d37d08ee4ecac84a1ca5e73e8a9619d_5f68ebe93d37d08ee4ecac84a1ca5e73e8a9619d/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TPC-DS: Total hot run time: 191680 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f68ebe93d37d08ee4ecac84a1ca5e73e8a9619d, data reload: false

query1	973	364	388	364
query2	6516	2184	2111	2111
query3	6700	211	225	211
query4	34697	23671	23289	23289
query5	4962	464	464	464
query6	268	176	161	161
query7	4625	309	314	309
query8	276	222	223	222
query9	9478	2708	2697	2697
query10	484	300	270	270
query11	18137	15027	15116	15027
query12	154	101	105	101
query13	1625	432	435	432
query14	10488	6638	7458	6638
query15	299	171	173	171
query16	8091	475	485	475
query17	1758	579	565	565
query18	2147	308	315	308
query19	385	156	155	155
query20	121	106	107	106
query21	215	104	108	104
query22	4717	4670	4592	4592
query23	34505	34054	33964	33964
query24	11201	2864	2851	2851
query25	678	410	422	410
query26	1196	165	162	162
query27	2493	302	301	301
query28	8178	2477	2437	2437
query29	874	437	448	437
query30	321	163	152	152
query31	1045	811	822	811
query32	103	57	58	57
query33	778	303	305	303
query34	960	495	493	493
query35	913	721	730	721
query36	1111	952	956	952
query37	161	85	85	85
query38	3970	4014	3908	3908
query39	1493	1407	1446	1407
query40	208	98	101	98
query41	52	49	51	49
query42	120	99	97	97
query43	540	506	502	502
query44	1281	831	803	803
query45	202	167	167	167
query46	1137	734	719	719
query47	1967	1859	1872	1859
query48	475	383	403	383
query49	1136	410	431	410
query50	838	419	409	409
query51	7065	6910	6958	6910
query52	100	86	86	86
query53	270	189	182	182
query54	1382	486	478	478
query55	83	78	76	76
query56	289	279	283	279
query57	1284	1099	1112	1099
query58	248	243	240	240
query59	3363	3303	2856	2856
query60	331	290	291	290
query61	106	99	101	99
query62	895	676	651	651
query63	224	193	186	186
query64	5219	660	624	624
query65	3345	3242	3200	3200
query66	1452	307	302	302
query67	15854	15752	16054	15752
query68	4998	567	556	556
query69	576	302	307	302
query70	1129	1125	1012	1012
query71	346	274	303	274
query72	7054	4027	3944	3944
query73	776	343	358	343
query74	10477	9104	9066	9066
query75	3372	2656	2656	2656
query76	3260	865	931	865
query77	414	298	279	279
query78	11011	9826	9681	9681
query79	2522	596	621	596
query80	1322	465	463	463
query81	579	244	262	244
query82	679	146	136	136
query83	271	135	138	135
query84	289	73	78	73
query85	1015	300	285	285
query86	509	272	299	272
query87	4461	4376	4391	4376
query88	3768	2430	2401	2401
query89	407	288	289	288
query90	2185	190	195	190
query91	184	141	158	141
query92	73	49	51	49
query93	1960	565	558	558
query94	1127	307	295	295
query95	351	257	254	254
query96	631	288	287	287
query97	3247	3126	3151	3126
query98	227	201	207	201
query99	1586	1303	1306	1303
Total cold run time: 307368 ms
Total hot run time: 191680 ms

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

ClickBench: Total hot run time: 33.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f68ebe93d37d08ee4ecac84a1ca5e73e8a9619d, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.52	0.51
query6	1.14	0.74	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.51	0.51
query10	0.55	0.56	0.53
query11	0.13	0.10	0.11
query12	0.14	0.10	0.10
query13	0.61	0.60	0.60
query14	2.71	2.74	2.72
query15	0.90	0.82	0.82
query16	0.39	0.39	0.39
query17	1.08	1.04	1.07
query18	0.23	0.21	0.21
query19	1.98	1.84	2.05
query20	0.01	0.01	0.01
query21	15.38	0.60	0.59
query22	2.59	2.03	2.26
query23	17.16	0.81	1.00
query24	2.92	1.23	1.39
query25	0.21	0.14	0.07
query26	0.40	0.14	0.14
query27	0.04	0.04	0.04
query28	10.34	1.10	1.07
query29	12.56	3.27	3.25
query30	0.24	0.05	0.05
query31	2.89	0.39	0.39
query32	3.26	0.45	0.46
query33	2.96	3.00	3.05
query34	17.05	4.48	4.45
query35	4.55	4.56	4.52
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.66 s
Total hot run time: 33.22 s

freemandealer
freemandealer reviewed Sep 26, 2024
View reviewed changes
be/src/cloud/cloud_rowset_writer.cpp Outdated
// In cloud mode, this branch implies it is an intermediate rowset for external merge sort,
// we use `global_local_filesystem` to write data to `tmp_file_dir`(see `local_segment_path`).
_context.tablet_path = io::FileCacheFactory::instance()->get_cache_path();
if (auto maybe_cache_path = io::FileCacheFactory::instance()->get_cache_path();
Copy link
Contributor

@freemandealer freemandealer Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so cache path will be used even if enable_file_cache = false ? This will increase user's cognitive load somehow. A new path should be used in my opinion.

FYI, with more Cache Storage being added to Doris, disk file system is not the only place for cached data, e.g. system RAM for Memory-based Cache Storage. In this case, cache path will have value but not a directory.

Copy link
Collaborator Author

@Hastyshell Hastyshell Sep 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We enable file cache by false after duscussion. As for the availability of cache path, currently every cache has a working disk path. If mem cache is introduced, we should modify the implementation of FileCacheFactory then.

@Hastyshell Hastyshell force-pushed the fix-cloud-sc-filecache branch 5 times, most recently from 6cd5504 to 4887f68 Compare September 26, 2024 13:46
@Hastyshell Hastyshell changed the title [fix](cloud-rowset-write) Reject to create a temp local rowset writer if file cache disabled [chore](file-cache) Enable file cache for cloud mode by force Sep 26, 2024
@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 26, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TeamCity be ut coverage result:
Function Coverage: 37.31% (9635/25823)
Line Coverage: 28.71% (79710/277657)
Region Coverage: 28.13% (41219/146505)
Branch Coverage: 24.76% (20988/84764)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4887f68c30c939e210b70ce2cfce33c9a1a0f09f_4887f68c30c939e210b70ce2cfce33c9a1a0f09f/report/index.html

freemandealer
freemandealer approved these changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@freemandealer freemandealer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring previously approved these changes Sep 26, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 26, 2024

PR approved by at least one committer and no changes requested.

@Hastyshell Hastyshell force-pushed the fix-cloud-sc-filecache branch from 4887f68 to 8f160b7 Compare September 26, 2024 14:57
@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 26, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 26, 2024
@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TPC-H: Total hot run time: 40906 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8f160b769f2d85f1bb1f4babef404a1182f9a9e4, data reload: false

------ Round 1 ----------------------------------
q1	17782	7539	7259	7259
q2	2043	296	274	274
q3	11951	1071	1198	1071
q4	10488	782	711	711
q5	7776	2892	2838	2838
q6	239	154	153	153
q7	995	599	613	599
q8	9359	1923	2037	1923
q9	6601	6445	6453	6445
q10	6995	2303	2353	2303
q11	441	252	250	250
q12	413	225	215	215
q13	17773	2981	2971	2971
q14	247	224	217	217
q15	576	526	522	522
q16	636	577	598	577
q17	984	554	624	554
q18	7425	6689	6732	6689
q19	1390	1015	1038	1015
q20	494	206	209	206
q21	3959	3130	3323	3130
q22	1132	984	1006	984
Total cold run time: 109699 ms
Total hot run time: 40906 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7338	7300	7280	7280
q2	329	232	232	232
q3	3026	2929	3048	2929
q4	2142	1865	1784	1784
q5	5745	5785	5785	5785
q6	233	142	142	142
q7	2285	1864	1805	1805
q8	3394	3569	3453	3453
q9	8952	8981	8922	8922
q10	3599	3569	3536	3536
q11	589	490	496	490
q12	842	623	619	619
q13	9976	3199	3212	3199
q14	308	268	270	268
q15	576	537	514	514
q16	692	654	640	640
q17	1850	1626	1562	1562
q18	8314	7798	7656	7656
q19	1696	1501	1577	1501
q20	2151	1889	1887	1887
q21	5586	5388	5458	5388
q22	1177	1078	1030	1030
Total cold run time: 70800 ms
Total hot run time: 60622 ms

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TPC-DS: Total hot run time: 192204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8f160b769f2d85f1bb1f4babef404a1182f9a9e4, data reload: false

query1	908	395	409	395
query2	6247	2052	2049	2049
query3	8691	196	206	196
query4	33629	23561	23409	23409
query5	3674	467	476	467
query6	276	167	162	162
query7	4187	318	310	310
query8	273	217	217	217
query9	9306	2697	2678	2678
query10	477	294	294	294
query11	17858	15240	15100	15100
query12	156	103	102	102
query13	1535	425	413	413
query14	9724	7012	7138	7012
query15	254	163	175	163
query16	7576	476	470	470
query17	1620	618	598	598
query18	2050	350	315	315
query19	313	157	158	157
query20	118	115	110	110
query21	219	109	107	107
query22	4908	4622	4708	4622
query23	35159	34316	33874	33874
query24	10828	2913	2883	2883
query25	612	411	420	411
query26	721	167	175	167
query27	2041	310	301	301
query28	6413	2446	2410	2410
query29	715	440	444	440
query30	252	153	152	152
query31	1045	767	824	767
query32	98	56	56	56
query33	730	309	311	309
query34	911	503	505	503
query35	908	755	733	733
query36	1118	956	951	951
query37	150	88	94	88
query38	4029	3957	3880	3880
query39	1482	1442	1423	1423
query40	216	100	96	96
query41	51	50	49	49
query42	119	95	93	93
query43	532	461	487	461
query44	1205	812	793	793
query45	197	166	185	166
query46	1127	728	719	719
query47	1928	1851	1845	1845
query48	464	389	374	374
query49	915	401	456	401
query50	850	417	429	417
query51	7174	6945	6932	6932
query52	101	87	89	87
query53	254	187	190	187
query54	1301	479	484	479
query55	80	73	83	73
query56	284	268	272	268
query57	1203	1115	1090	1090
query58	253	235	255	235
query59	3112	2985	2940	2940
query60	312	277	280	277
query61	105	105	105	105
query62	863	681	656	656
query63	220	188	188	188
query64	3655	643	618	618
query65	3324	3241	3196	3196
query66	818	305	312	305
query67	15756	15892	15723	15723
query68	4524	581	560	560
query69	572	303	303	303
query70	1239	1126	1126	1126
query71	397	264	284	264
query72	7834	4048	4068	4048
query73	766	347	351	347
query74	10326	9010	9068	9010
query75	4277	2666	2660	2660
query76	3735	955	909	909
query77	608	293	297	293
query78	10783	9734	9619	9619
query79	3653	605	596	596
query80	2058	454	458	454
query81	564	240	240	240
query82	666	140	138	138
query83	320	159	132	132
query84	282	77	76	76
query85	1549	315	288	288
query86	433	286	304	286
query87	4429	4394	4319	4319
query88	4122	2394	2378	2378
query89	407	290	288	288
query90	2096	195	189	189
query91	176	142	140	140
query92	69	48	49	48
query93	3036	553	553	553
query94	1118	296	302	296
query95	365	262	263	262
query96	625	286	283	283
query97	3241	3146	3148	3146
query98	210	206	196	196
query99	1548	1307	1290	1290
Total cold run time: 303658 ms
Total hot run time: 192204 ms

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

TeamCity be ut coverage result:
Function Coverage: 37.30% (9633/25823)
Line Coverage: 28.70% (79705/277672)
Region Coverage: 28.13% (41211/146513)
Branch Coverage: 24.76% (20990/84770)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8f160b769f2d85f1bb1f4babef404a1182f9a9e4_8f160b769f2d85f1bb1f4babef404a1182f9a9e4/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 26, 2024

ClickBench: Total hot run time: 32.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8f160b769f2d85f1bb1f4babef404a1182f9a9e4, data reload: false

query1	0.05	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.53	0.50	0.49
query6	1.12	0.75	0.73
query7	0.02	0.01	0.01
query8	0.05	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.57	0.54
query11	0.13	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.58	0.59
query14	2.72	2.68	2.80
query15	0.91	0.83	0.82
query16	0.38	0.38	0.39
query17	1.05	1.05	1.07
query18	0.20	0.20	0.20
query19	1.98	1.88	2.04
query20	0.01	0.01	0.01
query21	15.35	0.58	0.58
query22	2.58	1.92	2.26
query23	16.80	1.12	0.77
query24	2.38	1.07	0.68
query25	0.16	0.21	0.16
query26	0.34	0.14	0.15
query27	0.04	0.05	0.04
query28	11.36	1.08	1.07
query29	12.54	3.21	3.18
query30	0.25	0.06	0.06
query31	2.88	0.37	0.36
query32	3.29	0.46	0.46
query33	2.98	3.07	3.01
query34	16.84	4.44	4.56
query35	4.57	4.47	4.51
query36	0.65	0.49	0.48
query37	0.08	0.06	0.05
query38	0.04	0.04	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.38 s
Total hot run time: 32.46 s

@Hastyshell Hastyshell force-pushed the fix-cloud-sc-filecache branch from 8f160b7 to cb4840c Compare September 27, 2024 04:49
@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 27, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 27, 2024

TeamCity be ut coverage result:
Function Coverage: 37.29% (9626/25811)
Line Coverage: 28.70% (79660/277594)
Region Coverage: 28.13% (41187/146438)
Branch Coverage: 24.75% (20974/84750)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cb4840c1b07f2bbfda6c893354fb61f76c04b8ca_cb4840c1b07f2bbfda6c893354fb61f76c04b8ca/report/index.html

@Hastyshell Hastyshell force-pushed the fix-cloud-sc-filecache branch from cb4840c to fda89e2 Compare September 27, 2024 08:52
@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 27, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Sep 27, 2024

TPC-H: Total hot run time: 40878 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fda89e2b729e40947165d7a2f46a719da11678a6, data reload: false

------ Round 1 ----------------------------------
q1	17564	7353	7242	7242
q2	2015	285	271	271
q3	12196	1075	1228	1075
q4	10566	753	756	753
q5	7775	2861	2862	2861
q6	239	152	155	152
q7	958	599	610	599
q8	9353	1927	1929	1927
q9	6576	6468	6386	6386
q10	6960	2321	2320	2320
q11	441	254	250	250
q12	405	214	219	214
q13	17784	2982	2965	2965
q14	244	211	214	211
q15	561	537	533	533
q16	638	590	573	573
q17	973	662	523	523
q18	7163	6750	6699	6699
q19	1407	986	1090	986
q20	483	199	202	199
q21	4008	3160	3130	3130
q22	1093	1025	1009	1009
Total cold run time: 109402 ms
Total hot run time: 40878 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7222	7252	7189	7189
q2	326	223	237	223
q3	2932	2884	2974	2884
q4	2051	1809	1870	1809
q5	5777	5750	5739	5739
q6	229	150	149	149
q7	2247	1830	1822	1822
q8	3366	3539	3483	3483
q9	8890	8916	8919	8916
q10	3580	3525	3511	3511
q11	578	487	482	482
q12	825	636	620	620
q13	10963	3168	3140	3140
q14	306	286	283	283
q15	580	528	529	528
q16	702	663	648	648
q17	1862	1652	1612	1612
q18	8392	7865	7420	7420
q19	1681	1426	1446	1426
q20	2145	1874	1878	1874
q21	5538	5382	5375	5375
q22	1153	1064	1075	1064
Total cold run time: 71345 ms
Total hot run time: 60197 ms

@doris-robot
Copy link

doris-robot commented Sep 27, 2024

TeamCity be ut coverage result:
Function Coverage: 37.30% (9627/25811)
Line Coverage: 28.70% (79668/277598)
Region Coverage: 28.14% (41203/146439)
Branch Coverage: 24.75% (20979/84750)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fda89e2b729e40947165d7a2f46a719da11678a6_fda89e2b729e40947165d7a2f46a719da11678a6/report/index.html

@doris-robot
Copy link

doris-robot commented Sep 27, 2024

TPC-DS: Total hot run time: 192183 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fda89e2b729e40947165d7a2f46a719da11678a6, data reload: false

query1	891	398	415	398
query2	6248	2182	2109	2109
query3	8682	208	201	201
query4	33647	23457	23587	23457
query5	3773	489	488	488
query6	271	171	166	166
query7	4189	297	297	297
query8	287	215	215	215
query9	9525	2650	2654	2650
query10	471	277	283	277
query11	17799	15454	15073	15073
query12	156	98	100	98
query13	1530	427	424	424
query14	9305	7380	7320	7320
query15	245	168	170	168
query16	7819	472	427	427
query17	1676	645	595	595
query18	2133	317	314	314
query19	357	160	163	160
query20	123	118	117	117
query21	211	111	103	103
query22	4880	4840	4886	4840
query23	34806	34115	34110	34110
query24	10950	2791	2855	2791
query25	617	405	417	405
query26	1114	161	161	161
query27	2238	294	300	294
query28	7467	2429	2393	2393
query29	823	433	439	433
query30	264	156	152	152
query31	1023	804	796	796
query32	96	55	59	55
query33	765	299	325	299
query34	917	497	493	493
query35	894	745	745	745
query36	1103	966	946	946
query37	158	92	87	87
query38	4083	3849	3895	3849
query39	1511	1453	1418	1418
query40	208	96	101	96
query41	53	48	49	48
query42	124	95	93	93
query43	539	498	495	495
query44	1222	810	790	790
query45	199	167	170	167
query46	1149	713	721	713
query47	1945	1867	1870	1867
query48	463	371	382	371
query49	926	410	431	410
query50	811	404	428	404
query51	7037	6844	7001	6844
query52	100	88	87	87
query53	255	192	183	183
query54	1197	463	485	463
query55	82	73	76	73
query56	288	278	310	278
query57	1196	1121	1090	1090
query58	232	227	284	227
query59	3215	2997	2912	2912
query60	298	260	282	260
query61	105	104	103	103
query62	860	680	659	659
query63	212	182	183	182
query64	3930	657	620	620
query65	3254	3209	3199	3199
query66	859	303	319	303
query67	16190	15555	15493	15493
query68	4747	567	554	554
query69	506	295	293	293
query70	1104	1077	1106	1077
query71	360	280	270	270
query72	7295	4054	4238	4054
query73	773	341	347	341
query74	10437	8982	8970	8970
query75	3472	2637	2691	2637
query76	2992	985	946	946
query77	620	289	286	286
query78	10568	9644	9593	9593
query79	3644	612	580	580
query80	2758	443	452	443
query81	593	236	235	235
query82	696	136	143	136
query83	315	148	136	136
query84	288	80	80	80
query85	1742	303	293	293
query86	407	292	299	292
query87	4466	4379	4250	4250
query88	4254	2369	2363	2363
query89	407	290	290	290
query90	2067	189	186	186
query91	176	141	147	141
query92	62	48	47	47
query93	2754	553	542	542
query94	1145	300	288	288
query95	356	253	248	248
query96	601	286	279	279
query97	3273	3089	3129	3089
query98	224	200	192	192
query99	1548	1298	1331	1298
Total cold run time: 304646 ms
Total hot run time: 192183 ms

@doris-robot
Copy link

doris-robot commented Sep 27, 2024

ClickBench: Total hot run time: 32.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fda89e2b729e40947165d7a2f46a719da11678a6, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.02	0.02
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.53	0.52	0.50
query6	1.14	0.73	0.72
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.56	0.51	0.50
query10	0.56	0.59	0.55
query11	0.14	0.11	0.10
query12	0.13	0.10	0.11
query13	0.61	0.60	0.59
query14	2.71	2.71	2.83
query15	0.89	0.83	0.82
query16	0.38	0.38	0.38
query17	0.98	1.01	1.07
query18	0.19	0.19	0.20
query19	1.97	1.82	1.90
query20	0.02	0.01	0.01
query21	15.37	0.62	0.61
query22	2.18	2.09	2.00
query23	16.99	0.92	0.87
query24	3.16	0.79	1.48
query25	0.19	0.21	0.23
query26	0.30	0.15	0.15
query27	0.04	0.05	0.05
query28	10.67	1.08	1.06
query29	12.56	3.23	3.24
query30	0.24	0.06	0.05
query31	2.87	0.38	0.37
query32	3.29	0.46	0.46
query33	2.98	3.02	3.00
query34	17.14	4.44	4.48
query35	4.48	4.50	4.50
query36	0.65	0.48	0.48
query37	0.08	0.06	0.05
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 106.4 s
Total hot run time: 32.81 s

@Hastyshell
Copy link
Collaborator Author

Hastyshell commented Sep 27, 2024

run external

@Hastyshell Hastyshell mentioned this pull request Sep 27, 2024
Merged
dataroaring pushed a commit that referenced this pull request Sep 29, 2024
@Hastyshell
[chore](file-cache) Enable file cache for cloud mode by force (#41357) (
7a75c75 
#41421)

## Proposed changes

As title.

pick:#41357
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 29, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Sep 29, 2024

PR approved by at least one committer and no changes requested.

@hello-stephen hello-stephen merged commit cf5a159 into apache:master Sep 29, 2024
cjj2010 pushed a commit to cjj2010/doris that referenced this pull request Oct 12, 2024
@Hastyshell @cjj2010
[chore](file-cache) Enable file cache for cloud mode by force (apache…
bd1dcf0 
…#41357)

## Proposed changes

Temp local rowset writer for external sorting replies on file cache.
amorynan pushed a commit to amorynan/doris that referenced this pull request Oct 12, 2024
@Hastyshell
[chore](file-cache) Enable file cache for cloud mode by force (apache…
1c52d0b 
…#41357)

## Proposed changes

Temp local rowset writer for external sorting replies on file cache.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring left review comments

+1 more reviewer

@freemandealer freemandealer freemandealer approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.2-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@Hastyshell @doris-robot @freemandealer @gavinchou @dataroaring @hello-stephen