Skip to content

[chore](backup) limit the involved tablets in a backup job #39987 #40082

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
w41ter merged 1 commit into from
Aug 29, 2024
Merged

[chore](backup) limit the involved tablets in a backup job #39987 #40082

w41ter merged 1 commit into from
Aug 29, 2024

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Aug 29, 2024

cherry pick from #39987

@w41ter
[chore](backup) limit the involved tablets in a backup job (apache#39987
677aa49 
)

and to avoid FE OOM caused by saving too much metadata.

Assuming the average tablet size is 50MB, the default value of 300000
can support 14TB of data per backup job.

The docs PR: apache/doris-website#1056
@w41ter
Copy link
Contributor Author

w41ter commented Aug 29, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

TPC-H: Total hot run time: 51139 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 677aa491aac67dfe55ece87a4dfa99abbf53c823, data reload: false

------ Round 1 ----------------------------------
q1	17618	4543	4500	4500
q2	2040	191	155	155
q3	10212	1960	2060	1960
q4	10388	1319	1426	1319
q5	8892	3999	3973	3973
q6	254	128	124	124
q7	2076	1626	1605	1605
q8	9312	2887	2889	2887
q9	10784	10360	10449	10360
q10	8686	3600	3625	3600
q11	424	241	237	237
q12	498	301	302	301
q13	18364	4125	4163	4125
q14	360	333	342	333
q15	520	472	466	466
q16	711	567	562	562
q17	1186	998	1017	998
q18	7361	7019	7033	7019
q19	1759	1661	1579	1579
q20	542	304	298	298
q21	4703	4405	4339	4339
q22	505	402	399	399
Total cold run time: 117195 ms
Total hot run time: 51139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4518	4491	4505	4491
q2	335	232	228	228
q3	4299	4240	4245	4240
q4	2815	2842	2830	2830
q5	7101	6974	7043	6974
q6	259	121	123	121
q7	3244	2889	2851	2851
q8	4268	4420	4420	4420
q9	16995	16840	16876	16840
q10	4288	4298	4305	4298
q11	761	701	686	686
q12	1059	844	848	844
q13	6989	3926	3891	3891
q14	453	424	422	422
q15	512	466	468	466
q16	744	693	688	688
q17	3854	3885	3788	3788
q18	8955	8910	9040	8910
q19	1814	1810	1768	1768
q20	2463	2131	2098	2098
q21	8597	8542	8617	8542
q22	1056	931	958	931
Total cold run time: 85379 ms
Total hot run time: 80327 ms

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

TPC-DS: Total hot run time: 202392 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 677aa491aac67dfe55ece87a4dfa99abbf53c823, data reload: false

query1	915	389	417	389
query2	6555	2143	2233	2143
query3	6916	205	198	198
query4	19864	17983	18006	17983
query5	19738	6504	6458	6458
query6	300	218	228	218
query7	4160	298	307	298
query8	245	271	236	236
query9	3138	2709	2642	2642
query10	398	320	298	298
query11	11368	10747	10820	10747
query12	123	78	69	69
query13	5595	660	647	647
query14	17397	13801	13300	13300
query15	359	220	228	220
query16	6491	293	267	267
query17	1735	1467	870	870
query18	2317	428	421	421
query19	210	152	152	152
query20	80	81	79	79
query21	189	94	100	94
query22	5188	5086	5082	5082
query23	32371	31867	32066	31867
query24	7055	6469	6440	6440
query25	516	445	418	418
query26	528	163	161	161
query27	1925	299	296	296
query28	6058	2308	2259	2259
query29	2953	2763	2832	2763
query30	246	168	165	165
query31	902	717	738	717
query32	67	63	60	60
query33	400	253	247	247
query34	839	461	474	461
query35	1109	907	898	898
query36	1140	1095	1183	1095
query37	91	61	60	60
query38	3077	2924	2937	2924
query39	1376	1326	1309	1309
query40	208	97	98	97
query41	39	36	37	36
query42	86	89	86	86
query43	610	609	600	600
query44	1130	724	728	724
query45	245	235	234	234
query46	1231	947	988	947
query47	1744	1729	1792	1729
query48	970	655	669	655
query49	632	378	382	378
query50	857	607	643	607
query51	4813	4706	4643	4643
query52	84	76	87	76
query53	448	323	313	313
query54	2618	2460	2537	2460
query55	88	83	82	82
query56	232	227	207	207
query57	1252	1059	1236	1059
query58	220	207	202	202
query59	3557	3398	3251	3251
query60	219	207	205	205
query61	100	98	99	98
query62	834	456	448	448
query63	482	335	342	335
query64	2614	1557	1330	1330
query65	3610	3557	3546	3546
query66	800	387	381	381
query67	15384	16788	15853	15853
query68	8543	665	658	658
query69	582	355	357	355
query70	1590	1363	1353	1353
query71	408	311	317	311
query72	6519	3487	3525	3487
query73	742	325	318	318
query74	6305	5838	5895	5838
query75	5043	3623	3743	3623
query76	5084	1139	1129	1129
query77	850	260	250	250
query78	12441	11797	12880	11797
query79	10286	633	631	631
query80	1358	404	407	404
query81	495	232	232	232
query82	643	99	97	97
query83	172	137	138	137
query84	263	70	71	70
query85	984	313	321	313
query86	341	333	291	291
query87	3232	3010	3020	3010
query88	4588	2292	2286	2286
query89	375	291	306	291
query90	1895	200	196	196
query91	151	131	124	124
query92	57	51	55	51
query93	4405	566	596	566
query94	742	203	212	203
query95	1106	1081	1074	1074
query96	642	334	323	323
query97	6466	6386	6373	6373
query98	198	186	175	175
query99	2999	893	894	893
Total cold run time: 308546 ms
Total hot run time: 202392 ms

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

ClickBench: Total hot run time: 30.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 677aa491aac67dfe55ece87a4dfa99abbf53c823, data reload: false

query1	0.03	0.02	0.03
query2	0.08	0.03	0.02
query3	0.25	0.07	0.05
query4	1.76	0.07	0.07
query5	0.54	0.52	0.51
query6	1.24	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.51	0.49	0.48
query10	0.54	0.54	0.53
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.77	0.79
query15	0.77	0.75	0.75
query16	0.36	0.37	0.37
query17	1.00	1.01	1.02
query18	0.24	0.23	0.25
query19	1.93	1.85	1.86
query20	0.02	0.01	0.01
query21	15.48	0.58	0.59
query22	2.00	2.49	1.49
query23	17.07	1.21	1.10
query24	4.92	0.90	1.29
query25	0.37	0.06	0.06
query26	0.60	0.15	0.16
query27	0.05	0.05	0.03
query28	8.02	0.77	0.75
query29	12.62	2.27	2.34
query30	0.58	0.51	0.52
query31	2.81	0.37	0.38
query32	3.39	0.50	0.50
query33	3.07	3.05	3.08
query34	15.23	4.81	4.82
query35	4.82	4.86	4.87
query36	1.04	1.01	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.4 s
Total hot run time: 30.79 s

@doris-robot
Copy link

doris-robot commented Aug 29, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 677aa491aac67dfe55ece87a4dfa99abbf53c823 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

@w41ter w41ter merged commit 27adb46 into apache:branch-2.0 Aug 29, 2024
@w41ter w41ter deleted the pick_39987_to_origin_branch-2.0 branch August 29, 2024 03:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@w41ter @doris-robot