Skip to content

Conversation

@swjtu-zhanglei
Copy link
Contributor

@swjtu-zhanglei swjtu-zhanglei commented Jul 10, 2024

  • When creating a backupJob with huge of tables in a database, it can cause backupJob editlog size over 2GB and bdbje will throw exception because of ByteBuffer overflow

pick #35653

Proposed changes

Issue Number: close #xxx

* When creating a backupJob with huge of tables in a database, it can
cause backupJob editlog size over 2GB and bdbje will throw exception
because of ByteBuffer overflow
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@swjtu-zhanglei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49548 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b88d6b60c755faf1a2552c2a204b72dfd5a7a3e6, data reload: false

------ Round 1 ----------------------------------
q1	17641	4377	4334	4334
q2	2097	163	150	150
q3	10458	1895	1906	1895
q4	10343	1245	1323	1245
q5	8862	3867	3884	3867
q6	256	125	125	125
q7	2041	1584	1560	1560
q8	9327	2724	2693	2693
q9	10483	10392	10160	10160
q10	8603	3500	3505	3500
q11	421	245	250	245
q12	465	304	301	301
q13	18372	3945	3997	3945
q14	357	322	328	322
q15	492	474	455	455
q16	674	570	570	570
q17	1132	992	986	986
q18	7182	6799	6895	6799
q19	1795	1616	1616	1616
q20	520	288	286	286
q21	4413	4085	4048	4048
q22	534	449	446	446
Total cold run time: 116468 ms
Total hot run time: 49548 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4287	4254	4289	4254
q2	325	229	223	223
q3	4159	4118	4105	4105
q4	2755	2725	2719	2719
q5	7095	7114	7050	7050
q6	234	117	118	117
q7	3251	2818	2832	2818
q8	4345	4450	4437	4437
q9	16726	16910	16676	16676
q10	4214	4240	4251	4240
q11	766	682	672	672
q12	1024	851	840	840
q13	7110	3762	3736	3736
q14	467	431	423	423
q15	502	468	455	455
q16	752	670	674	670
q17	3744	3954	3796	3796
q18	8783	8633	8730	8633
q19	1721	1696	1638	1638
q20	2461	2104	2095	2095
q21	8355	8449	8366	8366
q22	1050	1022	973	973
Total cold run time: 84126 ms
Total hot run time: 78936 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203305 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b88d6b60c755faf1a2552c2a204b72dfd5a7a3e6, data reload: false

query1	934	416	381	381
query2	6562	2755	2684	2684
query3	6921	208	205	205
query4	20187	18055	17969	17969
query5	19720	6562	6485	6485
query6	301	219	228	219
query7	4147	296	307	296
query8	473	422	437	422
query9	3074	2642	2576	2576
query10	406	305	292	292
query11	11326	10750	10717	10717
query12	121	74	75	74
query13	5602	704	707	704
query14	17322	13253	13246	13246
query15	371	243	254	243
query16	6476	284	268	268
query17	1705	1466	866	866
query18	2322	404	405	404
query19	210	149	145	145
query20	76	82	84	82
query21	187	100	91	91
query22	5111	5025	4937	4937
query23	32416	31893	31882	31882
query24	7079	6515	6547	6515
query25	512	430	424	424
query26	518	162	158	158
query27	1838	297	294	294
query28	6072	2348	2324	2324
query29	2853	2911	2805	2805
query30	245	163	164	163
query31	881	753	748	748
query32	72	68	61	61
query33	406	264	261	261
query34	843	470	470	470
query35	1132	889	935	889
query36	1427	1320	1298	1298
query37	88	59	61	59
query38	3106	2924	2901	2901
query39	1368	1325	1329	1325
query40	209	91	93	91
query41	46	44	44	44
query42	78	85	77	77
query43	751	794	664	664
query44	1125	730	733	730
query45	246	237	237	237
query46	1229	967	962	962
query47	1748	1676	1764	1676
query48	998	706	704	704
query49	623	402	370	370
query50	856	604	598	598
query51	4730	4641	4620	4620
query52	101	79	79	79
query53	446	322	318	318
query54	2644	2456	2477	2456
query55	82	80	88	80
query56	233	217	199	199
query57	1183	1035	1048	1035
query58	219	219	201	201
query59	4301	4212	4349	4212
query60	215	193	202	193
query61	98	94	95	94
query62	854	469	487	469
query63	482	333	344	333
query64	2500	1564	1386	1386
query65	3660	3561	3557	3557
query66	776	388	391	388
query67	15935	15154	14974	14974
query68	10148	654	635	635
query69	581	347	355	347
query70	1787	1491	1303	1303
query71	396	314	309	309
query72	6544	3512	3486	3486
query73	733	324	311	311
query74	6316	5870	5806	5806
query75	5261	3759	3620	3620
query76	6608	1180	1193	1180
query77	1161	272	269	269
query78	12460	11787	12658	11787
query79	9727	656	645	645
query80	876	405	400	400
query81	489	233	241	233
query82	1440	101	103	101
query83	172	131	133	131
query84	258	70	73	70
query85	886	332	344	332
query86	347	304	340	304
query87	3210	3017	3026	3017
query88	4536	2316	2295	2295
query89	475	298	301	298
query90	1924	220	208	208
query91	174	138	142	138
query92	61	54	54	54
query93	5919	590	576	576
query94	697	209	207	207
query95	1106	1056	1062	1056
query96	642	331	319	319
query97	6477	6260	6424	6260
query98	193	175	171	171
query99	2744	896	807	807
Total cold run time: 315126 ms
Total hot run time: 203305 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b88d6b60c755faf1a2552c2a204b72dfd5a7a3e6, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.02
query3	0.24	0.05	0.05
query4	1.83	0.06	0.07
query5	0.53	0.52	0.52
query6	1.26	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.48	0.48
query10	0.54	0.54	0.52
query11	0.12	0.09	0.09
query12	0.11	0.08	0.08
query13	0.62	0.61	0.60
query14	0.78	0.78	0.78
query15	0.79	0.77	0.77
query16	0.35	0.36	0.36
query17	1.02	1.01	1.01
query18	0.21	0.25	0.24
query19	1.94	1.79	1.84
query20	0.01	0.01	0.01
query21	15.47	0.56	0.57
query22	2.09	2.72	1.79
query23	17.30	0.93	1.12
query24	4.84	1.05	1.78
query25	0.34	0.13	0.06
query26	0.56	0.16	0.15
query27	0.05	0.03	0.04
query28	7.68	0.73	0.73
query29	12.62	2.32	2.44
query30	0.58	0.53	0.54
query31	2.81	0.39	0.37
query32	3.40	0.50	0.50
query33	3.06	3.10	3.06
query34	15.25	4.83	4.79
query35	4.85	4.82	4.82
query36	1.05	1.01	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.38 s
Total hot run time: 31.01 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b88d6b60c755faf1a2552c2a204b72dfd5a7a3e6 with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@dataroaring dataroaring merged commit d085c5e into apache:branch-2.0 Jul 12, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
… (apache#37628)

* When creating a backupJob with huge of tables in a database, it can
cause backupJob editlog size over 2GB and bdbje will throw exception
because of ByteBuffer overflow

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants