Skip to content

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Aug 18, 2024

pick (#39360)

When fetch stream load record from BE node, if can not find database, StreamLoadRecordMgr will throw exception and the remaining records will not be recorded in memory.

For example: Ten stream load records were pulled, and the database associated with the stream load of the first record was deleted by the user. Therefore, the pull will end, resulting in the remaining nine records not being consumed recorded in memory.

This pr do not throw exception but skip record when can not find database to solve this problem.

…t find database (#39360)

When fetch stream load record from BE node, if can not find database,
StreamLoadRecordMgr will throw exception and the remaining records will
not be recorded in memory.

For example: Ten stream load records were pulled, and the database
associated with the stream load of the first record was deleted by the
user. Therefore, the pull will end, resulting in the remaining nine
records not being consumed recorded in memory.

This pr do not throw exception but skip record when can not find
database to solve this problem.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Aug 18, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50295 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b272d95a4f3830baaf923361c1eb3fe2eebe8107, data reload: false

------ Round 1 ----------------------------------
q1	18071	4385	4347	4347
q2	2081	199	185	185
q3	10343	1911	1917	1911
q4	10321	1266	1358	1266
q5	8511	3956	3949	3949
q6	274	144	143	143
q7	2084	1641	1638	1638
q8	9320	2768	2748	2748
q9	10850	10380	10510	10380
q10	8686	3525	3479	3479
q11	449	270	283	270
q12	502	346	334	334
q13	18367	3983	4073	3983
q14	382	357	354	354
q15	552	503	510	503
q16	710	602	597	597
q17	1133	982	971	971
q18	7206	6827	6947	6827
q19	1760	1631	1540	1540
q20	560	355	326	326
q21	4457	4117	4139	4117
q22	533	446	427	427
Total cold run time: 117152 ms
Total hot run time: 50295 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4358	4320	4319	4319
q2	363	272	273	272
q3	4153	4109	4168	4109
q4	2782	2760	2744	2744
q5	7201	7165	7172	7165
q6	255	138	137	137
q7	3226	2808	2922	2808
q8	4407	4496	4494	4494
q9	17327	17177	16990	16990
q10	4279	4280	4286	4280
q11	754	698	739	698
q12	1050	879	891	879
q13	6722	3744	3738	3738
q14	489	450	450	450
q15	525	502	510	502
q16	756	729	744	729
q17	3888	3849	3887	3849
q18	8834	8745	8732	8732
q19	1752	1727	1696	1696
q20	2391	2191	2170	2170
q21	8400	8522	8564	8522
q22	1052	979	957	957
Total cold run time: 84964 ms
Total hot run time: 80240 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 210154 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b272d95a4f3830baaf923361c1eb3fe2eebe8107, data reload: false

query1	997	432	413	413
query2	7098	2452	2096	2096
query3	6945	232	226	226
query4	21475	18314	18266	18266
query5	19949	6815	6796	6796
query6	387	289	303	289
query7	4655	357	358	357
query8	512	464	492	464
query9	3327	2884	2855	2855
query10	523	380	382	380
query11	12221	11059	11029	11029
query12	159	114	111	111
query13	6027	760	759	759
query14	21632	13825	13787	13787
query15	386	257	246	246
query16	6524	334	311	311
query17	1584	1455	940	940
query18	2336	487	491	487
query19	247	185	191	185
query20	112	115	127	115
query21	249	146	146	146
query22	5155	5145	5005	5005
query23	32788	32132	31967	31967
query24	6906	6548	6527	6527
query25	579	502	473	473
query26	554	200	191	191
query27	1693	335	335	335
query28	6042	2445	2415	2415
query29	3039	2855	2615	2615
query30	287	212	209	209
query31	1012	842	865	842
query32	106	99	88	88
query33	494	344	348	344
query34	901	542	526	526
query35	1194	1051	1012	1012
query36	1405	1285	1253	1253
query37	117	91	98	91
query38	3151	2968	2958	2958
query39	1487	1452	1420	1420
query40	261	147	142	142
query41	158	156	156	156
query42	118	118	120	118
query43	707	606	617	606
query44	1192	780	791	780
query45	283	279	268	268
query46	1291	1014	1036	1014
query47	1874	1883	1784	1784
query48	1042	755	738	738
query49	840	603	590	590
query50	946	706	666	666
query51	4893	4735	4739	4735
query52	128	106	111	106
query53	508	384	377	377
query54	2699	2512	2556	2512
query55	107	107	100	100
query56	324	302	292	292
query57	1252	1247	1155	1155
query58	329	322	322	322
query59	3569	3369	3249	3249
query60	297	282	299	282
query61	163	161	162	161
query62	911	532	597	532
query63	537	399	408	399
query64	2406	1717	1590	1590
query65	3639	3602	3587	3587
query66	1229	824	833	824
query67	15479	14959	16290	14959
query68	9777	688	704	688
query69	639	395	392	392
query70	1771	1492	1570	1492
query71	487	365	368	365
query72	6726	3571	3574	3571
query73	790	365	374	365
query74	6381	5920	5939	5920
query75	5533	3779	3841	3779
query76	6087	1196	1239	1196
query77	1223	427	448	427
query78	12793	15624	12356	12356
query79	10090	671	693	671
query80	927	562	570	562
query81	535	274	276	274
query82	269	123	123	123
query83	246	219	224	219
query84	286	99	99	99
query85	842	427	416	416
query86	348	331	332	331
query87	3302	3039	3073	3039
query88	4536	2538	2563	2538
query89	412	341	332	332
query90	2060	245	258	245
query91	193	165	165	165
query92	89	87	87	87
query93	2018	582	576	576
query94	709	258	254	254
query95	1186	1119	1099	1099
query96	658	338	347	338
query97	6709	6541	6534	6534
query98	230	211	206	206
query99	2955	986	922	922
Total cold run time: 322244 ms
Total hot run time: 210154 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.68 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b272d95a4f3830baaf923361c1eb3fe2eebe8107, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.03
query3	0.25	0.04	0.04
query4	1.79	0.07	0.06
query5	0.54	0.53	0.53
query6	1.26	0.60	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.52	0.49	0.49
query10	0.56	0.55	0.55
query11	0.12	0.09	0.10
query12	0.13	0.10	0.10
query13	0.62	0.62	0.61
query14	0.78	0.80	0.80
query15	0.79	0.75	0.79
query16	0.37	0.38	0.37
query17	1.00	1.03	1.01
query18	0.25	0.25	0.25
query19	1.95	1.87	1.80
query20	0.02	0.02	0.02
query21	15.45	0.55	0.55
query22	2.04	2.22	1.62
query23	17.09	0.98	0.95
query24	5.78	1.33	1.72
query25	0.41	0.11	0.05
query26	0.72	0.17	0.16
query27	0.04	0.04	0.04
query28	6.43	0.75	0.81
query29	12.72	2.33	2.29
query30	0.73	0.79	0.75
query31	2.83	0.40	0.37
query32	3.35	0.50	0.50
query33	3.10	3.06	3.08
query34	15.28	4.82	4.82
query35	4.89	4.88	4.86
query36	1.08	1.03	1.03
query37	0.08	0.06	0.06
query38	0.05	0.04	0.05
query39	0.04	0.03	0.03
query40	0.19	0.16	0.17
query41	0.08	0.04	0.04
query42	0.04	0.03	0.04
query43	0.04	0.04	0.03
Total cold run time: 103.56 s
Total hot run time: 31.68 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b272d95a4f3830baaf923361c1eb3fe2eebe8107 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@sollhui
Copy link
Contributor Author

sollhui commented Aug 18, 2024

run beut

@dataroaring dataroaring merged commit a472ed3 into apache:branch-2.0 Aug 19, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Dec 11, 2024
…t find database (apache#39360) (apache#39531)

pick (apache#39360)

When fetch stream load record from BE node, if can not find database,
StreamLoadRecordMgr will throw exception and the remaining records will
not be recorded in memory.

For example: Ten stream load records were pulled, and the database
associated with the stream load of the first record was deleted by the
user. Therefore, the pull will end, resulting in the remaining nine
records not being consumed recorded in memory.

This pr do not throw exception but skip record when can not find
database to solve this problem.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants