Skip to content

Conversation

@github-actions
Copy link
Contributor

Cherry-picked from #48313

Schema scanner runs on a separate thread which is executed
asynchronously. We should make sure all context used not be freed once
it is scheduled.

ERROR: AddressSanitizer: heap-buffer-overflow on address 0x613002f33eb2
at pc 0x55e085dccbe3 bp 0x7f345c0e1f10 sp 0x7f345c0e1f08
READ of size 1 at 0x613002f33eb2 thread T2776 (FragmentMgrAsyn)
#0 0x55e085dccbe2 in std::__atomic_base::load(std::memory_order) const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/atomic_base.h:481:9
#1 0x55e085dccbe2 in std::atomic::operator bool() const
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/atomic:87:22
#2 0x55e085dccbe2 in
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0::operator()()
const
/home/zcp/repo_center/doris_master/doris/be/src/exec/schema_scanner.cpp:118:5
#3 0x55e085dccbe2 in void std::__invoke_impl(std::__invoke_other,
doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14
#4 0x55e085dccbe2 in std::enable_if, void>::type
std::__invoke_r(doris::SchemaScanner::get_next_block_async(doris::RuntimeState*)::$_0&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:111:2
#5 0x55e085dccbe2 in std::_Function_handler::_M_invoke(std::_Any_data
const&)
/var/local/ldb-toolchain/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/std_function.h:291:9
#6 0x55e050f081ca in doris::ThreadPool::dispatch_thread()
/home/zcp/repo_center/doris_master/doris/be/src/util/threadpool.cpp:608:24
#7 0x55e050ede467 in doris::Thread::supervise_thread(void*)
/home/zcp/repo_center/doris_master/doris/be/src/util/thread.cpp:498:5
    #8 0x7f376ef5aac2 in start_thread nptl/pthread_create.c:442:8
#9 0x7f376efec84f misc/../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 26, 2025
@hello-stephen
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39998 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32842775861898fde6d310c8734d0bcfd8082960, data reload: false

------ Round 1 ----------------------------------
q1	17632	7012	6574	6574
q2	2064	181	165	165
q3	10680	1049	1150	1049
q4	10557	771	802	771
q5	7769	2840	2774	2774
q6	217	132	128	128
q7	957	603	603	603
q8	9340	1929	1985	1929
q9	6610	6414	6386	6386
q10	7051	2262	2293	2262
q11	464	264	260	260
q12	393	212	218	212
q13	17961	2960	3072	2960
q14	260	224	212	212
q15	523	473	466	466
q16	682	580	582	580
q17	972	587	545	545
q18	7224	6683	6752	6683
q19	1387	1102	1097	1097
q20	488	219	202	202
q21	3963	3166	3124	3124
q22	1100	1017	1016	1016
Total cold run time: 108294 ms
Total hot run time: 39998 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6798	6575	6574	6574
q2	324	239	235	235
q3	2894	2757	2964	2757
q4	2043	1859	1828	1828
q5	5719	5784	5753	5753
q6	207	127	123	123
q7	2210	1845	1787	1787
q8	3384	3527	3444	3444
q9	8930	8824	8913	8824
q10	3528	3541	3510	3510
q11	577	502	498	498
q12	813	614	602	602
q13	8754	3163	3253	3163
q14	299	270	255	255
q15	537	472	469	469
q16	697	685	675	675
q17	1828	1628	1613	1613
q18	8267	7903	7713	7713
q19	1706	1550	1545	1545
q20	2036	1843	1854	1843
q21	5461	5390	5327	5327
q22	1173	1088	1067	1067
Total cold run time: 68185 ms
Total hot run time: 59605 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197944 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32842775861898fde6d310c8734d0bcfd8082960, data reload: false

query1	1276	919	935	919
query2	6370	2120	2107	2107
query3	10976	4617	4472	4472
query4	60860	29909	23282	23282
query5	5141	464	446	446
query6	386	178	181	178
query7	5419	314	308	308
query8	308	208	209	208
query9	8645	2583	2576	2576
query10	453	277	262	262
query11	17975	15285	15674	15285
query12	158	110	111	110
query13	1459	461	418	418
query14	10486	6862	7624	6862
query15	203	189	183	183
query16	7036	484	519	484
query17	1138	615	608	608
query18	1544	328	327	327
query19	212	162	158	158
query20	120	107	114	107
query21	206	103	101	101
query22	5023	4668	4593	4593
query23	34760	33978	34317	33978
query24	6307	2998	2887	2887
query25	534	399	419	399
query26	666	170	167	167
query27	1930	360	358	358
query28	5230	2461	2431	2431
query29	692	447	434	434
query30	241	166	163	163
query31	1003	852	874	852
query32	69	54	55	54
query33	417	289	271	271
query34	919	515	511	511
query35	875	756	711	711
query36	1133	985	991	985
query37	126	69	68	68
query38	4135	3997	4114	3997
query39	1539	1478	1492	1478
query40	209	99	98	98
query41	48	48	47	47
query42	116	99	98	98
query43	552	503	507	503
query44	1196	831	832	831
query45	195	180	177	177
query46	1183	743	748	743
query47	2167	2011	2042	2011
query48	489	428	402	402
query49	757	409	389	389
query50	903	450	438	438
query51	7516	7572	7353	7353
query52	101	89	89	89
query53	261	187	189	187
query54	566	462	471	462
query55	85	81	78	78
query56	280	257	267	257
query57	1295	1160	1217	1160
query58	229	213	207	207
query59	3458	3267	3005	3005
query60	280	255	246	246
query61	110	108	124	108
query62	758	684	664	664
query63	213	187	192	187
query64	2115	765	782	765
query65	3271	3198	3197	3197
query66	748	310	319	310
query67	16136	15781	15677	15677
query68	4109	569	555	555
query69	430	277	276	276
query70	1176	1106	1044	1044
query71	375	267	263	263
query72	6510	4190	3981	3981
query73	747	348	343	343
query74	10500	9275	9135	9135
query75	3342	2674	2623	2623
query76	1967	1001	1069	1001
query77	488	271	272	271
query78	10543	9683	9650	9650
query79	2195	584	583	583
query80	1362	424	429	424
query81	522	244	236	236
query82	1228	89	85	85
query83	270	147	143	143
query84	282	78	77	77
query85	997	300	286	286
query86	398	302	295	295
query87	4457	4257	4192	4192
query88	3673	2416	2363	2363
query89	423	300	293	293
query90	1958	184	187	184
query91	185	150	150	150
query92	67	50	49	49
query93	2811	529	536	529
query94	799	286	296	286
query95	366	266	260	260
query96	619	281	281	281
query97	3317	3177	3192	3177
query98	212	201	192	192
query99	1561	1295	1290	1290
Total cold run time: 319096 ms
Total hot run time: 197944 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.48 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 32842775861898fde6d310c8734d0bcfd8082960, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.06
query4	1.62	0.10	0.10
query5	0.52	0.51	0.51
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.58	0.50	0.49
query10	0.55	0.56	0.57
query11	0.14	0.10	0.10
query12	0.16	0.11	0.11
query13	0.61	0.59	0.60
query14	2.72	2.75	2.82
query15	0.89	0.82	0.82
query16	0.37	0.41	0.39
query17	1.08	1.05	1.09
query18	0.23	0.22	0.22
query19	1.95	1.76	1.96
query20	0.02	0.02	0.01
query21	15.36	0.61	0.59
query22	2.30	2.36	2.36
query23	17.07	1.02	0.75
query24	2.72	1.86	0.80
query25	0.21	0.23	0.05
query26	0.52	0.14	0.15
query27	0.05	0.05	0.05
query28	10.26	0.46	0.46
query29	12.59	3.23	3.22
query30	0.24	0.06	0.06
query31	2.84	0.39	0.38
query32	3.26	0.48	0.45
query33	3.06	2.97	3.01
query34	17.13	4.56	4.54
query35	4.62	4.56	4.54
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 106.36 s
Total hot run time: 32.48 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit e83ddd0 into branch-3.0 Feb 27, 2025
22 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-48313-branch-3.0 branch February 27, 2025 09:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants