Skip to content

Conversation

@Yukang-Lian
Copy link
Collaborator

@Yukang-Lian Yukang-Lian commented Mar 12, 2025

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

W20250311 19:24:08.770080  1257 cloud_full_compaction.cpp:158] fail to do CloudFullCompaction. res=[INTERNAL_ERROR]unknown reader type

        0#  doris::vectorized::RowSourcesBuffer::_create_buffer_file() at /root/tools/ldb-16/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/1
1/sstream:1054
        1#  doris::vectorized::RowSourcesBuffer::append(std::vector<doris::vectorized::RowSource, std::allocator<doris::vectorized::RowSource> > const&) 
at /root/selectdb-core/be/src/common/status.h:501
        2#  doris::vectorized::VerticalHeapMergeIterator::next_batch(doris::vectorized::Block*) at /root/selectdb-core/be/src/common/status.h:501
        3#  doris::vectorized::VerticalBlockReader::_unique_key_next_block(doris::vectorized::Block*, bool*) at /root/selectdb-core/be/src/common/status.
h:501
        4#  doris::vectorized::VerticalBlockReader::next_block_with_aggregation(doris::vectorized::Block*, bool*) at /root/tools/ldb-16/bin/../lib/gcc/x8
6_64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:907
        5#  doris::Merger::vertical_compact_one_group(std::shared_ptr<doris::BaseTablet>, doris::ReaderType, doris::TabletSchema const&, bool, std::vecto
r<unsigned int, std::allocator<unsigned int> > const&, doris::vectorized::RowSourcesBuffer*, std::vector<std::shared_ptr<doris::RowsetReader>, std::alloc
ator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, doris::Merger::Statistics*, std::vector<unsigned int, std::allocator<un
signed int> >, long, doris::CompactionSampleInfo*) at /root/selectdb-core/be/src/olap/merger.cpp:0
        6#  doris::Merger::vertical_merge_rowsets(std::shared_ptr<doris::BaseTablet>, doris::ReaderType, doris::TabletSchema const&, std::vector<std::sha
red_ptr<doris::RowsetReader>, std::allocator<std::shared_ptr<doris::RowsetReader> > > const&, doris::RowsetWriter*, long, long, doris::Merger::Statistics
*) at /root/selectdb-core/be/src/olap/merger.cpp:454
        7#  doris::Compaction::merge_input_rowsets() at /root/selectdb-core/be/src/olap/compaction.cpp:191
        8#  doris::CloudCompactionMixin::execute_compact_impl(long) at /root/selectdb-core/be/src/common/status.h:501
        9#  doris::CloudCompactionMixin::execute_compact() at /root/selectdb-core/be/src/common/status.h:501
        10# doris::CloudFullCompaction::execute_compact() at /root/selectdb-core/be/src/common/status.h:501
        11# std::_Function_handler<void (), doris::CloudStorageEngine::_submit_full_compaction_task(std::shared_ptr<doris::CloudTablet> const&)::$_0>::_M
_invoke(std::_Any_data const&) at /root/selectdb-core/be/src/common/status.h:501
        12# doris::ThreadPool::dispatch_thread() at /root/selectdb-core/be/src/util/threadpool.cpp:0
        13# doris::Thread::supervise_thread(void*) at /root/tools/ldb-16/bin/../usr/include/pthread.h:562
        14# ?
        15# clone

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 12, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it possible to add a regression test?

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2025
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32687 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b5129b43f2c142e9f72703fc115f34ad93425e1, data reload: false

------ Round 1 ----------------------------------
q1	17592	5318	5092	5092
q2	2043	292	163	163
q3	10424	1266	771	771
q4	10198	1055	558	558
q5	7539	2320	2363	2320
q6	186	164	133	133
q7	936	793	627	627
q8	9307	1290	1131	1131
q9	4967	4888	4889	4888
q10	6811	2322	1895	1895
q11	476	274	249	249
q12	355	374	219	219
q13	17768	3699	3056	3056
q14	223	226	212	212
q15	524	474	470	470
q16	648	611	615	611
q17	584	877	353	353
q18	6824	6380	6288	6288
q19	1218	936	550	550
q20	336	329	191	191
q21	2934	2210	1981	1981
q22	1025	1007	929	929
Total cold run time: 102918 ms
Total hot run time: 32687 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5153	5127	5141	5127
q2	253	338	229	229
q3	2174	2641	2303	2303
q4	1459	1819	1461	1461
q5	4265	4150	4166	4150
q6	206	164	127	127
q7	1871	1881	1830	1830
q8	2662	2597	2590	2590
q9	7396	7220	7192	7192
q10	2995	3256	2752	2752
q11	587	516	482	482
q12	716	741	590	590
q13	3544	3904	3255	3255
q14	284	334	282	282
q15	528	502	475	475
q16	660	704	660	660
q17	1157	1580	1381	1381
q18	7644	7524	7457	7457
q19	875	818	937	818
q20	1972	2046	1908	1908
q21	5579	4982	4805	4805
q22	1096	1024	984	984
Total cold run time: 53076 ms
Total hot run time: 50858 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192023 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b5129b43f2c142e9f72703fc115f34ad93425e1, data reload: false

query1	1400	1018	1004	1004
query2	6259	1919	1877	1877
query3	11003	4626	4589	4589
query4	56073	25828	23026	23026
query5	5234	492	476	476
query6	406	193	222	193
query7	5197	504	294	294
query8	328	249	241	241
query9	6935	2652	2663	2652
query10	433	317	252	252
query11	15325	15136	14935	14935
query12	169	118	107	107
query13	1249	526	406	406
query14	10726	6479	6602	6479
query15	198	205	195	195
query16	7041	664	491	491
query17	1099	718	560	560
query18	1563	415	302	302
query19	205	187	169	169
query20	125	129	114	114
query21	203	123	105	105
query22	4481	4412	4337	4337
query23	33864	33282	33369	33282
query24	5875	2433	2458	2433
query25	458	459	401	401
query26	670	279	158	158
query27	1781	534	341	341
query28	2778	2448	2514	2448
query29	578	560	465	465
query30	284	220	187	187
query31	892	888	825	825
query32	69	68	65	65
query33	462	368	310	310
query34	778	880	518	518
query35	815	849	771	771
query36	949	1031	886	886
query37	120	104	79	79
query38	4164	4183	4360	4183
query39	1531	1460	1444	1444
query40	217	133	104	104
query41	52	51	49	49
query42	121	110	105	105
query43	505	524	471	471
query44	1358	806	845	806
query45	186	180	167	167
query46	852	1048	649	649
query47	1850	1864	1798	1798
query48	394	433	316	316
query49	721	514	426	426
query50	704	771	427	427
query51	4282	4268	4215	4215
query52	117	111	111	111
query53	244	277	200	200
query54	491	530	425	425
query55	81	82	80	80
query56	280	290	289	289
query57	1152	1178	1100	1100
query58	248	230	253	230
query59	2766	2763	2706	2706
query60	293	286	276	276
query61	148	121	122	121
query62	733	755	673	673
query63	261	191	201	191
query64	1451	1037	670	670
query65	4535	4433	4458	4433
query66	710	417	293	293
query67	15699	15452	15394	15394
query68	7646	886	506	506
query69	559	306	259	259
query70	1190	1075	1073	1073
query71	493	306	265	265
query72	5703	3624	3818	3624
query73	1213	739	348	348
query74	9362	9010	8720	8720
query75	3668	3158	2683	2683
query76	4158	1199	764	764
query77	597	370	275	275
query78	9986	10364	9310	9310
query79	1899	829	590	590
query80	624	529	461	461
query81	500	256	222	222
query82	267	128	97	97
query83	177	167	152	152
query84	287	99	73	73
query85	822	358	307	307
query86	385	307	303	303
query87	4372	4516	4497	4497
query88	3552	2289	2290	2289
query89	417	305	280	280
query90	1887	215	216	215
query91	144	140	110	110
query92	77	56	52	52
query93	1223	1056	592	592
query94	600	417	273	273
query95	349	282	306	282
query96	483	565	282	282
query97	3274	3440	3299	3299
query98	224	211	206	206
query99	1356	1406	1289	1289
Total cold run time: 299665 ms
Total hot run time: 192023 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b5129b43f2c142e9f72703fc115f34ad93425e1, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.08	0.06
query4	1.62	0.10	0.10
query5	0.57	0.54	0.54
query6	1.19	0.71	0.73
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.53	0.51
query10	0.58	0.59	0.59
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	2.81	2.80	2.81
query15	0.92	0.86	0.85
query16	0.38	0.37	0.37
query17	0.99	1.02	1.06
query18	0.21	0.20	0.19
query19	1.87	1.77	1.99
query20	0.01	0.01	0.01
query21	15.36	0.87	0.57
query22	0.74	1.17	0.94
query23	14.69	1.35	0.61
query24	7.47	1.50	1.10
query25	0.49	0.23	0.08
query26	0.54	0.16	0.14
query27	0.05	0.04	0.04
query28	9.80	0.81	0.43
query29	12.52	3.94	3.26
query30	0.24	0.09	0.06
query31	2.82	0.59	0.37
query32	3.22	0.55	0.46
query33	3.04	3.01	3.00
query34	15.71	5.14	4.47
query35	4.50	4.51	4.54
query36	0.65	0.50	0.48
query37	0.08	0.06	0.07
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.16	0.13	0.14
query41	0.09	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.4 s
Total hot run time: 31.3 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.27% (12912/26750)
Line Coverage 37.63% (110280/293082)
Region Coverage 36.70% (56437/153776)
Branch Coverage 31.91% (28397/88988)

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32319 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0b5129b43f2c142e9f72703fc115f34ad93425e1, data reload: false

------ Round 1 ----------------------------------
q1	24393	5094	5042	5042
q2	2036	318	199	199
q3	10352	1298	693	693
q4	10226	1044	556	556
q5	7523	2444	2307	2307
q6	186	174	132	132
q7	914	729	600	600
q8	9294	1312	1114	1114
q9	5158	4862	4655	4655
q10	6820	2296	1884	1884
q11	483	292	257	257
q12	351	355	211	211
q13	17764	3636	3054	3054
q14	228	235	215	215
q15	526	480	490	480
q16	625	620	599	599
q17	559	874	338	338
q18	6921	6420	6331	6331
q19	1225	947	553	553
q20	320	337	212	212
q21	3054	2263	1897	1897
q22	1101	1007	990	990
Total cold run time: 110059 ms
Total hot run time: 32319 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5143	5032	5173	5032
q2	236	336	234	234
q3	2172	2687	2308	2308
q4	1462	1836	1394	1394
q5	4224	4086	4395	4086
q6	211	173	128	128
q7	2021	1970	1777	1777
q8	2679	2547	2558	2547
q9	7238	7156	7110	7110
q10	2989	3199	2747	2747
q11	588	517	493	493
q12	680	758	582	582
q13	3470	3864	3298	3298
q14	278	298	275	275
q15	517	473	461	461
q16	638	711	635	635
q17	1142	1584	1350	1350
q18	7837	7646	7520	7520
q19	803	806	879	806
q20	1997	2048	1840	1840
q21	5687	4924	4676	4676
q22	1077	1035	996	996
Total cold run time: 53089 ms
Total hot run time: 50295 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185220 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0b5129b43f2c142e9f72703fc115f34ad93425e1, data reload: false

query1	992	476	505	476
query2	6546	1934	1903	1903
query3	6795	223	219	219
query4	26489	23664	23268	23268
query5	4299	653	496	496
query6	290	203	193	193
query7	4611	501	313	313
query8	319	253	238	238
query9	8633	2589	2591	2589
query10	483	303	250	250
query11	15869	15412	14790	14790
query12	164	115	107	107
query13	1660	521	394	394
query14	11155	6350	6427	6350
query15	200	196	176	176
query16	7604	668	470	470
query17	1603	717	567	567
query18	1942	400	308	308
query19	246	185	157	157
query20	124	121	123	121
query21	212	123	105	105
query22	4151	4189	4013	4013
query23	33753	32967	33185	32967
query24	7493	2363	2383	2363
query25	514	465	422	422
query26	1236	264	155	155
query27	2546	475	331	331
query28	4311	2389	2404	2389
query29	702	574	418	418
query30	280	220	186	186
query31	926	875	760	760
query32	73	63	64	63
query33	561	339	314	314
query34	778	839	490	490
query35	781	856	728	728
query36	932	991	899	899
query37	114	95	74	74
query38	4216	4192	4057	4057
query39	1459	1411	1390	1390
query40	204	114	100	100
query41	58	52	51	51
query42	118	103	105	103
query43	483	523	485	485
query44	1273	798	789	789
query45	174	171	166	166
query46	849	1028	616	616
query47	1714	1818	1729	1729
query48	365	414	305	305
query49	796	498	438	438
query50	689	723	409	409
query51	4235	4213	4129	4129
query52	111	99	96	96
query53	225	252	191	191
query54	473	481	414	414
query55	79	78	84	78
query56	283	265	252	252
query57	1116	1145	1063	1063
query58	260	239	233	233
query59	2436	2695	2534	2534
query60	310	285	253	253
query61	126	118	115	115
query62	801	733	672	672
query63	224	191	189	189
query64	4225	1004	689	689
query65	4395	4326	4351	4326
query66	1052	401	303	303
query67	15609	15422	15447	15422
query68	8462	869	501	501
query69	474	302	259	259
query70	1186	1112	1029	1029
query71	474	290	269	269
query72	5610	3506	3731	3506
query73	793	710	346	346
query74	9008	9147	8789	8789
query75	3905	3158	2709	2709
query76	3678	1184	743	743
query77	793	369	370	369
query78	9909	10157	9392	9392
query79	2636	821	585	585
query80	700	531	462	462
query81	464	259	227	227
query82	248	126	95	95
query83	202	169	150	150
query84	278	94	72	72
query85	733	353	358	353
query86	329	277	287	277
query87	4398	4607	4275	4275
query88	2943	2303	2283	2283
query89	402	320	280	280
query90	2074	208	217	208
query91	137	134	107	107
query92	77	60	56	56
query93	1907	1057	584	584
query94	666	419	308	308
query95	358	274	265	265
query96	486	572	280	280
query97	3324	3392	3308	3308
query98	227	209	204	204
query99	1387	1403	1270	1270
Total cold run time: 276013 ms
Total hot run time: 185220 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0b5129b43f2c142e9f72703fc115f34ad93425e1, data reload: false

query1	0.04	0.04	0.03
query2	0.11	0.10	0.10
query3	0.25	0.19	0.20
query4	1.61	0.19	0.19
query5	0.59	0.58	0.58
query6	1.19	0.72	0.72
query7	0.03	0.01	0.02
query8	0.05	0.03	0.04
query9	0.59	0.52	0.52
query10	0.58	0.61	0.57
query11	0.16	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.81	2.70	2.71
query15	0.92	0.84	0.84
query16	0.38	0.38	0.38
query17	1.02	1.04	1.03
query18	0.21	0.19	0.21
query19	1.90	1.84	1.91
query20	0.01	0.01	0.01
query21	15.36	0.91	0.54
query22	0.77	1.23	0.64
query23	14.92	1.40	0.61
query24	6.42	2.97	0.56
query25	0.52	0.26	0.10
query26	0.46	0.15	0.15
query27	0.05	0.05	0.05
query28	10.51	0.85	0.43
query29	12.61	3.98	3.29
query30	0.24	0.09	0.06
query31	2.84	0.60	0.39
query32	3.22	0.55	0.45
query33	3.07	3.08	3.06
query34	15.87	5.14	4.54
query35	4.55	4.58	4.55
query36	0.68	0.49	0.50
query37	0.09	0.07	0.07
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.13
query41	0.09	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.78 s
Total hot run time: 30.97 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.79% (13067/26781)
Line Coverage 38.36% (112659/293668)
Region Coverage 37.16% (57279/154133)
Branch Coverage 32.27% (28797/89236)

@dataroaring dataroaring merged commit 80ee746 into apache:master Mar 25, 2025
32 of 35 checks passed
dataroaring pushed a commit that referenced this pull request Mar 26, 2025
…on size is too large #48958 (#49437)

Cherry-picked from #48958

Co-authored-by: abmdocrt <lianyukang@selectdb.com>
yiguolei pushed a commit that referenced this pull request Mar 29, 2025
…on size is too large #48958 (#49493)

Cherry-picked from #48958

Co-authored-by: abmdocrt <lianyukang@selectdb.com>
@gavinchou gavinchou mentioned this pull request Apr 23, 2025
@yiguolei yiguolei mentioned this pull request May 13, 2025
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.10-merged dev/3.0.5-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants