[fix](memory) Fix task repeat attach task DCHECK failed #32784

xinyiZzz · 2024-03-25T10:00:57Z

Proposed changes

EngineChecksumTask repeats attach task

Check failed: thread_mem_tracker()->label() == "Orphan" , attach mem tracker label: EngineChecksumTask#tabletId=10162
*** Check failure stack trace: ***
    @     0x560c4647daad  google::LogMessage::Fail()
    @     0x560c4647ffe9  google::LogMessage::SendToLog()
    @     0x560c4647d616  google::LogMessage::Flush()
    @     0x560c46480659  google::LogMessageFatal::~LogMessageFatal()
    @     0x560c20f6d257  doris::ThreadContext::attach_task()
    @     0x560c20f68019  doris::AttachTask::AttachTask()
    @     0x560c206473b9  doris::EngineChecksumTask::execute()
    @     0x560c1dfaf7c5  doris::Compaction::do_compaction()
    @     0x560c203c60c2  doris::CumulativeCompaction::execute_compact_impl()
    @     0x560c1dfaef7d  doris::Compaction::execute_compact()
    @     0x560c202c5a5a  doris::Tablet::execute_compaction()
    @     0x560c1dd7c4b0  doris::StorageEngine::_submit_compaction_task()::$_0::operator()()

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-03-25T10:01:02Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

xinyiZzz · 2024-03-25T10:01:07Z

run buildall

github-actions · 2024-03-25T10:06:23Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-25T10:45:42Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8733/24769)
Line Coverage: 27.06% (71527/264339)
Region Coverage: 26.30% (37112/141123)
Branch Coverage: 23.19% (18980/81834)
Coverage Report: http://coverage.selectdb-in.cc/coverage/35e451ab1a73a90aacd193d65cbc837f1042ad3e_35e451ab1a73a90aacd193d65cbc837f1042ad3e/report/index.html

doris-robot · 2024-03-25T10:56:23Z

TPC-H: Total hot run time: 38056 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35e451ab1a73a90aacd193d65cbc837f1042ad3e, data reload: false

------ Round 1 ----------------------------------
q1	17635	4434	4137	4137
q2	2111	159	149	149
q3	10581	1136	1222	1136
q4	10225	690	762	690
q5	7463	3082	3023	3023
q6	200	125	126	125
q7	1054	601	577	577
q8	9340	2047	2014	2014
q9	7261	6634	6565	6565
q10	8459	3401	3643	3401
q11	436	234	227	227
q12	470	203	203	203
q13	17789	2903	2870	2870
q14	240	218	204	204
q15	505	460	463	460
q16	503	373	379	373
q17	952	557	641	557
q18	7165	6480	6428	6428
q19	3027	1450	1425	1425
q20	549	248	257	248
q21	3700	2991	2940	2940
q22	338	309	304	304
Total cold run time: 110003 ms
Total hot run time: 38056 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4112	4123	4067	4067
q2	328	233	235	233
q3	2977	2900	2845	2845
q4	1868	1547	1533	1533
q5	5360	5361	5380	5361
q6	196	118	122	118
q7	2288	1862	1898	1862
q8	3173	3326	3343	3326
q9	8734	8724	8695	8695
q10	3783	3795	3764	3764
q11	553	463	441	441
q12	719	576	557	557
q13	16918	2839	2874	2839
q14	275	243	257	243
q15	500	458	450	450
q16	468	419	414	414
q17	1728	1512	1482	1482
q18	7443	7147	7222	7147
q19	1636	1540	1559	1540
q20	1919	1726	1731	1726
q21	4840	4625	4554	4554
q22	538	452	448	448
Total cold run time: 70356 ms
Total hot run time: 53645 ms

doris-robot · 2024-03-25T11:08:34Z

TPC-DS: Total hot run time: 188610 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35e451ab1a73a90aacd193d65cbc837f1042ad3e, data reload: false

query1	955	381	381	381
query2	6554	2063	1948	1948
query3	6704	205	207	205
query4	31877	21545	21605	21545
query5	4359	417	403	403
query6	280	175	181	175
query7	4633	294	292	292
query8	233	173	175	173
query9	9331	2290	2294	2290
query10	575	254	269	254
query11	17408	14321	14491	14321
query12	136	90	91	90
query13	1637	433	412	412
query14	13201	12370	12400	12370
query15	262	202	200	200
query16	8092	265	256	256
query17	1990	588	573	573
query18	1856	315	292	292
query19	367	158	161	158
query20	90	86	91	86
query21	202	135	142	135
query22	4987	4734	4763	4734
query23	33712	32805	32714	32714
query24	10747	2857	2858	2857
query25	606	384	401	384
query26	1133	161	162	161
query27	2839	373	372	372
query28	7195	1865	1861	1861
query29	901	691	646	646
query30	306	162	161	161
query31	981	738	745	738
query32	96	59	60	59
query33	769	269	265	265
query34	1024	492	500	492
query35	866	631	649	631
query36	1042	886	909	886
query37	124	82	80	80
query38	3587	3432	3446	3432
query39	1482	1461	1430	1430
query40	208	118	114	114
query41	52	48	50	48
query42	104	98	99	98
query43	530	467	476	467
query44	1243	739	714	714
query45	287	273	263	263
query46	1142	709	723	709
query47	1909	1820	1824	1820
query48	460	356	350	350
query49	1106	345	343	343
query50	779	379	384	379
query51	6787	6695	6678	6678
query52	115	89	99	89
query53	350	282	287	282
query54	316	266	248	248
query55	87	84	81	81
query56	246	234	230	230
query57	1226	1132	1131	1131
query58	246	225	223	223
query59	2915	2772	2719	2719
query60	288	252	257	252
query61	119	118	114	114
query62	642	439	441	439
query63	317	286	293	286
query64	5859	4145	4143	4143
query65	3137	3068	3070	3068
query66	846	376	368	368
query67	15568	14917	14878	14878
query68	7072	514	524	514
query69	622	386	380	380
query70	1234	1145	1088	1088
query71	530	275	266	266
query72	6753	2715	2563	2563
query73	742	318	310	310
query74	6658	6286	6446	6286
query75	4282	2992	2921	2921
query76	4926	925	876	876
query77	789	263	263	263
query78	11286	10284	10187	10187
query79	10067	530	537	530
query80	1361	380	380	380
query81	548	225	233	225
query82	1112	218	218	218
query83	204	145	145	145
query84	292	83	79	79
query85	1521	321	317	317
query86	470	306	316	306
query87	3767	3553	3545	3545
query88	5014	2274	2280	2274
query89	520	369	378	369
query90	2005	180	180	180
query91	175	135	136	135
query92	58	51	49	49
query93	7148	504	485	485
query94	1265	176	178	176
query95	429	331	347	331
query96	624	265	272	265
query97	3113	2922	2873	2873
query98	239	212	203	203
query99	1203	875	900	875
Total cold run time: 316052 ms
Total hot run time: 188610 ms

doris-robot · 2024-03-25T11:11:48Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 35e451ab1a73a90aacd193d65cbc837f1042ad3e with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

xinyiZzz · 2024-03-25T16:04:10Z

run buildall

github-actions · 2024-03-25T16:09:45Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2024-03-25T23:02:54Z

run buildall

yiguolei · 2024-03-25T23:03:51Z

run buildall

github-actions · 2024-03-25T23:08:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-03-25T23:08:04Z

PR approved by anyone and no changes requested.

github-actions · 2024-03-25T23:09:16Z

clang-tidy review says "All clean, LGTM! 👍"

yiguolei · 2024-03-25T23:16:55Z

run p0

yiguolei · 2024-03-26T01:03:14Z

run buildall

doris-robot · 2024-03-26T02:14:54Z

TeamCity be ut coverage result:
Function Coverage: 35.26% (8736/24778)
Line Coverage: 27.04% (71533/264518)
Region Coverage: 26.29% (37110/141176)
Branch Coverage: 23.18% (18977/81872)
Coverage Report: http://coverage.selectdb-in.cc/coverage/be705be74f3fbbbaa0f790d3875e7c973e37758f_be705be74f3fbbbaa0f790d3875e7c973e37758f/report/index.html

github-actions · 2024-03-26T07:01:43Z

clang-tidy review says "All clean, LGTM! 👍"

xinyiZzz · 2024-03-26T07:12:51Z

run buildall

github-actions · 2024-03-26T07:18:36Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-03-26T08:08:22Z

TeamCity be ut coverage result:
Function Coverage: 35.27% (8738/24778)
Line Coverage: 27.05% (71556/264530)
Region Coverage: 26.29% (37124/141194)
Branch Coverage: 23.18% (18979/81874)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9149166e86ab3488e828dbe48ae5e781ff7a813b_9149166e86ab3488e828dbe48ae5e781ff7a813b/report/index.html

doris-robot · 2024-03-26T08:20:23Z

TPC-H: Total hot run time: 37687 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9149166e86ab3488e828dbe48ae5e781ff7a813b, data reload: false

------ Round 1 ----------------------------------
q1	17639	4423	4093	4093
q2	2117	159	151	151
q3	10600	1124	1148	1124
q4	10225	787	694	694
q5	7440	3044	3041	3041
q6	203	124	123	123
q7	1033	598	560	560
q8	9339	1995	1947	1947
q9	7333	6639	6546	6546
q10	8431	3474	3546	3474
q11	434	221	213	213
q12	413	202	193	193
q13	17805	2845	2875	2845
q14	245	212	206	206
q15	499	454	454	454
q16	514	376	366	366
q17	957	595	565	565
q18	7322	6532	6339	6339
q19	3227	1408	1450	1408
q20	553	251	244	244
q21	3546	2945	2808	2808
q22	348	293	294	293
Total cold run time: 110223 ms
Total hot run time: 37687 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4124	4097	4064	4064
q2	323	231	232	231
q3	2990	2840	2812	2812
q4	1847	1576	1533	1533
q5	5280	5338	5373	5338
q6	192	116	117	116
q7	2232	1848	1875	1848
q8	3142	3307	3281	3281
q9	8761	8645	8694	8645
q10	3801	3730	3731	3730
q11	549	450	446	446
q12	735	537	542	537
q13	16894	2861	2895	2861
q14	283	263	263	263
q15	495	460	452	452
q16	467	428	422	422
q17	1737	1502	1478	1478
q18	7558	7192	7123	7123
q19	1622	1514	1498	1498
q20	1908	1717	1715	1715
q21	4699	4746	4711	4711
q22	529	458	451	451
Total cold run time: 70168 ms
Total hot run time: 53555 ms

doris-robot · 2024-03-26T08:32:01Z

TPC-DS: Total hot run time: 180688 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9149166e86ab3488e828dbe48ae5e781ff7a813b, data reload: false

query1	937	369	348	348
query2	6525	1872	1918	1872
query3	6711	214	208	208
query4	31796	21299	21313	21299
query5	4334	399	390	390
query6	269	176	176	176
query7	4635	290	293	290
query8	230	185	181	181
query9	9139	2272	2246	2246
query10	584	250	258	250
query11	17071	14142	14231	14142
query12	143	91	84	84
query13	1640	436	421	421
query14	9976	6964	7506	6964
query15	241	199	185	185
query16	8042	266	261	261
query17	1961	587	547	547
query18	2055	282	278	278
query19	230	162	157	157
query20	101	84	86	84
query21	194	128	128	128
query22	5051	4802	4753	4753
query23	33492	32925	32680	32680
query24	10727	2871	2842	2842
query25	595	360	380	360
query26	1201	149	159	149
query27	2688	345	356	345
query28	7267	1891	1875	1875
query29	867	645	608	608
query30	307	147	148	147
query31	994	712	743	712
query32	94	56	54	54
query33	766	241	240	240
query34	1113	472	494	472
query35	826	627	599	599
query36	993	878	888	878
query37	121	63	65	63
query38	3478	3478	3451	3451
query39	1483	1432	1413	1413
query40	209	111	111	111
query41	47	47	43	43
query42	106	94	101	94
query43	490	449	452	449
query44	1180	723	719	719
query45	282	265	263	263
query46	1114	691	687	687
query47	1927	1844	1884	1844
query48	447	352	351	351
query49	1154	344	337	337
query50	756	364	363	363
query51	6838	6760	6766	6760
query52	108	88	89	88
query53	346	268	271	268
query54	298	223	227	223
query55	88	85	90	85
query56	240	226	219	219
query57	1221	1143	1140	1140
query58	238	211	204	204
query59	2700	2615	2432	2432
query60	254	244	239	239
query61	97	97	100	97
query62	664	464	442	442
query63	302	275	269	269
query64	5558	4062	3983	3983
query65	3052	2990	3025	2990
query66	860	365	369	365
query67	15045	14701	14675	14675
query68	6247	508	523	508
query69	588	376	367	367
query70	1212	1189	1156	1156
query71	491	267	259	259
query72	6397	2835	2687	2687
query73	728	313	317	313
query74	6865	6489	6577	6489
query75	3384	2225	2204	2204
query76	4353	885	958	885
query77	640	265	262	262
query78	10840	10159	10080	10080
query79	4233	530	529	529
query80	1876	391	395	391
query81	539	214	211	211
query82	1543	83	89	83
query83	347	158	150	150
query84	292	84	91	84
query85	1686	373	367	367
query86	488	303	288	288
query87	3700	3503	3456	3456
query88	4627	2386	2360	2360
query89	480	368	362	362
query90	1973	176	174	174
query91	177	148	162	148
query92	60	50	49	49
query93	4601	493	483	483
query94	1229	182	179	179
query95	446	337	333	333
query96	603	270	267	267
query97	2638	2546	2532	2532
query98	242	216	219	216
query99	1225	924	927	924
Total cold run time: 299248 ms
Total hot run time: 180688 ms

doris-robot · 2024-03-26T08:35:15Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9149166e86ab3488e828dbe48ae5e781ff7a813b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

wangbo

LGTM

[branch-2.1](memory) Fix CCR task repeat attach task DCHECK failed3 #33366

xinyiZzz force-pushed the 20240325_fix_checksum branch from 35e451a to e4fb60c Compare March 25, 2024 16:04

1

be705be

yiguolei force-pushed the 20240325_fix_checksum branch from e4fb60c to be705be Compare March 25, 2024 23:03

yiguolei approved these changes Mar 25, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 25, 2024

github-actions bot added the reviewed label Mar 25, 2024

Merge branch 'master' into 20240325_fix_checksum

998a304

Merge branch 'master' into 20240325_fix_checksum

9149166

hello-stephen approved these changes Mar 26, 2024

View reviewed changes

wangbo approved these changes Mar 26, 2024

View reviewed changes

yiguolei merged commit faa3052 into apache:master Mar 26, 2024

yiguolei added the 2.1.0-conflict label Mar 26, 2024

wm1581066 added dev/2.0.x p0_c labels Mar 27, 2024

xinyiZzz mentioned this pull request Mar 27, 2024

[branch-2.0](cherry-pick) Fix task repeat attach task DCHECK failed #32914

Merged

yiguolei added the dev/2.0.8-merged label Mar 27, 2024

wm1581066 removed the dev/2.0.x label Mar 27, 2024

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Apr 7, 2024

[fix](memory) Fix task repeat attach task DCHECK failed apache#32784

c698a1a

xinyiZzz mentioned this pull request Apr 7, 2024

[branch-2.1](memory) Fix CCR task repeat attach task DCHECK failed #33309

Merged

dataroaring pushed a commit that referenced this pull request Apr 8, 2024

[fix](memory) Fix task repeat attach task DCHECK failed #32784 (#33309)

b95211c

xinyiZzz added a commit to xinyiZzz/incubator-doris that referenced this pull request Apr 8, 2024

[fix](memory) Fix task repeat attach task DCHECK failed apache#32784

5fb3d08

xinyiZzz mentioned this pull request Apr 8, 2024

[branch-2.1](memory) Fix CCR task repeat attach task DCHECK failed(retry) #33343

Merged

dataroaring pushed a commit that referenced this pull request Apr 8, 2024

[fix](memory) Fix task repeat attach task DCHECK failed #32784 (#33343)

208079a

yiguolei pushed a commit that referenced this pull request Apr 8, 2024

[fix](memory) Fix task repeat attach task DCHECK failed #32784 (#33343)

d60d804

[branch-2.1](memory) Fix CCR task repeat attach task DCHECK failed3 #33366

yiguolei added dev/2.1.3-merged and removed 2.1.0-conflict labels Apr 11, 2024

[fix](memory) Fix task repeat attach task DCHECK failed #32784

[fix](memory) Fix task repeat attach task DCHECK failed #32784

Uh oh!

Conversation

xinyiZzz commented Mar 25, 2024

Proposed changes

Further comments

Uh oh!

doris-robot commented Mar 25, 2024

Uh oh!

xinyiZzz commented Mar 25, 2024

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

doris-robot commented Mar 25, 2024

Uh oh!

doris-robot commented Mar 25, 2024

Uh oh!

doris-robot commented Mar 25, 2024

Uh oh!

doris-robot commented Mar 25, 2024

Uh oh!

xinyiZzz commented Mar 25, 2024

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

yiguolei commented Mar 25, 2024

Uh oh!

yiguolei commented Mar 25, 2024

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

github-actions bot commented Mar 25, 2024

Uh oh!

yiguolei commented Mar 25, 2024

Uh oh!

yiguolei commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

github-actions bot commented Mar 26, 2024

Uh oh!

xinyiZzz commented Mar 26, 2024

Uh oh!

github-actions bot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

doris-robot commented Mar 26, 2024

Uh oh!

wangbo left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants