Skip to content

[fix](load) Fix the issue of insert into select concurrent reporting causing no error url #49669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Mar 31, 2025
Merged

[fix](load) Fix the issue of insert into select concurrent reporting causing no error url #49669

dataroaring merged 1 commit into from
Mar 31, 2025

Conversation

@liaoxin01
Copy link
Contributor

@liaoxin01 liaoxin01 commented Mar 30, 2025 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Fix the issue where concurrent fragment status reporting leads to missing error URLs. For example, when there are two fragments (one scan fragment and one sink fragment), if the sink fragment is canceled due to data quality issues, the scan fragment will also be canceled. However, if the scan fragment reports its status to the Frontend (FE) first but lacks an error URL, the INSERT INTO SELECT operation ultimately fails to return an error URL to the user.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 30, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
Copy link
Contributor Author

liaoxin01 commented Mar 30, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 30, 2025

TPC-H: Total hot run time: 33977 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit add17d4a26a0b6376bb6a60d1dcec7b68a3c0249, data reload: false

------ Round 1 ----------------------------------
q1	26088	5044	5028	5028
q2	2069	288	194	194
q3	10366	1266	704	704
q4	10224	1014	518	518
q5	7532	2340	2397	2340
q6	187	164	134	134
q7	901	764	607	607
q8	9338	1304	1086	1086
q9	6910	5091	5149	5091
q10	6826	2292	1883	1883
q11	492	275	251	251
q12	354	357	221	221
q13	17768	3729	3112	3112
q14	233	229	213	213
q15	537	487	485	485
q16	627	654	589	589
q17	596	852	339	339
q18	7504	7264	7117	7117
q19	1222	944	535	535
q20	333	321	194	194
q21	4198	2630	2372	2372
q22	1087	1033	964	964
Total cold run time: 115392 ms
Total hot run time: 33977 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5098	5105	5107	5105
q2	248	326	233	233
q3	2239	2705	2276	2276
q4	1410	1813	1365	1365
q5	4538	4409	4395	4395
q6	217	166	132	132
q7	1993	1932	1770	1770
q8	2631	2643	2556	2556
q9	7386	7273	7111	7111
q10	2991	3201	2726	2726
q11	594	500	486	486
q12	712	766	617	617
q13	3490	4015	3318	3318
q14	310	333	280	280
q15	523	481	487	481
q16	645	699	661	661
q17	1167	1554	1365	1365
q18	7801	7702	7404	7404
q19	829	848	1028	848
q20	1975	1950	1815	1815
q21	5291	5026	4923	4923
q22	1166	1087	1036	1036
Total cold run time: 53254 ms
Total hot run time: 50903 ms

@doris-robot
Copy link

doris-robot commented Mar 30, 2025

TPC-DS: Total hot run time: 192778 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit add17d4a26a0b6376bb6a60d1dcec7b68a3c0249, data reload: false

query1	1403	1057	1042	1042
query2	6226	1966	1936	1936
query3	11132	4637	4424	4424
query4	54424	25434	23224	23224
query5	4970	652	462	462
query6	330	195	202	195
query7	4856	494	282	282
query8	311	255	239	239
query9	5361	2522	2514	2514
query10	412	304	238	238
query11	15090	14983	14832	14832
query12	151	110	104	104
query13	1013	505	381	381
query14	10161	6241	6378	6241
query15	205	191	165	165
query16	7055	662	499	499
query17	1073	709	553	553
query18	1521	389	324	324
query19	191	186	166	166
query20	134	131	128	128
query21	214	127	103	103
query22	4373	4610	4397	4397
query23	33914	33568	33217	33217
query24	6705	2433	2463	2433
query25	476	494	454	454
query26	714	285	149	149
query27	2404	507	334	334
query28	3181	2483	2454	2454
query29	623	595	453	453
query30	273	226	195	195
query31	853	884	837	837
query32	74	66	63	63
query33	462	430	323	323
query34	760	858	519	519
query35	782	842	754	754
query36	966	987	913	913
query37	119	98	79	79
query38	4184	4185	4206	4185
query39	1498	1478	1429	1429
query40	221	117	109	109
query41	52	54	50	50
query42	118	105	105	105
query43	494	550	484	484
query44	1327	818	828	818
query45	187	178	175	175
query46	855	1040	634	634
query47	1837	1896	1820	1820
query48	390	412	313	313
query49	681	506	427	427
query50	731	756	420	420
query51	4223	4286	4157	4157
query52	111	100	97	97
query53	223	260	182	182
query54	503	530	455	455
query55	83	84	85	84
query56	265	268	267	267
query57	1197	1201	1153	1153
query58	250	267	249	249
query59	2802	2800	2765	2765
query60	298	281	270	270
query61	127	123	125	123
query62	835	740	678	678
query63	229	190	179	179
query64	1662	1028	698	698
query65	4471	4373	4349	4349
query66	787	396	295	295
query67	15881	15808	15394	15394
query68	7300	886	505	505
query69	529	318	261	261
query70	1218	1108	1077	1077
query71	504	306	266	266
query72	6044	4806	4868	4806
query73	1483	627	342	342
query74	9106	8945	8863	8863
query75	4155	3225	2701	2701
query76	4275	1194	748	748
query77	754	384	285	285
query78	10122	10247	9270	9270
query79	2241	821	583	583
query80	595	536	440	440
query81	485	252	228	228
query82	484	127	99	99
query83	183	175	160	160
query84	286	103	75	75
query85	762	346	310	310
query86	360	289	296	289
query87	4633	4765	4565	4565
query88	3656	2241	2237	2237
query89	406	315	275	275
query90	1816	218	211	211
query91	148	147	114	114
query92	76	64	59	59
query93	1792	1074	575	575
query94	652	424	306	306
query95	357	286	278	278
query96	496	568	271	271
query97	3127	3263	3108	3108
query98	244	210	204	204
query99	1321	1390	1277	1277
Total cold run time: 299044 ms
Total hot run time: 192778 ms

@doris-robot
Copy link

doris-robot commented Mar 30, 2025

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit add17d4a26a0b6376bb6a60d1dcec7b68a3c0249, data reload: false

query1	0.04	0.03	0.03
query2	0.14	0.10	0.10
query3	0.24	0.20	0.19
query4	1.59	0.19	0.19
query5	0.59	0.56	0.57
query6	1.19	0.71	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.53	0.52
query10	0.57	0.60	0.58
query11	0.16	0.11	0.11
query12	0.15	0.10	0.12
query13	0.61	0.60	0.60
query14	2.69	2.69	2.81
query15	0.93	0.86	0.84
query16	0.39	0.37	0.38
query17	1.05	1.06	1.01
query18	0.21	0.20	0.20
query19	1.97	1.94	1.85
query20	0.02	0.01	0.02
query21	15.36	0.89	0.54
query22	0.77	1.28	0.72
query23	14.78	1.37	0.65
query24	7.52	1.13	0.30
query25	0.31	0.25	0.08
query26	0.63	0.17	0.15
query27	0.05	0.05	0.05
query28	9.16	0.93	0.44
query29	12.57	4.04	3.38
query30	0.25	0.10	0.07
query31	2.81	0.60	0.38
query32	3.23	0.54	0.47
query33	3.12	3.07	3.08
query34	15.63	5.12	4.51
query35	4.63	4.53	4.60
query36	0.67	0.49	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.13	0.12
query41	0.09	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.14 s
Total hot run time: 30.82 s

@doris-robot
Copy link

doris-robot commented Mar 30, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/44) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 50.89% (13625/26773)
Line Coverage 40.26% (118311/293836)
Region Coverage 38.96% (60129/154355)
Branch Coverage 33.87% (30243/89292)

@dataroaring dataroaring added usercase Important user case type label dev/2.1.x dev/3.0.x labels Mar 31, 2025
dataroaring
dataroaring approved these changes Mar 31, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 31, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 31, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 31, 2025

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 4926c19 into apache:master Mar 31, 2025
28 of 31 checks passed
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Mar 31, 2025
@liaoxin01
[fix](load) Fix the issue of insert into select concurrent reporting …
00f0c47 
…causing no error url (apache#49669)

Fix the issue where concurrent fragment status reporting leads to
missing error URLs. For example, when there are two fragments (one scan
fragment and one sink fragment), if the sink fragment is canceled due to
data quality issues, the scan fragment will also be canceled. However,
if the scan fragment reports its status to the Frontend (FE) first but
lacks an error URL, the INSERT INTO SELECT operation ultimately fails to
return an error URL to the user.
@liaoxin01 liaoxin01 mentioned this pull request Mar 31, 2025
Merged
dataroaring pushed a commit that referenced this pull request Mar 31, 2025
@liaoxin01
branch-3.0: [fix](load) Fix the issue of insert into select concurren…
2acf093 
…t reporting causing no error url #49669 (#49687)

cherry pick from #49669
englefly pushed a commit to englefly/incubator-doris that referenced this pull request May 6, 2025
@liaoxin01
[fix](load) Fix the issue of insert into select concurrent reporting …
895e70d 
…causing no error url (apache#49669) (apache#3861)

cherry-pick from: apache#49669
http://jira.selectdb-in.cc/browse/CIR-14236
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@liaoxin01 @koarz
[fix](load) Fix the issue of insert into select concurrent reporting …
16e06ad 
…causing no error url (apache#49669)

Fix the issue where concurrent fragment status reporting leads to
missing error URLs. For example, when there are two fragments (one scan
fragment and one sink fragment), if the sink fragment is canceled due to
data quality issues, the scan fragment will also be canceled. However,
if the scan fragment reports its status to the Frontend (FE) first but
lacks an error URL, the INSERT INTO SELECT operation ultimately fails to
return an error URL to the user.
suxiaogang223 pushed a commit to suxiaogang223/doris that referenced this pull request Jul 15, 2025
@liaoxin01
[fix](load) Fix the issue of insert into select concurrent reporting …
739bc6c 
…causing no error url (apache#49669) (apache#3917)

cherry-pick from: apache#49669
http://jira.selectdb-in.cc/browse/CIR-14236
@dtkavin dtkavin mentioned this pull request Aug 12, 2025
Open
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@JNSimba JNSimba JNSimba approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.5-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@liaoxin01 @hello-stephen @doris-robot @JNSimba @dataroaring