Skip to content

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jul 23, 2025

What problem does this PR solve?

multi table load plan fail after restart master Fe or leader change:

mysql> show routine load for test_multi_table\G
*************************** 
Id: 1753247186255
Name: test2
CreateTime: 2025-07-23 13:06:53
PauseTime: NULL
EndTime: NULL
DbName: db
TableName:
IsMultiTable: true
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {"max_batch_rows": "3000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","loadd_to_single_tablet":"false","column_separator":";'''","line_delimiter":"\n","delete":"*","
current_concurrent_number":"1","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval": 20","max_batch_size": "209715200","esscape":"\u
0000","enclose":"\u0000","partitions":"**","columnToColumnExpr":"","whereExpr":"*****'',"desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.
0","sequence_col":"****}
DataSourceProperties: {"topic":"my-topic","currentkafkaPartitions": "0", "brokerList": "10.16.10.10.10.77:19092"}
CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING","group.id": "test2_7f6143d8-f270-4667-851a-e8fb87c27d32"}
Statistic: {"receivedBytes":89,"runningTxns": [1542060502549504],"errorRows":0, "committedTaskNum":0, "loadedRows":1,"LoadRowsRate":0,"abortedTaskNum":7,"errorRowsAfterResumed":0,"totalRows"
:1,"unselectedRows":0,"receivedBytesRate":1,"taskExecuteTimeMs":51588}
Progress: {"0":"0"}
Lag: {"0":1}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg: 2025-07-23 13:08:07: [INTERNAL_ERROR]TStatus:AnalysisException: errCode = 2, detailMessage = , connect context's user is null, ComputeGroupException: CURRENT_USER_NO_AUTH_TO_US
E_DEFAULT_COMPUTE_GROUP, you can contact the system admministrator and request that they grant you the defaultcompute group permissions, use SQL 'SHOW PROPERTY like'default_compute_group'` and
NT USAGE_PRIV ON COMPUTE GROUP {compute_group_name}TO{user}
GRA
0# #
doris::Status doris::Status::create<true>(doris::TStatus const&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/114/include/g++-v14/bits/basic_string.h:228
1# doris::io::MultiTablePipe::request_and_exec_plans() at /mnt/disk1/laihui/doris/be/src/common/status.h:522
2#
doris: RoutineLoadTaskExecutor::exec_task(std::shared_ptr<doris: StreamLoadContext>, doris::DataConsumerPool*, std::function<void (std::shared_ptr<doris::StreamLoadContext>)>) at /mnt/di
sk1/laihui/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:0
3#
std::_Function_handler<void (), ... (reason istruncated, check fe.log with txnId for details(1
User: root
Comment:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Jul 23, 2025

run buildall

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 23, 2025
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 33535 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 37c401012a8227292d97b688fd393b47e36a6262, data reload: false

------ Round 1 ----------------------------------
q1	17596	5223	5020	5020
q2	1930	284	179	179
q3	10307	1211	705	705
q4	10232	1000	528	528
q5	7801	2359	2362	2359
q6	177	160	128	128
q7	880	744	586	586
q8	9310	1296	1024	1024
q9	6895	5058	5095	5058
q10	6889	2346	1958	1958
q11	483	284	264	264
q12	348	346	216	216
q13	17748	3640	3074	3074
q14	223	226	218	218
q15	559	488	474	474
q16	410	419	372	372
q17	590	836	359	359
q18	7364	7174	6945	6945
q19	1318	972	567	567
q20	346	327	219	219
q21	3689	2494	2321	2321
q22	1056	1036	961	961
Total cold run time: 106151 ms
Total hot run time: 33535 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5252	5170	5185	5170
q2	239	317	228	228
q3	2197	2674	2301	2301
q4	1359	1784	1325	1325
q5	4174	4375	4518	4375
q6	215	174	141	141
q7	2010	1966	1797	1797
q8	2582	2557	2575	2557
q9	7215	7148	7229	7148
q10	3065	3295	2914	2914
q11	551	510	501	501
q12	708	779	638	638
q13	3619	4063	3390	3390
q14	295	299	287	287
q15	509	490	486	486
q16	465	503	426	426
q17	1214	1553	1406	1406
q18	8115	7714	7123	7123
q19	752	780	868	780
q20	1883	1979	1812	1812
q21	4805	4444	4350	4350
q22	1065	1039	981	981
Total cold run time: 52289 ms
Total hot run time: 50136 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187070 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 37c401012a8227292d97b688fd393b47e36a6262, data reload: false

query1	989	385	414	385
query2	6527	1718	1773	1718
query3	6737	221	218	218
query4	26748	23705	23245	23245
query5	4350	601	473	473
query6	324	214	198	198
query7	4639	499	291	291
query8	272	229	221	221
query9	8612	2833	2832	2832
query10	499	358	287	287
query11	15326	14973	14760	14760
query12	164	115	115	115
query13	1653	519	402	402
query14	8600	5872	5856	5856
query15	209	192	161	161
query16	7500	623	445	445
query17	1213	755	601	601
query18	1996	415	319	319
query19	199	189	166	166
query20	134	117	116	116
query21	212	125	108	108
query22	4253	4133	4176	4133
query23	33874	32958	32964	32958
query24	8064	2370	2354	2354
query25	522	456	422	422
query26	1229	268	152	152
query27	2733	497	339	339
query28	4371	2169	2145	2145
query29	753	559	438	438
query30	278	223	182	182
query31	877	785	730	730
query32	83	78	72	72
query33	542	372	323	323
query34	782	837	503	503
query35	810	808	719	719
query36	952	1011	887	887
query37	120	104	82	82
query38	4088	4167	4058	4058
query39	1471	1402	1411	1402
query40	229	123	108	108
query41	63	60	61	60
query42	124	110	111	110
query43	483	482	475	475
query44	1330	853	849	849
query45	177	168	163	163
query46	821	995	632	632
query47	1776	1840	1763	1763
query48	386	425	307	307
query49	741	493	408	408
query50	635	692	404	404
query51	5348	5413	5318	5318
query52	115	109	100	100
query53	228	258	190	190
query54	593	581	523	523
query55	85	83	83	83
query56	313	307	303	303
query57	1212	1195	1139	1139
query58	276	257	259	257
query59	2617	2703	2753	2703
query60	343	342	315	315
query61	126	126	164	126
query62	783	726	664	664
query63	231	195	183	183
query64	4297	995	672	672
query65	4229	4193	4144	4144
query66	1156	418	348	348
query67	15873	15610	15488	15488
query68	7993	895	564	564
query69	482	311	280	280
query70	1229	1181	1163	1163
query71	462	320	309	309
query72	5538	4726	4939	4726
query73	763	668	355	355
query74	8806	9053	8815	8815
query75	3789	3116	2697	2697
query76	3651	1115	761	761
query77	789	401	321	321
query78	9929	10139	9343	9343
query79	2616	805	580	580
query80	613	531	479	479
query81	454	258	223	223
query82	429	143	110	110
query83	282	249	238	238
query84	292	101	96	96
query85	800	355	318	318
query86	339	309	304	304
query87	4381	4376	4263	4263
query88	2946	2220	2212	2212
query89	388	321	287	287
query90	1924	220	216	216
query91	139	139	113	113
query92	83	67	67	67
query93	1151	933	636	636
query94	676	392	303	303
query95	395	359	312	312
query96	483	583	273	273
query97	2660	2712	2580	2580
query98	236	211	211	211
query99	1422	1380	1264	1264
Total cold run time: 273435 ms
Total hot run time: 187070 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.5 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 37c401012a8227292d97b688fd393b47e36a6262, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.24	0.07	0.08
query4	1.62	0.11	0.11
query5	0.42	0.44	0.40
query6	1.16	0.66	0.65
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.59	0.53	0.52
query10	0.58	0.58	0.58
query11	0.16	0.10	0.12
query12	0.15	0.11	0.11
query13	0.63	0.64	0.61
query14	0.79	0.83	0.84
query15	0.89	0.86	0.87
query16	0.39	0.39	0.39
query17	1.03	1.05	1.05
query18	0.20	0.21	0.20
query19	1.92	1.82	1.80
query20	0.01	0.01	0.02
query21	15.38	0.87	0.54
query22	0.80	1.15	0.77
query23	14.86	1.39	0.64
query24	6.72	1.56	0.66
query25	0.59	0.16	0.14
query26	0.65	0.16	0.14
query27	0.06	0.05	0.05
query28	9.99	0.92	0.43
query29	12.55	3.98	3.26
query30	3.00	3.07	2.96
query31	2.83	0.58	0.39
query32	3.25	0.54	0.48
query33	3.12	3.10	3.19
query34	15.92	5.36	4.82
query35	4.82	4.89	4.87
query36	0.68	0.51	0.49
query37	0.09	0.07	0.07
query38	0.05	0.05	0.04
query39	0.03	0.03	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 106.69 s
Total hot run time: 32.5 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 043e13c into apache:master Jul 24, 2025
31 checks passed
@dataroaring dataroaring added the usercase Important user case type label label Jul 24, 2025
sollhui added a commit to sollhui/doris that referenced this pull request Jul 24, 2025
… leader change (apache#53799)

multi table load plan fail after restart master Fe or leader change:
```
mysql> show routine load for test_multi_table\G
***************************
Id: 1753247186255
Name: test2
CreateTime: 2025-07-23 13:06:53
PauseTime: NULL
EndTime: NULL
DbName: db
TableName:
IsMultiTable: true
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {"max_batch_rows": "3000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","loadd_to_single_tablet":"false","column_separator":";'''","line_delimiter":"\n","delete":"*","
current_concurrent_number":"1","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval": 20","max_batch_size": "209715200","esscape":"\u
0000","enclose":"\u0000","partitions":"**","columnToColumnExpr":"","whereExpr":"*****'',"desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.
0","sequence_col":"****}
DataSourceProperties: {"topic":"my-topic","currentkafkaPartitions": "0", "brokerList": "10.16.10.10.10.77:19092"}
CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING","group.id": "test2_7f6143d8-f270-4667-851a-e8fb87c27d32"}
Statistic: {"receivedBytes":89,"runningTxns": [1542060502549504],"errorRows":0, "committedTaskNum":0, "loadedRows":1,"LoadRowsRate":0,"abortedTaskNum":7,"errorRowsAfterResumed":0,"totalRows"
:1,"unselectedRows":0,"receivedBytesRate":1,"taskExecuteTimeMs":51588}
Progress: {"0":"0"}
Lag: {"0":1}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg: 2025-07-23 13:08:07: [INTERNAL_ERROR]TStatus:AnalysisException: errCode = 2, detailMessage = , connect context's user is null, ComputeGroupException: CURRENT_USER_NO_AUTH_TO_US
E_DEFAULT_COMPUTE_GROUP, you can contact the system admministrator and request that they grant you the defaultcompute group permissions, use SQL 'SHOW PROPERTY like'default_compute_group'` and
NT USAGE_PRIV ON COMPUTE GROUP {compute_group_name}TO{user}
GRA
0# #
doris::Status doris::Status::create<true>(doris::TStatus const&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/114/include/g++-v14/bits/basic_string.h:228
1# doris::io::MultiTablePipe::request_and_exec_plans() at /mnt/disk1/laihui/doris/be/src/common/status.h:522
2#
doris: RoutineLoadTaskExecutor::exec_task(std::shared_ptr<doris: StreamLoadContext>, doris::DataConsumerPool*, std::function<void (std::shared_ptr<doris::StreamLoadContext>)>) at /mnt/di
sk1/laihui/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:0
3#
std::_Function_handler<void (), ... (reason istruncated, check fe.log with txnId for details(1
User: root
Comment:
```

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
sollhui added a commit to sollhui/doris that referenced this pull request Jul 24, 2025
… leader change (apache#53799)

multi table load plan fail after restart master Fe or leader change:
```
mysql> show routine load for test_multi_table\G
***************************
Id: 1753247186255
Name: test2
CreateTime: 2025-07-23 13:06:53
PauseTime: NULL
EndTime: NULL
DbName: db
TableName:
IsMultiTable: true
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {"max_batch_rows": "3000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","loadd_to_single_tablet":"false","column_separator":";'''","line_delimiter":"\n","delete":"*","
current_concurrent_number":"1","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval": 20","max_batch_size": "209715200","esscape":"\u
0000","enclose":"\u0000","partitions":"**","columnToColumnExpr":"","whereExpr":"*****'',"desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.
0","sequence_col":"****}
DataSourceProperties: {"topic":"my-topic","currentkafkaPartitions": "0", "brokerList": "10.16.10.10.10.77:19092"}
CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING","group.id": "test2_7f6143d8-f270-4667-851a-e8fb87c27d32"}
Statistic: {"receivedBytes":89,"runningTxns": [1542060502549504],"errorRows":0, "committedTaskNum":0, "loadedRows":1,"LoadRowsRate":0,"abortedTaskNum":7,"errorRowsAfterResumed":0,"totalRows"
:1,"unselectedRows":0,"receivedBytesRate":1,"taskExecuteTimeMs":51588}
Progress: {"0":"0"}
Lag: {"0":1}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg: 2025-07-23 13:08:07: [INTERNAL_ERROR]TStatus:AnalysisException: errCode = 2, detailMessage = , connect context's user is null, ComputeGroupException: CURRENT_USER_NO_AUTH_TO_US
E_DEFAULT_COMPUTE_GROUP, you can contact the system admministrator and request that they grant you the defaultcompute group permissions, use SQL 'SHOW PROPERTY like'default_compute_group'` and
NT USAGE_PRIV ON COMPUTE GROUP {compute_group_name}TO{user}
GRA
0# #
doris::Status doris::Status::create<true>(doris::TStatus const&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/114/include/g++-v14/bits/basic_string.h:228
1# doris::io::MultiTablePipe::request_and_exec_plans() at /mnt/disk1/laihui/doris/be/src/common/status.h:522
2#
doris: RoutineLoadTaskExecutor::exec_task(std::shared_ptr<doris: StreamLoadContext>, doris::DataConsumerPool*, std::function<void (std::shared_ptr<doris::StreamLoadContext>)>) at /mnt/di
sk1/laihui/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:0
3#
std::_Function_handler<void (), ... (reason istruncated, check fe.log with txnId for details(1
User: root
Comment:
```

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
dataroaring pushed a commit that referenced this pull request Jul 24, 2025
…master Fe or leader change (#53799) (#53829)

pick (#53799)

multi table load plan fail after restart master Fe or leader change:
```
mysql> show routine load for test_multi_table\G
***************************
Id: 1753247186255
Name: test2
CreateTime: 2025-07-23 13:06:53
PauseTime: NULL
EndTime: NULL
DbName: db
TableName:
IsMultiTable: true
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {"max_batch_rows": "3000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","loadd_to_single_tablet":"false","column_separator":";'''","line_delimiter":"\n","delete":"*","
current_concurrent_number":"1","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval": 20","max_batch_size": "209715200","esscape":"\u
0000","enclose":"\u0000","partitions":"**","columnToColumnExpr":"","whereExpr":"*****'',"desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.
0","sequence_col":"****}
DataSourceProperties: {"topic":"my-topic","currentkafkaPartitions": "0", "brokerList": "10.16.10.10.10.77:19092"}
CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING","group.id": "test2_7f6143d8-f270-4667-851a-e8fb87c27d32"}
Statistic: {"receivedBytes":89,"runningTxns": [1542060502549504],"errorRows":0, "committedTaskNum":0, "loadedRows":1,"LoadRowsRate":0,"abortedTaskNum":7,"errorRowsAfterResumed":0,"totalRows"
:1,"unselectedRows":0,"receivedBytesRate":1,"taskExecuteTimeMs":51588}
Progress: {"0":"0"}
Lag: {"0":1}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg: 2025-07-23 13:08:07: [INTERNAL_ERROR]TStatus:AnalysisException: errCode = 2, detailMessage = , connect context's user is null, ComputeGroupException: CURRENT_USER_NO_AUTH_TO_US
E_DEFAULT_COMPUTE_GROUP, you can contact the system admministrator and request that they grant you the defaultcompute group permissions, use SQL 'SHOW PROPERTY like'default_compute_group'` and
NT USAGE_PRIV ON COMPUTE GROUP {compute_group_name}TO{user}
GRA
0# #
doris::Status doris::Status::create<true>(doris::TStatus const&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/114/include/g++-v14/bits/basic_string.h:228
1# doris::io::MultiTablePipe::request_and_exec_plans() at /mnt/disk1/laihui/doris/be/src/common/status.h:522
2#
doris: RoutineLoadTaskExecutor::exec_task(std::shared_ptr<doris: StreamLoadContext>, doris::DataConsumerPool*, std::function<void (std::shared_ptr<doris::StreamLoadContext>)>) at /mnt/di
sk1/laihui/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:0
3#
std::_Function_handler<void (), ... (reason istruncated, check fe.log with txnId for details(1
User: root
Comment:
```

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
morrySnow pushed a commit that referenced this pull request Jul 25, 2025
w41ter pushed a commit to w41ter/incubator-doris that referenced this pull request Jul 30, 2025
… leader change (apache#53799)

### What problem does this PR solve?

multi table load plan fail after restart master Fe or leader change:
```
mysql> show routine load for test_multi_table\G
*************************** 
Id: 1753247186255
Name: test2
CreateTime: 2025-07-23 13:06:53
PauseTime: NULL
EndTime: NULL
DbName: db
TableName:
IsMultiTable: true
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {"max_batch_rows": "3000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","loadd_to_single_tablet":"false","column_separator":";'''","line_delimiter":"\n","delete":"*","
current_concurrent_number":"1","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval": 20","max_batch_size": "209715200","esscape":"\u
0000","enclose":"\u0000","partitions":"**","columnToColumnExpr":"","whereExpr":"*****'',"desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.
0","sequence_col":"****}
DataSourceProperties: {"topic":"my-topic","currentkafkaPartitions": "0", "brokerList": "10.16.10.10.10.77:19092"}
CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING","group.id": "test2_7f6143d8-f270-4667-851a-e8fb87c27d32"}
Statistic: {"receivedBytes":89,"runningTxns": [1542060502549504],"errorRows":0, "committedTaskNum":0, "loadedRows":1,"LoadRowsRate":0,"abortedTaskNum":7,"errorRowsAfterResumed":0,"totalRows"
:1,"unselectedRows":0,"receivedBytesRate":1,"taskExecuteTimeMs":51588}
Progress: {"0":"0"}
Lag: {"0":1}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg: 2025-07-23 13:08:07: [INTERNAL_ERROR]TStatus:AnalysisException: errCode = 2, detailMessage = , connect context's user is null, ComputeGroupException: CURRENT_USER_NO_AUTH_TO_US
E_DEFAULT_COMPUTE_GROUP, you can contact the system admministrator and request that they grant you the defaultcompute group permissions, use SQL 'SHOW PROPERTY like'default_compute_group'` and
NT USAGE_PRIV ON COMPUTE GROUP {compute_group_name}TO{user}
GRA
0# #
doris::Status doris::Status::create<true>(doris::TStatus const&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/114/include/g++-v14/bits/basic_string.h:228
1# doris::io::MultiTablePipe::request_and_exec_plans() at /mnt/disk1/laihui/doris/be/src/common/status.h:522
2#
doris: RoutineLoadTaskExecutor::exec_task(std::shared_ptr<doris: StreamLoadContext>, doris::DataConsumerPool*, std::function<void (std::shared_ptr<doris::StreamLoadContext>)>) at /mnt/di
sk1/laihui/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:0
3#
std::_Function_handler<void (), ... (reason istruncated, check fe.log with txnId for details(1
User: root
Comment:
```

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
zhiqiang-hhhh pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Aug 4, 2025
… leader change (apache#53799)

### What problem does this PR solve?

multi table load plan fail after restart master Fe or leader change:
```
mysql> show routine load for test_multi_table\G
*************************** 
Id: 1753247186255
Name: test2
CreateTime: 2025-07-23 13:06:53
PauseTime: NULL
EndTime: NULL
DbName: db
TableName:
IsMultiTable: true
State: RUNNING
DataSourceType: KAFKA
CurrentTaskNum: 1
JobProperties: {"max_batch_rows": "3000000","timezone":"Asia/Shanghai","send_batch_parallelism":"1","loadd_to_single_tablet":"false","column_separator":";'''","line_delimiter":"\n","delete":"*","
current_concurrent_number":"1","partial_columns":"false","merge_type":"APPEND","exec_mem_limit":"2147483648","strict_mode":"false","max_batch_interval": 20","max_batch_size": "209715200","esscape":"\u
0000","enclose":"\u0000","partitions":"**","columnToColumnExpr":"","whereExpr":"*****'',"desired_concurrent_number":"256","precedingFilter":"*","format":"csv","max_error_number":"0","max_filter_ratio":"1.
0","sequence_col":"****}
DataSourceProperties: {"topic":"my-topic","currentkafkaPartitions": "0", "brokerList": "10.16.10.10.10.77:19092"}
CustomProperties: {"kafka_default_offsets":"OFFSET_BEGINNING","group.id": "test2_7f6143d8-f270-4667-851a-e8fb87c27d32"}
Statistic: {"receivedBytes":89,"runningTxns": [1542060502549504],"errorRows":0, "committedTaskNum":0, "loadedRows":1,"LoadRowsRate":0,"abortedTaskNum":7,"errorRowsAfterResumed":0,"totalRows"
:1,"unselectedRows":0,"receivedBytesRate":1,"taskExecuteTimeMs":51588}
Progress: {"0":"0"}
Lag: {"0":1}
ReasonOfStateChanged:
ErrorLogUrls:
OtherMsg: 2025-07-23 13:08:07: [INTERNAL_ERROR]TStatus:AnalysisException: errCode = 2, detailMessage = , connect context's user is null, ComputeGroupException: CURRENT_USER_NO_AUTH_TO_US
E_DEFAULT_COMPUTE_GROUP, you can contact the system admministrator and request that they grant you the defaultcompute group permissions, use SQL 'SHOW PROPERTY like'default_compute_group'` and
NT USAGE_PRIV ON COMPUTE GROUP {compute_group_name}TO{user}
GRA
0# #
doris::Status doris::Status::create<true>(doris::TStatus const&) at /mnt/disk1/laihui/build/ldb_toolchain/bin/../lib/gcc/x86_64-pc-linux-gnu/114/include/g++-v14/bits/basic_string.h:228
1# doris::io::MultiTablePipe::request_and_exec_plans() at /mnt/disk1/laihui/doris/be/src/common/status.h:522
2#
doris: RoutineLoadTaskExecutor::exec_task(std::shared_ptr<doris: StreamLoadContext>, doris::DataConsumerPool*, std::function<void (std::shared_ptr<doris::StreamLoadContext>)>) at /mnt/di
sk1/laihui/doris/be/src/runtime/routine_load/routine_load_task_executor.cpp:0
3#
std::_Function_handler<void (), ... (reason istruncated, check fe.log with txnId for details(1
User: root
Comment:
```

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.7-merged dev/3.1.0-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants