Skip to content

[fix](load) fix concurrent read and write to runningTxnIds #51615

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Jun 11, 2025
Merged

[fix](load) fix concurrent read and write to runningTxnIds #51615

dataroaring merged 1 commit into from
Jun 11, 2025

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jun 10, 2025 •
edited
Loading

What problem does this PR solve?

Execute show routine load meets java.util.ConcurrentModificationException: null, using ConcurrentHashMap to fix concurrent read and write to runningTxnIds.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 10, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Contributor Author

sollhui commented Jun 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 10, 2025

TPC-H: Total hot run time: 33678 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7c8fec38a184afae803ba6ef3fdb4efbaa0a91d, data reload: false

------ Round 1 ----------------------------------
q1	26113	4990	4893	4893
q2	1938	280	189	189
q3	10285	1239	709	709
q4	10223	982	516	516
q5	7532	2344	2295	2295
q6	175	161	130	130
q7	910	713	602	602
q8	9306	1267	1108	1108
q9	6913	5084	5090	5084
q10	6853	2322	1893	1893
q11	489	284	288	284
q12	337	346	216	216
q13	17843	3745	3150	3150
q14	222	234	225	225
q15	555	479	471	471
q16	437	427	377	377
q17	597	858	363	363
q18	7573	7276	7176	7176
q19	1736	936	545	545
q20	331	334	211	211
q21	3693	2555	2285	2285
q22	1037	1048	956	956
Total cold run time: 115098 ms
Total hot run time: 33678 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5088	5040	5003	5003
q2	235	307	223	223
q3	2094	2604	2307	2307
q4	1371	1788	1348	1348
q5	4342	4380	4364	4364
q6	215	181	129	129
q7	2041	1894	1806	1806
q8	2585	2515	2460	2460
q9	7259	7219	7027	7027
q10	3006	3222	2793	2793
q11	583	517	498	498
q12	691	773	628	628
q13	3510	3927	3314	3314
q14	289	298	275	275
q15	535	517	492	492
q16	451	481	433	433
q17	1128	1543	1373	1373
q18	7785	7459	7462	7459
q19	803	817	959	817
q20	1960	2050	1834	1834
q21	4712	4543	4455	4455
q22	1122	1050	1025	1025
Total cold run time: 51805 ms
Total hot run time: 50063 ms

@doris-robot
Copy link

doris-robot commented Jun 10, 2025

TPC-DS: Total hot run time: 191798 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7c8fec38a184afae803ba6ef3fdb4efbaa0a91d, data reload: false

query1	1439	1100	1056	1056
query2	6342	1889	1875	1875
query3	10977	4634	4413	4413
query4	52338	24473	22955	22955
query5	5132	606	460	460
query6	358	207	193	193
query7	4878	510	279	279
query8	279	241	212	212
query9	5291	2643	2616	2616
query10	450	328	265	265
query11	15108	15076	14921	14921
query12	167	111	107	107
query13	1021	504	399	399
query14	9916	6221	6394	6221
query15	239	217	189	189
query16	7011	634	475	475
query17	1084	735	582	582
query18	1533	425	300	300
query19	194	182	160	160
query20	122	122	124	122
query21	211	125	107	107
query22	4249	4623	4220	4220
query23	34184	33492	33472	33472
query24	6467	2462	2398	2398
query25	455	506	397	397
query26	666	277	151	151
query27	2155	509	348	348
query28	3050	2196	2174	2174
query29	616	602	445	445
query30	285	229	201	201
query31	852	897	786	786
query32	80	63	65	63
query33	478	377	332	332
query34	774	883	523	523
query35	784	876	778	778
query36	954	997	900	900
query37	117	104	84	84
query38	4235	4319	4160	4160
query39	1510	1454	1453	1453
query40	216	127	114	114
query41	70	67	64	64
query42	133	108	115	108
query43	497	526	491	491
query44	1366	848	840	840
query45	188	175	174	174
query46	887	1018	646	646
query47	1854	1923	1786	1786
query48	409	442	325	325
query49	658	498	409	409
query50	673	702	412	412
query51	4290	4289	4172	4172
query52	110	121	107	107
query53	223	281	196	196
query54	588	596	514	514
query55	82	80	85	80
query56	307	317	291	291
query57	1204	1198	1149	1149
query58	280	272	260	260
query59	2746	2777	2659	2659
query60	334	328	309	309
query61	128	126	139	126
query62	729	731	696	696
query63	232	191	191	191
query64	1484	1065	687	687
query65	4314	4343	4176	4176
query66	795	396	298	298
query67	15977	15530	15410	15410
query68	7177	933	521	521
query69	542	316	262	262
query70	1147	1111	1141	1111
query71	489	321	299	299
query72	5585	4744	4773	4744
query73	1487	606	359	359
query74	9066	9175	8775	8775
query75	4134	3174	2655	2655
query76	4275	1208	765	765
query77	775	366	284	284
query78	10145	10197	9334	9334
query79	2268	822	580	580
query80	626	534	446	446
query81	482	259	219	219
query82	436	125	98	98
query83	258	253	227	227
query84	295	116	80	80
query85	800	353	321	321
query86	363	300	272	272
query87	4385	4426	4387	4387
query88	3773	2290	2298	2290
query89	405	315	292	292
query90	1809	213	211	211
query91	144	142	113	113
query92	80	62	60	60
query93	1718	929	576	576
query94	667	418	295	295
query95	363	296	290	290
query96	492	575	291	291
query97	2723	2774	2676	2676
query98	229	201	206	201
query99	1431	1403	1282	1282
Total cold run time: 295344 ms
Total hot run time: 191798 ms

@doris-robot
Copy link

doris-robot commented Jun 10, 2025

ClickBench: Total hot run time: 29.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7c8fec38a184afae803ba6ef3fdb4efbaa0a91d, data reload: false

query1	0.03	0.03	0.04
query2	0.12	0.10	0.11
query3	0.27	0.19	0.20
query4	1.59	0.20	0.10
query5	0.44	0.41	0.42
query6	1.17	0.66	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.59	0.52	0.52
query10	0.57	0.59	0.58
query11	0.15	0.11	0.12
query12	0.15	0.12	0.11
query13	0.62	0.60	0.61
query14	0.80	0.80	0.81
query15	0.89	0.86	0.86
query16	0.37	0.39	0.38
query17	1.01	1.06	1.03
query18	0.23	0.21	0.20
query19	1.94	1.86	1.89
query20	0.01	0.02	0.01
query21	15.39	0.89	0.56
query22	0.76	1.09	0.71
query23	15.04	1.36	0.60
query24	6.64	1.37	1.53
query25	0.68	0.20	0.21
query26	0.95	0.16	0.13
query27	0.05	0.06	0.05
query28	9.79	0.88	0.44
query29	12.55	4.06	3.39
query30	0.25	0.09	0.07
query31	2.82	0.61	0.38
query32	3.24	0.54	0.48
query33	3.05	3.04	3.04
query34	15.68	5.09	4.52
query35	4.50	4.51	4.53
query36	0.67	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 103.56 s
Total hot run time: 29.88 s

dataroaring
dataroaring approved these changes Jun 10, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 10, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 10, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 10, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 10, 2025

FE UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉
Increment coverage report
Complete coverage report

liaoxin01
liaoxin01 approved these changes Jun 10, 2025
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

dataroaring
dataroaring approved these changes Jun 11, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit e71a259 into apache:master Jun 11, 2025
31 of 33 checks passed
github-actions bot pushed a commit that referenced this pull request Jun 11, 2025
@sollhui
[fix](load) fix concurrent read and write to runningTxnIds (#51615)
852ceb0 
### What problem does this PR solve?

Execute `show routine load` meets
`java.util.ConcurrentModificationException: null`, using
ConcurrentHashMap to fix concurrent read and write to runningTxnIds.
@github-actions github-actions bot mentioned this pull request Jun 11, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Jun 11, 2025
@sollhui
[fix](load) fix concurrent read and write to runningTxnIds (#51615)
77ba725 
### What problem does this PR solve?

Execute `show routine load` meets
`java.util.ConcurrentModificationException: null`, using
ConcurrentHashMap to fix concurrent read and write to runningTxnIds.
@github-actions github-actions bot mentioned this pull request Jun 11, 2025
Merged
dataroaring pushed a commit that referenced this pull request Jun 11, 2025
@github-actions @sollhui
branch-3.0: [fix](load) fix concurrent read and write to runningTxnIds …
cf057bf 
…#51615 (#51639)

Cherry-picked from #51615

Co-authored-by: hui lai <laihui@selectdb.com>
yiguolei pushed a commit that referenced this pull request Jun 13, 2025
@sollhui @yiguolei
[fix](load) fix concurrent read and write to runningTxnIds (#51615)
bd890df 
### What problem does this PR solve?

Execute `show routine load` meets
`java.util.ConcurrentModificationException: null`, using
ConcurrentHashMap to fix concurrent read and write to runningTxnIds.
yiguolei pushed a commit that referenced this pull request Jun 17, 2025
@github-actions @sollhui
branch-2.1: [fix](load) fix concurrent read and write to runningTxnIds …
a461ff3 
…#51615 (#51640)

Cherry-picked from #51615

Co-authored-by: hui lai <laihui@selectdb.com>
@gavinchou gavinchou mentioned this pull request Aug 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.11-merged dev/3.0.7-merged dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@sollhui @Thearas @doris-robot @hello-stephen @liaoxin01 @dataroaring @yiguolei @morrySnow