Skip to content

branch-3.1: [fix](audit) fix audit loader thread hang and label already exists issue #54031 #54170

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Aug 1, 2025
Merged

branch-3.1: [fix](audit) fix audit loader thread hang and label already exists issue #54031 #54170

morrySnow merged 1 commit into from
Aug 1, 2025

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jul 31, 2025

Cherry-picked from #54031

@cambyzju
[fix](audit) fix audit loader thread hang and label already exists is…
0acd2a9 
…sue (#54031)

### What problem does this PR solve?

audit loader thread hang, and there are no new audit log in table
__internal_schema.audit_log

```
"audit loader thread" #59 prio=5 os_prio=0 cpu=19288.63ms elapsed=714273.53s tid=0x00007f7edaf559d0 nid=0x302e1e runnable  [0x00007f7e9c3fe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.SocketDispatcher.read0(java.base@17.0.13/Native Method)
        at sun.nio.ch.SocketDispatcher.read(java.base@17.0.13/SocketDispatcher.java:47)
        at sun.nio.ch.NioSocketImpl.tryRead(java.base@17.0.13/NioSocketImpl.java:266)
        at sun.nio.ch.NioSocketImpl.implRead(java.base@17.0.13/NioSocketImpl.java:317)
        at sun.nio.ch.NioSocketImpl.read(java.base@17.0.13/NioSocketImpl.java:355)
        at sun.nio.ch.NioSocketImpl$1.read(java.base@17.0.13/NioSocketImpl.java:808)
        at java.net.Socket$SocketInputStream.read(java.base@17.0.13/Socket.java:985)
        at java.io.BufferedInputStream.fill(java.base@17.0.13/BufferedInputStream.java:244)
        at java.io.BufferedInputStream.read1(java.base@17.0.13/BufferedInputStream.java:284)
        at java.io.BufferedInputStream.read(java.base@17.0.13/BufferedInputStream.java:343)
        - locked <0x00000005dbc6ecf8> (a java.io.BufferedInputStream)
        at sun.net.www.http.HttpClient.parseHTTPHeader(java.base@17.0.13/HttpClient.java:826)
        at sun.net.www.http.HttpClient.parseHTTP(java.base@17.0.13/HttpClient.java:761)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(java.base@17.0.13/HttpURLConnection.java:1724)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(java.base@17.0.13/HttpURLConnection.java:1625)
        at java.net.HttpURLConnection.getResponseCode(java.base@17.0.13/HttpURLConnection.java:529)
        at org.apache.doris.plugin.audit.AuditStreamLoader.loadBatch(AuditStreamLoader.java:138)
        at org.apache.doris.plugin.audit.AuditLoader.loadIfNecessary(AuditLoader.java:196)
        - locked <0x00000005c6182238> (a org.apache.doris.plugin.audit.AuditLoader)
        at org.apache.doris.plugin.audit.AuditLoader$LoadWorker.run(AuditLoader.java:234)
        at java.lang.Thread.run(java.base@17.0.13/Thread.java:840)
```

Also modify the label of audit log load, use true FE ip to avoid
conflict between FEs
@github-actions github-actions bot requested a review from morrySnow as a code owner July 31, 2025 16:59
@Thearas
Copy link
Contributor

Thearas commented Jul 31, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Jul 31, 2025
@Thearas
Copy link
Contributor

Thearas commented Jul 31, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

TPC-H: Total hot run time: 32234 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0acd2a9c72f6c3eae4a75cafdaa3f6d5a4fd6c21, data reload: false

------ Round 1 ----------------------------------
q1	17617	5440	5414	5414
q2	2070	267	160	160
q3	10451	1232	750	750
q4	10214	861	450	450
q5	7731	2363	2085	2085
q6	177	165	133	133
q7	894	747	603	603
q8	9334	1423	1134	1134
q9	5161	4875	4871	4871
q10	6790	2258	1828	1828
q11	479	280	253	253
q12	329	365	207	207
q13	17786	3594	2999	2999
q14	223	227	206	206
q15	517	472	455	455
q16	421	443	380	380
q17	600	845	352	352
q18	6867	6304	6256	6256
q19	1307	950	546	546
q20	329	343	214	214
q21	2896	2160	1952	1952
q22	1034	986	1004	986
Total cold run time: 103227 ms
Total hot run time: 32234 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5572	5524	5649	5524
q2	232	335	238	238
q3	2259	2587	2311	2311
q4	1368	1791	1347	1347
q5	4413	4789	4893	4789
q6	172	168	132	132
q7	2048	1970	1831	1831
q8	2635	2882	2722	2722
q9	7385	7296	7354	7296
q10	3067	3332	2874	2874
q11	593	519	498	498
q12	645	712	574	574
q13	3383	3807	3180	3180
q14	280	290	272	272
q15	521	475	483	475
q16	435	484	433	433
q17	1210	1710	1255	1255
q18	7561	7459	7289	7289
q19	807	1169	1100	1100
q20	2012	2004	1874	1874
q21	5382	4990	4702	4702
q22	1072	1058	996	996
Total cold run time: 53052 ms
Total hot run time: 51712 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

TPC-DS: Total hot run time: 196604 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0acd2a9c72f6c3eae4a75cafdaa3f6d5a4fd6c21, data reload: false

query1	1328	942	898	898
query2	6220	2043	1916	1916
query3	10938	4395	4547	4395
query4	32967	23655	24018	23655
query5	4796	630	461	461
query6	291	220	172	172
query7	4015	482	318	318
query8	309	254	241	241
query9	9346	2578	2564	2564
query10	494	324	265	265
query11	18242	15675	15214	15214
query12	162	105	102	102
query13	1557	526	422	422
query14	10705	6635	6785	6635
query15	229	205	185	185
query16	8093	625	522	522
query17	1525	740	618	618
query18	2169	406	315	315
query19	240	189	170	170
query20	129	118	122	118
query21	203	128	109	109
query22	4667	4789	4577	4577
query23	35155	34125	33787	33787
query24	7354	2645	2669	2645
query25	506	455	405	405
query26	819	286	165	165
query27	2138	479	351	351
query28	5411	2218	2162	2162
query29	683	570	433	433
query30	246	187	157	157
query31	975	897	838	838
query32	83	56	58	56
query33	481	377	313	313
query34	768	855	514	514
query35	777	801	739	739
query36	1012	1097	955	955
query37	108	97	67	67
query38	4124	4018	3893	3893
query39	1538	1458	1463	1458
query40	209	116	100	100
query41	49	47	47	47
query42	124	111	103	103
query43	502	506	498	498
query44	1412	833	826	826
query45	180	173	165	165
query46	899	1063	696	696
query47	1931	1995	1927	1927
query48	398	427	345	345
query49	722	506	411	411
query50	682	679	423	423
query51	7422	7244	7237	7237
query52	98	98	92	92
query53	227	263	187	187
query54	556	552	467	467
query55	84	77	85	77
query56	294	274	255	255
query57	1258	1286	1199	1199
query58	239	228	223	223
query59	3125	3148	3026	3026
query60	296	300	270	270
query61	110	114	117	114
query62	806	743	707	707
query63	231	200	190	190
query64	3649	991	741	741
query65	3322	3327	3270	3270
query66	805	421	322	322
query67	16134	15724	15643	15643
query68	4485	846	567	567
query69	518	315	284	284
query70	1173	1129	1130	1129
query71	406	301	269	269
query72	5936	3791	3842	3791
query73	624	742	347	347
query74	10664	9122	8928	8928
query75	3204	3146	2681	2681
query76	3101	1168	775	775
query77	469	359	269	269
query78	10454	10377	9784	9784
query79	3831	894	581	581
query80	819	520	414	414
query81	518	263	223	223
query82	1060	120	94	94
query83	169	164	140	140
query84	248	93	76	76
query85	790	358	312	312
query86	395	320	309	309
query87	4327	4334	4247	4247
query88	5230	2384	2372	2372
query89	429	330	297	297
query90	1769	187	188	187
query91	132	140	111	111
query92	61	53	51	51
query93	3046	872	545	545
query94	706	399	300	300
query95	332	267	268	267
query96	487	607	277	277
query97	3184	3307	3099	3099
query98	222	212	198	198
query99	1668	1438	1309	1309
Total cold run time: 297024 ms
Total hot run time: 196604 ms

@doris-robot
Copy link

doris-robot commented Jul 31, 2025

ClickBench: Total hot run time: 28.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0acd2a9c72f6c3eae4a75cafdaa3f6d5a4fd6c21, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.50	0.49	0.51
query6	1.14	0.76	0.74
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.58	0.52	0.51
query10	0.55	0.55	0.54
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	0.78	0.83	0.81
query15	0.83	0.84	0.84
query16	0.38	0.39	0.40
query17	1.08	1.00	1.06
query18	0.24	0.22	0.23
query19	1.89	1.88	1.89
query20	0.02	0.01	0.02
query21	15.40	0.90	0.58
query22	0.76	0.99	0.60
query23	15.03	1.43	0.61
query24	3.35	1.64	0.63
query25	0.13	0.10	0.22
query26	0.36	0.16	0.14
query27	0.06	0.05	0.04
query28	13.61	0.99	0.43
query29	12.58	3.90	3.21
query30	0.25	0.08	0.06
query31	2.81	0.61	0.38
query32	3.22	0.53	0.46
query33	3.00	3.02	3.06
query34	16.56	5.26	4.55
query35	4.65	4.53	4.55
query36	0.66	0.52	0.48
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 103.85 s
Total hot run time: 28.64 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 31, 2025

FE UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow merged commit dba1a10 into branch-3.1 Aug 1, 2025
23 checks passed
@github-actions github-actions bot deleted the auto-pick-54031-branch-3.1 branch August 1, 2025 02:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Thearas @doris-robot @hello-stephen @morrySnow @dataroaring @cambyzju