Skip to content

branch-3.1: [fix](audit)Fixed an issue that the audit log would record the previous queryId when parseSQL fails. #53107 #54007

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 1 commit into from
Jul 29, 2025
Merged

branch-3.1: [fix](audit)Fixed an issue that the audit log would record the previous queryId when parseSQL fails. #53107 #54007

morrySnow merged 1 commit into from
Jul 29, 2025

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jul 29, 2025 •
edited by morrySnow
Loading

picked from #53107

@zddr
[fix](audit)Fixed an issue that the audit log would record the previo…
5b7674f 
…us queryId when parseSQL fails. (apache#53107)

### What problem does this PR solve?

Issue:
- When parsing SQL, the queryId has not yet been generated. However, the
audit log retrieves the queryId from ConnectContext, which means it uses
the queryId from the previous execution.
- The queryId stored in queryCache is from the previous query, not the
current one.

Solution:
- Since parsing may produce multiple SQL statements, each should have
its own queryId. Therefore, the queryId should not be set during
parsing. As a compromise, we clear the queryId before parsing, ensuring
that the audit log records an empty queryId if parsing fails.
- Set the queryId when adding the cache, not in the constructor.
# Conflicts:
#	fe/fe-core/src/main/java/org/apache/doris/qe/ConnectContext.java
@zddr zddr requested a review from morrySnow as a code owner July 29, 2025 03:29
@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 29, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Jul 29, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jul 29, 2025

TPC-H: Total hot run time: 32546 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5b7674f3e1ea9ff59b00b3b430a1ca952d590d32, data reload: false

------ Round 1 ----------------------------------
q1	17588	5424	5466	5424
q2	2050	284	155	155
q3	10453	1319	726	726
q4	10194	852	447	447
q5	7647	2354	2122	2122
q6	183	163	133	133
q7	900	740	606	606
q8	9329	1413	1173	1173
q9	5326	4951	4896	4896
q10	6748	2272	1812	1812
q11	490	279	277	277
q12	335	352	203	203
q13	17788	3634	3004	3004
q14	234	228	209	209
q15	530	470	479	470
q16	406	421	374	374
q17	590	852	361	361
q18	6838	6522	6470	6470
q19	1217	956	556	556
q20	330	350	206	206
q21	2802	2155	1945	1945
q22	1063	1037	977	977
Total cold run time: 103041 ms
Total hot run time: 32546 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5494	5449	5482	5449
q2	234	325	235	235
q3	2218	2646	2306	2306
q4	1432	1775	1370	1370
q5	4391	4804	4958	4804
q6	173	164	131	131
q7	2078	1963	1813	1813
q8	2559	2712	2705	2705
q9	7234	7143	7118	7118
q10	3020	3270	2764	2764
q11	568	509	507	507
q12	631	749	610	610
q13	3393	3827	3081	3081
q14	276	297	261	261
q15	515	476	475	475
q16	456	483	431	431
q17	1232	1730	1280	1280
q18	7610	7446	7305	7305
q19	779	1019	1139	1019
q20	1990	2008	1874	1874
q21	5186	4837	4519	4519
q22	1137	1078	1029	1029
Total cold run time: 52606 ms
Total hot run time: 51086 ms

@doris-robot
Copy link

doris-robot commented Jul 29, 2025

TPC-DS: Total hot run time: 189995 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5b7674f3e1ea9ff59b00b3b430a1ca952d590d32, data reload: false

query1	1008	389	389	389
query2	6524	1895	1847	1847
query3	6715	219	234	219
query4	33613	23734	23408	23408
query5	4336	599	474	474
query6	284	201	187	187
query7	4628	497	325	325
query8	294	254	234	234
query9	9518	2587	2591	2587
query10	486	354	272	272
query11	18355	15349	15117	15117
query12	156	112	105	105
query13	1660	557	429	429
query14	9914	6559	6525	6525
query15	212	197	179	179
query16	7810	648	524	524
query17	1449	745	604	604
query18	2009	425	320	320
query19	214	180	166	166
query20	119	117	109	109
query21	213	125	106	106
query22	4632	4643	4588	4588
query23	34831	33630	33640	33630
query24	7685	2624	2599	2599
query25	525	465	412	412
query26	1219	286	169	169
query27	2186	463	339	339
query28	5130	2146	2116	2116
query29	825	581	452	452
query30	258	190	162	162
query31	963	868	823	823
query32	92	61	56	56
query33	542	362	331	331
query34	755	832	504	504
query35	743	817	709	709
query36	978	1061	944	944
query37	97	91	69	69
query38	3906	3886	3797	3797
query39	1487	1423	1430	1423
query40	201	119	114	114
query41	53	50	49	49
query42	123	108	107	107
query43	504	499	467	467
query44	1245	789	791	789
query45	180	174	180	174
query46	863	1056	656	656
query47	1871	1932	1829	1829
query48	409	410	357	357
query49	775	485	399	399
query50	637	663	417	417
query51	7233	7273	7030	7030
query52	98	99	98	98
query53	229	257	181	181
query54	535	535	469	469
query55	81	76	80	76
query56	271	257	252	252
query57	1264	1226	1158	1158
query58	233	208	223	208
query59	2856	2946	2931	2931
query60	290	284	264	264
query61	115	115	110	110
query62	810	725	642	642
query63	222	188	185	185
query64	4576	962	622	622
query65	3278	3179	3197	3179
query66	1064	411	304	304
query67	15975	15718	15504	15504
query68	7913	831	527	527
query69	487	294	278	278
query70	1193	1143	1072	1072
query71	482	290	261	261
query72	5660	3726	3769	3726
query73	621	751	350	350
query74	10312	9131	9195	9131
query75	3187	3169	2657	2657
query76	3125	1174	755	755
query77	462	372	283	283
query78	10272	10500	9610	9610
query79	3317	888	600	600
query80	786	524	445	445
query81	493	253	217	217
query82	712	113	89	89
query83	170	165	143	143
query84	248	103	84	84
query85	776	355	300	300
query86	403	311	311	311
query87	4302	4299	4280	4280
query88	5343	2465	2417	2417
query89	400	339	299	299
query90	1848	185	189	185
query91	136	138	108	108
query92	68	56	53	53
query93	2624	917	537	537
query94	701	404	299	299
query95	350	286	263	263
query96	485	621	283	283
query97	3230	3286	3151	3151
query98	244	214	197	197
query99	1507	1417	1264	1264
Total cold run time: 294219 ms
Total hot run time: 189995 ms

@doris-robot
Copy link

doris-robot commented Jul 29, 2025

ClickBench: Total hot run time: 28.87 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5b7674f3e1ea9ff59b00b3b430a1ca952d590d32, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.04
query3	0.25	0.07	0.06
query4	1.61	0.10	0.10
query5	0.53	0.49	0.55
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.58	0.50	0.50
query10	0.55	0.56	0.56
query11	0.15	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	0.78	0.78	0.82
query15	0.84	0.84	0.81
query16	0.37	0.39	0.39
query17	1.06	1.05	0.97
query18	0.23	0.23	0.23
query19	1.94	1.77	1.86
query20	0.02	0.01	0.01
query21	15.42	0.89	0.56
query22	0.74	0.78	0.73
query23	15.03	1.42	0.59
query24	3.55	1.02	1.65
query25	0.14	0.18	0.07
query26	0.27	0.15	0.15
query27	0.06	0.04	0.05
query28	13.82	0.95	0.44
query29	12.55	3.97	3.26
query30	0.26	0.09	0.06
query31	2.82	0.59	0.39
query32	3.22	0.53	0.45
query33	2.95	2.99	3.03
query34	16.72	5.19	4.53
query35	4.59	4.53	4.52
query36	0.64	0.48	0.48
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.14	0.13
query41	0.07	0.03	0.03
query42	0.03	0.03	0.02
query43	0.03	0.03	0.02
Total cold run time: 104.17 s
Total hot run time: 28.87 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Jul 29, 2025

FE UT Coverage Report

Increment line coverage 45.45% (5/11) 🎉
Increment coverage report
Complete coverage report

@morrySnow morrySnow changed the title branch-3.1:[fix](audit)Fixed an issue that the audit log would record the previo… branch-3.1: [fix](audit)Fixed an issue that the audit log would record the previous queryId when parseSQL fails. #53107 Jul 29, 2025
@morrySnow morrySnow merged commit 6b0c1af into apache:branch-3.1 Jul 29, 2025
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@zddr @hello-stephen @doris-robot @morrySnow