Skip to content

[opt](audit) use one line in audit log and origin statement in audit table (#52032) #52205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 2 commits into from
Jun 24, 2025
Merged

[opt](audit) use one line in audit log and origin statement in audit table (#52032) #52205

morningman merged 2 commits into from
Jun 24, 2025

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jun 24, 2025

bp #52032

@morningman
[opt](audit) use one line in audit log and origin statement in audit …
0928933 
…table (apache#52032)

Previously, when auditing, we use replace all `\n`, `\t` in origin sql
string with `\\n`, `\\t`,
so that the sql string can be written in one line. But this lead to some
problem:

1. User can not direct use the sql in audit log to execute.
2. Some replacement is wrong, eg, replace the `\n` in a quota string.

This PR changes the logic:

1. For audit log, only replace `\n` with `\\n` to keep SQL in one line.
2. For audit table, keep the origin string.
3. Use special column and line separator for audit log load data, to
avoid conflict with char in SQL
@morningman morningman requested a review from dataroaring as a code owner June 24, 2025 07:48
@hello-stephen
Copy link
Contributor

hello-stephen commented Jun 24, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman
Copy link
Contributor Author

morningman commented Jun 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-H: Total hot run time: 39682 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9f6af8255c503915a336d68703316ef8a9a883d, data reload: false

------ Round 1 ----------------------------------
q1	17589	6714	6570	6570
q2	2050	170	190	170
q3	10562	1088	1157	1088
q4	10567	798	683	683
q5	7773	2849	2824	2824
q6	211	136	136	136
q7	974	613	621	613
q8	9355	1923	2028	1923
q9	6596	6359	6401	6359
q10	7016	2275	2297	2275
q11	466	262	260	260
q12	390	215	214	214
q13	17777	2964	3028	2964
q14	249	210	214	210
q15	504	479	475	475
q16	515	378	378	378
q17	963	566	561	561
q18	7331	6633	6629	6629
q19	1402	1080	1073	1073
q20	481	210	203	203
q21	3838	3229	3100	3100
q22	1098	982	974	974
Total cold run time: 107707 ms
Total hot run time: 39682 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6565	6605	6534	6534
q2	326	234	232	232
q3	2868	2719	2755	2719
q4	2028	1813	1800	1800
q5	5813	5901	5847	5847
q6	214	132	138	132
q7	2255	1874	1898	1874
q8	3435	3645	3580	3580
q9	8979	8809	8935	8809
q10	3551	3546	3516	3516
q11	599	502	508	502
q12	814	627	597	597
q13	9919	3117	3133	3117
q14	299	268	283	268
q15	529	465	461	461
q16	487	443	427	427
q17	1844	1641	1585	1585
q18	8129	7720	7731	7720
q19	1685	1564	1568	1564
q20	2127	1879	1894	1879
q21	5056	5025	5057	5025
q22	1148	1030	1028	1028
Total cold run time: 68670 ms
Total hot run time: 59216 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-DS: Total hot run time: 196741 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9f6af8255c503915a336d68703316ef8a9a883d, data reload: false

query1	1287	910	907	907
query2	6364	1854	1861	1854
query3	10814	4326	4348	4326
query4	61862	29193	23596	23596
query5	5183	450	458	450
query6	410	182	183	182
query7	5463	317	318	317
query8	313	235	228	228
query9	8399	2542	2543	2542
query10	455	271	259	259
query11	17337	15218	15855	15218
query12	155	104	104	104
query13	1384	431	421	421
query14	9891	6841	7792	6841
query15	199	185	170	170
query16	7151	531	525	525
query17	1155	597	588	588
query18	1958	330	327	327
query19	206	165	163	163
query20	115	113	105	105
query21	209	103	114	103
query22	4596	4316	4613	4316
query23	34464	33801	33869	33801
query24	6156	3056	2923	2923
query25	551	423	437	423
query26	655	174	171	171
query27	1719	359	353	353
query28	3864	2150	2113	2113
query29	703	474	461	461
query30	238	161	159	159
query31	974	827	820	820
query32	72	60	59	59
query33	507	308	311	308
query34	924	508	516	508
query35	860	761	727	727
query36	1066	941	963	941
query37	112	69	70	69
query38	4114	3959	3965	3959
query39	1576	1458	1469	1458
query40	199	101	101	101
query41	51	51	47	47
query42	106	101	104	101
query43	549	498	478	478
query44	1180	803	835	803
query45	195	170	169	169
query46	1137	741	730	730
query47	2027	1896	1906	1896
query48	488	384	387	384
query49	724	417	424	417
query50	852	423	428	423
query51	7341	7177	7259	7177
query52	105	96	88	88
query53	265	196	187	187
query54	563	476	480	476
query55	77	78	82	78
query56	265	274	245	245
query57	1311	1216	1200	1200
query58	226	232	214	214
query59	3130	3017	3004	3004
query60	289	282	270	270
query61	155	114	108	108
query62	797	699	725	699
query63	223	196	193	193
query64	1406	680	646	646
query65	3325	3198	3190	3190
query66	709	296	302	296
query67	15982	15432	15572	15432
query68	4037	587	587	587
query69	419	264	261	261
query70	1181	1103	1065	1065
query71	344	274	260	260
query72	6404	4099	4131	4099
query73	747	341	351	341
query74	10132	9247	9260	9247
query75	3385	2645	2719	2645
query76	2027	1060	1112	1060
query77	480	285	273	273
query78	10685	9585	9593	9585
query79	1679	583	590	583
query80	1104	431	434	431
query81	528	222	227	222
query82	1270	93	92	92
query83	263	144	144	144
query84	290	80	82	80
query85	946	311	294	294
query86	334	294	309	294
query87	4475	4284	4297	4284
query88	4002	2361	2359	2359
query89	425	295	293	293
query90	2011	185	189	185
query91	180	148	148	148
query92	60	52	50	50
query93	2004	553	557	553
query94	744	293	284	284
query95	354	268	257	257
query96	605	281	283	281
query97	3288	3113	3162	3113
query98	213	196	203	196
query99	1581	1336	1328	1328
Total cold run time: 313335 ms
Total hot run time: 196741 ms

@morningman
Copy link
Contributor Author

morningman commented Jun 24, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-H: Total hot run time: 39464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9f6af8255c503915a336d68703316ef8a9a883d, data reload: false

------ Round 1 ----------------------------------
q1	17631	6689	6557	6557
q2	2076	161	161	161
q3	10721	1043	1168	1043
q4	10475	726	706	706
q5	7772	2813	2797	2797
q6	215	139	133	133
q7	965	609	598	598
q8	9354	1926	1918	1918
q9	6639	6371	6391	6371
q10	7013	2254	2248	2248
q11	470	261	258	258
q12	393	214	209	209
q13	17758	2943	2959	2943
q14	231	212	207	207
q15	507	463	473	463
q16	493	367	373	367
q17	971	617	562	562
q18	7366	6730	6637	6637
q19	1405	1055	1061	1055
q20	480	202	207	202
q21	4011	3177	3042	3042
q22	1055	1013	987	987
Total cold run time: 108001 ms
Total hot run time: 39464 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6597	6549	6567	6549
q2	327	234	229	229
q3	2879	2754	2754	2754
q4	2038	1758	1753	1753
q5	5769	5697	5707	5697
q6	207	132	125	125
q7	2218	1777	1809	1777
q8	3352	3483	3527	3483
q9	8844	8697	8898	8697
q10	3540	3516	3497	3497
q11	589	493	491	491
q12	799	608	573	573
q13	8434	3131	3083	3083
q14	296	267	288	267
q15	519	465	478	465
q16	483	430	437	430
q17	1849	1583	1608	1583
q18	8188	7777	7861	7777
q19	1673	1676	1609	1609
q20	2142	1864	1805	1805
q21	5121	4951	4900	4900
q22	1124	1048	1059	1048
Total cold run time: 66988 ms
Total hot run time: 58592 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

TPC-DS: Total hot run time: 194680 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9f6af8255c503915a336d68703316ef8a9a883d, data reload: false

query1	1327	910	897	897
query2	6404	1902	1870	1870
query3	10850	4293	4302	4293
query4	61126	29344	23452	23452
query5	5220	477	452	452
query6	412	170	194	170
query7	5464	310	308	308
query8	316	232	225	225
query9	8303	2547	2548	2547
query10	452	276	270	270
query11	17459	15228	15682	15228
query12	174	102	102	102
query13	1451	456	450	450
query14	9912	7637	7246	7246
query15	203	176	181	176
query16	7176	468	491	468
query17	1149	619	598	598
query18	1910	317	317	317
query19	223	164	164	164
query20	119	115	114	114
query21	209	112	109	109
query22	4715	4429	4652	4429
query23	34926	34142	33951	33951
query24	6220	2895	2948	2895
query25	526	421	418	418
query26	686	168	166	166
query27	2017	367	362	362
query28	4034	2140	2119	2119
query29	725	445	440	440
query30	240	154	162	154
query31	971	785	803	785
query32	67	55	55	55
query33	448	288	297	288
query34	923	505	520	505
query35	848	731	727	727
query36	1085	971	935	935
query37	112	68	68	68
query38	4071	3999	3977	3977
query39	1533	1456	1478	1456
query40	209	100	100	100
query41	48	47	45	45
query42	116	102	102	102
query43	518	471	468	468
query44	1207	815	817	815
query45	183	169	165	165
query46	1165	728	727	727
query47	2014	1840	1900	1840
query48	479	375	394	375
query49	755	385	391	385
query50	836	427	422	422
query51	7451	7260	7317	7260
query52	106	88	92	88
query53	260	196	194	194
query54	585	495	462	462
query55	81	81	78	78
query56	282	256	255	255
query57	1277	1246	1191	1191
query58	227	215	244	215
query59	3198	2982	2985	2982
query60	292	264	248	248
query61	112	107	115	107
query62	797	684	689	684
query63	215	189	188	188
query64	1381	638	644	638
query65	3260	3223	3195	3195
query66	704	298	293	293
query67	15880	15369	15440	15369
query68	4228	562	567	562
query69	418	262	266	262
query70	1158	1050	1119	1050
query71	352	245	260	245
query72	6377	3973	2448	2448
query73	752	336	345	336
query74	10440	8814	9027	8814
query75	3397	2627	2690	2627
query76	2059	1010	1097	1010
query77	473	279	264	264
query78	10534	9621	9581	9581
query79	1540	589	586	586
query80	873	425	434	425
query81	500	220	221	220
query82	1259	89	92	89
query83	240	149	152	149
query84	279	79	80	79
query85	903	300	296	296
query86	325	307	296	296
query87	4376	4240	4273	4240
query88	3619	2376	2332	2332
query89	415	290	289	289
query90	2048	185	187	185
query91	183	150	149	149
query92	69	48	51	48
query93	2002	551	556	551
query94	762	290	299	290
query95	352	264	257	257
query96	613	279	280	279
query97	3284	3119	3317	3119
query98	208	206	204	204
query99	1544	1338	1321	1321
Total cold run time: 313226 ms
Total hot run time: 194680 ms

@doris-robot
Copy link

doris-robot commented Jun 24, 2025

ClickBench: Total hot run time: 30.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9f6af8255c503915a336d68703316ef8a9a883d, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.62	0.10	0.10
query5	0.54	0.49	0.51
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.50
query10	0.54	0.55	0.55
query11	0.15	0.10	0.11
query12	0.14	0.10	0.11
query13	0.60	0.60	0.60
query14	0.78	0.79	0.82
query15	0.84	0.83	0.83
query16	0.38	0.37	0.38
query17	1.01	0.99	1.03
query18	0.22	0.23	0.23
query19	1.91	1.83	1.86
query20	0.01	0.01	0.01
query21	15.43	0.59	0.60
query22	2.54	1.70	1.20
query23	16.85	1.13	0.88
query24	3.64	2.05	1.81
query25	0.23	0.05	0.04
query26	0.69	0.14	0.13
query27	0.04	0.04	0.04
query28	8.37	0.52	0.47
query29	12.60	3.25	3.22
query30	0.24	0.06	0.05
query31	2.85	0.40	0.38
query32	3.23	0.47	0.46
query33	2.97	3.03	2.98
query34	16.97	4.49	4.48
query35	4.57	4.52	4.51
query36	0.66	0.48	0.47
query37	0.09	0.06	0.07
query38	0.05	0.04	0.03
query39	0.05	0.02	0.03
query40	0.16	0.13	0.11
query41	0.09	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.2 s
Total hot run time: 30.39 s

dataroaring
dataroaring approved these changes Jun 24, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 24, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 24, 2025

PR approved by anyone and no changes requested.

@morningman morningman merged commit cf68a64 into apache:branch-3.0 Jun 24, 2025
22 of 24 checks passed
@wanlce wanlce mentioned this pull request Jul 24, 2025
Closed
16 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @hello-stephen @doris-robot @dataroaring