Skip to content

Conversation

@morningman
Copy link
Contributor

cherry pick from #42262

…error (apache#42262)

### What problem does this PR solve?
Problem Summary:

#### Issue 1
There are some fields that is missing in audit log table.
This PR add them all:
- shuffle_send_rows
- shuffle_send_bytes
- scan_bytes_from_local_storage
- scan_bytes_from_remote_storage
- is_nereids
- compute_group

Notice that `compute_group` is previously name `cloudClusterName` in
fe.audit.log,
which is incorrect, so I change it to the right name.

After this PR, all these fields will be saved in both audit log table
and fe.audit.log

#### Issue 2
The `AuditEventBuilder` need to be reset at each run, the there will be
duplicate audit log.

#### Issue 3
Add a new statement `call flush_audit_log()`. It will flush the audit
log immediately to audit_log table.
This is useful in test case, so that we don't need to wait 1min to flush
the audit log data.

### Release note

[fix](auditlog) add missing audit log fields and duplicate audit log error
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 40535 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3d03a8261523378c94494661f6aa0d9a2014e2c7, data reload: false

------ Round 1 ----------------------------------
q1	17597	7426	7438	7426
q2	2032	147	145	145
q3	10599	1092	1134	1092
q4	10227	742	690	690
q5	7760	2742	2790	2742
q6	237	145	141	141
q7	946	605	588	588
q8	9338	1879	1987	1879
q9	6621	6398	6383	6383
q10	6960	2275	2288	2275
q11	467	252	254	252
q12	409	210	207	207
q13	17786	2955	2965	2955
q14	242	201	207	201
q15	558	521	522	521
q16	654	608	589	589
q17	953	613	554	554
q18	7316	6625	6498	6498
q19	1383	1077	1068	1068
q20	471	193	187	187
q21	4024	3182	3187	3182
q22	1075	979	960	960
Total cold run time: 107655 ms
Total hot run time: 40535 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7233	7184	7194	7184
q2	319	244	225	225
q3	2867	2759	2713	2713
q4	1896	1624	1667	1624
q5	5396	5408	5425	5408
q6	216	137	138	137
q7	2067	1656	1650	1650
q8	3160	3321	3357	3321
q9	8459	8455	8468	8455
q10	3431	3347	3402	3347
q11	585	491	504	491
q12	748	563	570	563
q13	16859	2981	2996	2981
q14	294	275	280	275
q15	553	505	511	505
q16	687	656	660	656
q17	1796	1556	1537	1537
q18	7562	7428	7220	7220
q19	1639	1489	1484	1484
q20	2021	1800	1771	1771
q21	5130	5030	5093	5030
q22	1081	960	950	950
Total cold run time: 73999 ms
Total hot run time: 57527 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 188947 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3d03a8261523378c94494661f6aa0d9a2014e2c7, data reload: false

query1	976	367	365	365
query2	6518	2139	2019	2019
query3	6698	219	220	219
query4	34062	23536	23450	23450
query5	4306	428	429	428
query6	272	165	192	165
query7	4636	309	306	306
query8	292	230	231	230
query9	9678	2633	2600	2600
query10	460	254	264	254
query11	18168	14958	15144	14958
query12	150	98	98	98
query13	1623	417	402	402
query14	9658	6845	6672	6672
query15	220	169	175	169
query16	7833	498	469	469
query17	1624	566	562	562
query18	1995	302	299	299
query19	202	155	154	154
query20	121	110	109	109
query21	211	103	103	103
query22	4188	4264	4269	4264
query23	34569	33631	33507	33507
query24	12250	2875	2870	2870
query25	724	394	410	394
query26	1820	168	162	162
query27	2982	294	296	294
query28	8210	2483	2458	2458
query29	1139	424	435	424
query30	333	165	150	150
query31	1023	797	781	781
query32	96	57	58	57
query33	768	278	269	269
query34	996	466	497	466
query35	846	697	701	697
query36	1088	951	937	937
query37	279	79	75	75
query38	3948	3821	3835	3821
query39	1497	1471	1455	1455
query40	287	100	101	100
query41	54	51	51	51
query42	111	101	104	101
query43	519	500	481	481
query44	1255	779	787	779
query45	180	165	163	163
query46	1114	697	699	697
query47	1875	1814	1770	1770
query48	451	363	374	363
query49	1328	437	394	394
query50	791	401	414	401
query51	7182	6899	7017	6899
query52	110	92	91	91
query53	262	184	186	184
query54	1244	461	463	461
query55	80	82	76	76
query56	268	246	243	243
query57	1184	1100	1084	1084
query58	256	210	207	207
query59	3097	2879	2926	2879
query60	285	262	256	256
query61	122	121	162	121
query62	852	661	675	661
query63	216	191	190	190
query64	5315	612	585	585
query65	3289	3208	3176	3176
query66	1432	311	296	296
query67	16058	15182	15223	15182
query68	4902	548	547	547
query69	421	252	251	251
query70	1151	1115	1138	1115
query71	435	280	253	253
query72	6325	3918	3979	3918
query73	758	342	345	342
query74	10317	8878	8871	8871
query75	3339	2670	2640	2640
query76	2923	983	942	942
query77	393	267	263	263
query78	10558	9680	9498	9498
query79	8625	592	612	592
query80	2054	413	409	409
query81	570	245	240	240
query82	1426	113	114	113
query83	257	144	139	139
query84	291	74	81	74
query85	2285	291	266	266
query86	500	275	294	275
query87	4455	4333	4280	4280
query88	5833	2374	2400	2374
query89	506	280	288	280
query90	2090	178	182	178
query91	177	138	139	138
query92	68	47	47	47
query93	6168	549	535	535
query94	951	274	269	269
query95	349	243	246	243
query96	624	282	277	277
query97	3337	3093	3141	3093
query98	213	195	195	195
query99	1853	1287	1296	1287
Total cold run time: 319830 ms
Total hot run time: 188947 ms

@morningman morningman merged commit 1e93d71 into apache:branch-3.0 Nov 6, 2024
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants