[feature](audit) add dynamic configuration for sql_digest generation #59102

xuchenhao · 2025-12-17T02:52:30Z

To analyze SQL patterns across the entire Doris cluster using sql_digest and implement workload optimization based on SQL patterns, a more flexible sql_digest generation mechanism is required.

Previously, sql_digest generation was implicitly tied to the qe_slow_log_ms configuration - digests were only generated for queries considered "slow". This coupling limited flexibility in audit logging.

This PR introduces a new dynamic configuration sql_digest_generation_threshold_ms to optimize the generation of sql_digest in audit logging.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-12-17T02:52:34Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

xuchenhao · 2025-12-17T03:02:09Z

run buildall

doris-robot · 2025-12-17T03:21:36Z

TPC-H: Total hot run time: 36123 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b3d07db86dee59ba8ba7ca562edebd681d6eeb90, data reload: false

------ Round 1 ----------------------------------
q1	17628	4303	4083	4083
q2	2013	366	241	241
q3	10182	1330	762	762
q4	10220	800	310	310
q5	7549	2138	1882	1882
q6	187	168	133	133
q7	1028	853	720	720
q8	9376	1458	1115	1115
q9	7052	5291	5361	5291
q10	6795	2405	1974	1974
q11	523	327	302	302
q12	662	749	588	588
q13	17767	3636	3053	3053
q14	288	300	267	267
q15	571	539	535	535
q16	707	681	651	651
q17	712	782	587	587
q18	7493	7597	7825	7597
q19	1141	1027	663	663
q20	422	391	248	248
q21	4609	4409	4076	4076
q22	1079	1048	1045	1045
Total cold run time: 108004 ms
Total hot run time: 36123 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4476	4221	4211	4211
q2	326	392	312	312
q3	2385	3071	2462	2462
q4	1446	1883	1362	1362
q5	4529	4679	4430	4430
q6	214	167	121	121
q7	1901	2008	1955	1955
q8	2717	2434	2480	2434
q9	7532	7590	7541	7541
q10	3085	3212	2863	2863
q11	621	523	527	523
q12	693	758	606	606
q13	3549	3708	3026	3026
q14	268	294	264	264
q15	539	496	490	490
q16	605	641	590	590
q17	1092	1339	1303	1303
q18	7239	7155	7006	7006
q19	840	792	863	792
q20	1893	1998	1822	1822
q21	4663	4296	4130	4130
q22	1061	1004	1004	1004
Total cold run time: 51674 ms
Total hot run time: 49247 ms

doris-robot · 2025-12-17T03:32:47Z

TPC-DS: Total hot run time: 178492 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b3d07db86dee59ba8ba7ca562edebd681d6eeb90, data reload: false

query5	5401	616	462	462
query6	354	252	227	227
query7	4229	478	285	285
query8	340	254	244	244
query9	8759	2525	2531	2525
query10	572	386	324	324
query11	15381	14793	14580	14580
query12	186	114	116	114
query13	1268	503	413	413
query14	6729	3011	2748	2748
query14_1	2656	2613	2640	2613
query15	214	203	182	182
query16	940	473	481	473
query17	1132	737	613	613
query18	2717	453	355	355
query19	232	235	214	214
query20	124	117	114	114
query21	222	150	121	121
query22	4035	3939	3905	3905
query23	16733	16542	16106	16106
query23_1	16131	16120	16027	16027
query24	7355	1643	1242	1242
query24_1	1248	1241	1237	1237
query25	577	518	452	452
query26	1264	272	163	163
query27	2737	470	309	309
query28	4457	2158	2154	2154
query29	865	582	436	436
query30	330	248	220	220
query31	826	701	626	626
query32	78	74	69	69
query33	538	329	289	289
query34	901	924	571	571
query35	783	821	728	728
query36	866	896	845	845
query37	137	90	76	76
query38	2850	2851	2843	2843
query39	739	746	722	722
query39_1	693	710	703	703
query40	232	140	123	123
query41	70	63	61	61
query42	111	103	104	103
query43	427	446	398	398
query44	1349	750	752	750
query45	192	192	187	187
query46	887	978	612	612
query47	1681	1710	1650	1650
query48	332	324	253	253
query49	628	438	352	352
query50	683	292	220	220
query51	3843	3820	3813	3813
query52	105	110	98	98
query53	320	354	288	288
query54	290	278	249	249
query55	80	72	71	71
query56	289	299	300	299
query57	1160	1123	1062	1062
query58	274	266	259	259
query59	2445	2567	2411	2411
query60	314	314	296	296
query61	164	159	168	159
query62	700	676	636	636
query63	329	289	297	289
query64	4941	1331	1011	1011
query65	4025	3969	3947	3947
query66	1390	457	312	312
query67	15171	14847	14907	14847
query68	8260	1015	725	725
query69	492	349	318	318
query70	1066	993	945	945
query71	391	308	283	283
query72	6064	4949	5036	4949
query73	691	614	310	310
query74	8917	8715	8603	8603
query75	3215	3153	2845	2845
query76	3870	1150	730	730
query77	524	390	293	293
query78	9401	9600	8852	8852
query79	1764	886	610	610
query80	727	661	561	561
query81	508	273	234	234
query82	404	137	103	103
query83	261	255	237	237
query84	256	121	105	105
query85	899	504	458	458
query86	388	304	279	279
query87	3038	3064	2942	2942
query88	3710	2290	2275	2275
query89	458	433	380	380
query90	2222	167	162	162
query91	176	166	144	144
query92	84	70	67	67
query93	2462	903	552	552
query94	499	305	279	279
query95	575	382	316	316
query96	590	478	207	207
query97	2273	2324	2264	2264
query98	245	192	193	192
query99	1256	1287	1215	1215
Total cold run time: 263666 ms
Total hot run time: 178492 ms

doris-robot · 2025-12-17T03:37:52Z

ClickBench: Total hot run time: 27.51 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b3d07db86dee59ba8ba7ca562edebd681d6eeb90, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.60	0.11	0.10
query5	0.28	0.26	0.25
query6	1.20	0.63	0.62
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.57	0.52	0.50
query10	0.56	0.56	0.55
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.60
query14	0.98	0.97	0.98
query15	0.83	0.80	0.81
query16	0.42	0.39	0.42
query17	0.98	1.03	1.04
query18	0.23	0.21	0.21
query19	1.87	1.85	1.86
query20	0.02	0.02	0.01
query21	15.43	0.28	0.14
query22	4.78	0.05	0.04
query23	15.86	0.28	0.10
query24	2.19	0.48	0.72
query25	0.12	0.05	0.13
query26	0.14	0.14	0.13
query27	0.12	0.07	0.06
query28	5.81	1.20	1.03
query29	12.59	4.07	3.26
query30	0.28	0.14	0.12
query31	2.83	0.62	0.40
query32	3.24	0.54	0.45
query33	2.94	3.05	3.01
query34	16.70	5.12	4.53
query35	4.54	4.56	4.54
query36	0.70	0.51	0.49
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 99.77 s
Total hot run time: 27.51 s

hello-stephen · 2025-12-17T04:36:54Z

FE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉
Increment coverage report
Complete coverage report

github-actions · 2025-12-17T07:59:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-12-17T07:59:38Z

PR approved by anyone and no changes requested.

…59102) To analyze SQL patterns across the entire Doris cluster using sql_digest and implement workload optimization based on SQL patterns, a more flexible sql_digest generation mechanism is required. Previously, sql_digest generation was implicitly tied to the `qe_slow_log_ms` configuration - digests were only generated for queries considered "slow". This coupling limited flexibility in audit logging. This PR introduces a new dynamic configuration `sql_digest_generation_threshold_ms` to optimize the generation of sql_digest in audit logging. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… generation #59102 (#59120) Cherry-picked from #59102 Co-authored-by: xuchenhao <48084123+xuchenhao@users.noreply.github.com>

…59102) To analyze SQL patterns across the entire Doris cluster using sql_digest and implement workload optimization based on SQL patterns, a more flexible sql_digest generation mechanism is required. Previously, sql_digest generation was implicitly tied to the `qe_slow_log_ms` configuration - digests were only generated for queries considered "slow". This coupling limited flexibility in audit logging. This PR introduces a new dynamic configuration `sql_digest_generation_threshold_ms` to optimize the generation of sql_digest in audit logging. ### What problem does this PR solve? Issue Number: close #xxx Related PR: #xxx Problem Summary: ### Release note None ### Check List (For Author) - Test  - [x] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [x] No. - [ ] Yes.  - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

… generation #59102 (#59343) picked from #59102

xuchenhao added 3 commits December 16, 2025 15:05

Add dynamic configuration parameter for sql_digest generation.

2d8ad28

Add some descriptions.

66e2a7d

Add rt cases.

b3d07db

xuchenhao changed the title ~~[feature](audit) add dynamic configuration for sql_digest generation~~ [draft][feature](audit) add dynamic configuration for sql_digest generation Dec 17, 2025

xuchenhao changed the title ~~[draft][feature](audit) add dynamic configuration for sql_digest generation~~ [feature](audit) add dynamic configuration for sql_digest generation Dec 17, 2025

yiguolei added usercase Important user case type label dev/4.0.x labels Dec 17, 2025

morrySnow approved these changes Dec 17, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 17, 2025

github-actions bot added the reviewed label Dec 17, 2025

yiguolei approved these changes Dec 17, 2025

View reviewed changes

yiguolei merged commit 2cacaf5 into apache:master Dec 17, 2025
39 of 41 checks passed

github-actions bot mentioned this pull request Dec 17, 2025

branch-4.0: [feature](audit) add dynamic configuration for sql_digest generation #59102 #59120

Merged

xuchenhao deleted the sql_digest branch December 18, 2025 00:16

yiguolei pushed a commit that referenced this pull request Dec 19, 2025

branch-4.0: [feature](audit) add dynamic configuration for sql_digest…

c628b6d

… generation #59102 (#59120) Cherry-picked from #59102 Co-authored-by: xuchenhao <48084123+xuchenhao@users.noreply.github.com>

yiguolei added dev/4.0.3-merged dev/3.1.x and removed dev/4.0.x labels Dec 19, 2025

github-actions bot mentioned this pull request Dec 19, 2025

branch-3.1: [feature](audit) add dynamic configuration for sql_digest generation #59102 #59184

Closed

morrySnow added the dev/3.1.x-conflict label Dec 19, 2025

xuchenhao changed the title ~~[feature](audit) add dynamic configuration for sql_digest generation~~ branch-3.1: [feature](audit) add dynamic configuration for sql_digest generation Dec 25, 2025

xuchenhao changed the title ~~branch-3.1: [feature](audit) add dynamic configuration for sql_digest generation~~ [feature](audit) add dynamic configuration for sql_digest generation Dec 25, 2025

cambyzju mentioned this pull request Dec 25, 2025

branch-3.1: [feature](audit) add dynamic configuration for sql_digest generation #59102 #59343

Merged

morrySnow pushed a commit that referenced this pull request Dec 25, 2025

branch-3.1: [feature](audit) add dynamic configuration for sql_digest…

d5f02e7

… generation #59102 (#59343) picked from #59102

morrySnow added dev/3.1.4-merged and removed dev/3.1.x dev/3.1.x-conflict labels Dec 25, 2025

yiguolei mentioned this pull request Jan 23, 2026

4.0.3 Release Notes #60193

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](audit) add dynamic configuration for sql_digest generation #59102

[feature](audit) add dynamic configuration for sql_digest generation #59102

Uh oh!

xuchenhao commented Dec 17, 2025 •

edited

Loading

Uh oh!

hello-stephen commented Dec 17, 2025

Uh oh!

xuchenhao commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

hello-stephen commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

[feature](audit) add dynamic configuration for sql_digest generation #59102

[feature](audit) add dynamic configuration for sql_digest generation #59102

Uh oh!

Conversation

xuchenhao commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Dec 17, 2025

Uh oh!

xuchenhao commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

doris-robot commented Dec 17, 2025

Uh oh!

hello-stephen commented Dec 17, 2025

FE UT Coverage Report

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

xuchenhao commented Dec 17, 2025 •

edited

Loading