Skip to content

[improvement](statistics)Remove read lock when doing db analyze. #49250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jibing-Li merged 1 commit into from
Mar 20, 2025
Merged

[improvement](statistics)Remove read lock when doing db analyze. #49250

Jibing-Li merged 1 commit into from
Mar 20, 2025

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Mar 19, 2025

What problem does this PR solve?

Remove read lock when doing db analyze. Don't need to lock db.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 19, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Jibing-Li Jibing-Li marked this pull request as ready for review March 19, 2025 08:20
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Mar 19, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Mar 19, 2025

TPC-H: Total hot run time: 32204 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ad5c9d9f4c09a03460c884dd9e4b374493bff0f, data reload: false

------ Round 1 ----------------------------------
q1	24150	5306	4994	4994
q2	2040	302	168	168
q3	10397	1255	685	685
q4	10218	1033	537	537
q5	7881	2455	2337	2337
q6	187	165	133	133
q7	900	743	596	596
q8	9303	1284	1052	1052
q9	5522	4772	4861	4772
q10	6825	2318	1884	1884
q11	476	268	261	261
q12	341	345	212	212
q13	17762	3604	3044	3044
q14	236	226	218	218
q15	525	474	492	474
q16	623	622	575	575
q17	578	853	343	343
q18	7192	6496	6253	6253
q19	1360	942	549	549
q20	319	338	200	200
q21	2761	2190	1932	1932
q22	1020	995	985	985
Total cold run time: 110616 ms
Total hot run time: 32204 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5167	5110	5097	5097
q2	239	337	230	230
q3	2126	2672	2316	2316
q4	1387	1803	1337	1337
q5	4226	4179	4548	4179
q6	208	165	124	124
q7	2012	1931	1743	1743
q8	2625	2652	2639	2639
q9	7145	7133	7273	7133
q10	2992	3205	2773	2773
q11	584	502	493	493
q12	666	758	638	638
q13	3422	3894	3265	3265
q14	272	297	258	258
q15	523	482	471	471
q16	665	672	641	641
q17	1125	1581	1333	1333
q18	7652	7505	7360	7360
q19	814	814	957	814
q20	1941	2014	1855	1855
q21	5380	4801	4705	4705
q22	1113	1013	984	984
Total cold run time: 52284 ms
Total hot run time: 50388 ms

@doris-robot
Copy link

doris-robot commented Mar 19, 2025

TPC-DS: Total hot run time: 184627 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1ad5c9d9f4c09a03460c884dd9e4b374493bff0f, data reload: false

query1	1014	483	472	472
query2	6542	1986	1904	1904
query3	6793	221	217	217
query4	26816	23349	23130	23130
query5	4302	629	464	464
query6	291	194	180	180
query7	4610	506	298	298
query8	292	241	236	236
query9	8610	2608	2607	2607
query10	474	316	251	251
query11	15702	15033	14795	14795
query12	159	111	108	108
query13	1657	503	397	397
query14	9431	6181	6262	6181
query15	206	185	168	168
query16	7203	622	449	449
query17	1196	704	553	553
query18	1951	414	311	311
query19	197	185	159	159
query20	130	121	124	121
query21	213	121	101	101
query22	4345	4283	4213	4213
query23	33929	32856	32993	32856
query24	8429	2332	2342	2332
query25	536	464	384	384
query26	1244	269	152	152
query27	2745	491	322	322
query28	4331	2399	2414	2399
query29	765	558	416	416
query30	287	214	192	192
query31	981	805	742	742
query32	71	62	62	62
query33	555	366	298	298
query34	780	827	492	492
query35	784	798	740	740
query36	958	989	885	885
query37	113	99	75	75
query38	4316	4084	4235	4084
query39	1482	1392	1414	1392
query40	203	121	114	114
query41	55	49	49	49
query42	117	106	101	101
query43	497	504	480	480
query44	1268	764	779	764
query45	170	169	161	161
query46	819	1019	619	619
query47	1737	1816	1735	1735
query48	374	405	294	294
query49	795	529	412	412
query50	685	742	400	400
query51	4198	4177	4134	4134
query52	107	104	96	96
query53	236	256	184	184
query54	478	487	408	408
query55	80	78	80	78
query56	263	287	259	259
query57	1095	1126	1072	1072
query58	242	237	233	233
query59	2772	2692	2654	2654
query60	286	277	259	259
query61	118	115	127	115
query62	775	720	690	690
query63	225	184	193	184
query64	4436	1034	657	657
query65	4378	4330	4357	4330
query66	1115	407	295	295
query67	15721	15554	15322	15322
query68	8763	866	500	500
query69	459	313	260	260
query70	1162	1107	1102	1102
query71	437	278	270	270
query72	5644	3505	3709	3505
query73	793	738	345	345
query74	8967	8959	8827	8827
query75	3977	3166	2705	2705
query76	3612	1179	736	736
query77	797	375	272	272
query78	10125	10151	9286	9286
query79	2750	832	595	595
query80	648	536	439	439
query81	470	261	220	220
query82	491	129	94	94
query83	203	166	150	150
query84	299	96	71	71
query85	773	355	318	318
query86	393	313	294	294
query87	4414	4397	4254	4254
query88	3492	2260	2261	2260
query89	400	315	275	275
query90	1926	214	219	214
query91	142	153	110	110
query92	80	61	57	57
query93	2740	1063	577	577
query94	653	402	299	299
query95	356	281	267	267
query96	475	562	274	274
query97	3382	3355	3323	3323
query98	225	207	201	201
query99	1461	1400	1290	1290
Total cold run time: 278194 ms
Total hot run time: 184627 ms

@doris-robot
Copy link

doris-robot commented Mar 19, 2025

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1ad5c9d9f4c09a03460c884dd9e4b374493bff0f, data reload: false

query1	0.04	0.03	0.03
query2	0.11	0.11	0.11
query3	0.24	0.19	0.19
query4	1.59	0.20	0.20
query5	0.59	0.58	0.60
query6	1.20	0.72	0.73
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.58	0.59	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.63	0.60	0.60
query14	2.77	2.70	2.67
query15	0.92	0.84	0.85
query16	0.37	0.37	0.38
query17	1.02	1.02	1.03
query18	0.22	0.20	0.20
query19	1.90	1.87	1.86
query20	0.01	0.01	0.01
query21	15.36	0.88	0.54
query22	0.75	1.20	0.74
query23	14.82	1.36	0.62
query24	7.03	2.69	0.54
query25	0.50	0.19	0.05
query26	0.56	0.16	0.13
query27	0.06	0.04	0.05
query28	9.46	0.82	0.43
query29	12.54	4.01	3.36
query30	0.25	0.10	0.07
query31	2.81	0.57	0.39
query32	3.23	0.56	0.46
query33	2.98	3.08	3.01
query34	15.86	5.13	4.51
query35	4.53	4.54	4.55
query36	0.66	0.49	0.48
query37	0.10	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.03	0.03
query42	0.04	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 105 s
Total hot run time: 30.95 s

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Mar 19, 2025

run cloud_p0

@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 19, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Mar 19, 2025

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Mar 19, 2025

run cloud_p0

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Mar 19, 2025

run cloud_p0

1 similar comment
@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Mar 20, 2025

run cloud_p0

@Jibing-Li Jibing-Li merged commit 6069438 into apache:master Mar 20, 2025
29 of 30 checks passed
This was referenced Mar 20, 2025
Merged
Merged
dataroaring pushed a commit that referenced this pull request Mar 21, 2025
@Jibing-Li
[improvement](statistics)Remove read lock when doing db analyze. (#49250
b4a0d42 
) (#49321)

backport: #49250
@Jibing-Li Jibing-Li deleted the lock branch March 21, 2025 02:17
yiguolei pushed a commit that referenced this pull request Mar 21, 2025
@Jibing-Li
[improvement](statistics)Remove read lock when doing db analyze. (#49250
7b28e33 
) (#49320)

backport: #49250
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@Jibing-Li
[improvement](statistics)Remove read lock when doing db analyze. (apa…
8e0b3c3 
…che#49250)

### What problem does this PR solve?

Remove read lock when doing db analyze. Don't need to lock db.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@Jibing-Li @Thearas @doris-robot @morningman @morrySnow @yiguolei @dataroaring