Skip to content

[fix](memory) Fix adjust cache capacity #45603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xinyiZzz merged 1 commit into from
Dec 19, 2024
Merged

[fix](memory) Fix adjust cache capacity #45603

xinyiZzz merged 1 commit into from
Dec 19, 2024

Conversation

@xinyiZzz
Copy link
Contributor

@xinyiZzz xinyiZzz commented Dec 18, 2024 •
edited
Loading

What problem does this PR solve?

If the cache capacity adjustment is not completed within 500ms (conf::memory_gc_sleep_time_ms), the next adjustment will be skipped.

In some scenarios, after Memory GC adjusts the cache capacity to 0, the next adjustment to restore the cache capacity is skipped, the cache capacity will remain at 0 for a long time.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 18, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@xinyiZzz
Copy link
Contributor Author

xinyiZzz commented Dec 18, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Dec 18, 2024

sh-checker report

To get the full details, please check in the job output.

shellcheck errors 

'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.
shfmt errors 
'shfmt ' found no issues.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 18, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Dec 18, 2024

TPC-H: Total hot run time: 39830 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7348e64c8e1d9492bd90e1339abf5ef37c5d0ef9, data reload: false

------ Round 1 ----------------------------------
q1	17585	7405	7263	7263
q2	2051	177	172	172
q3	10549	1152	1192	1152
q4	10233	701	732	701
q5	7597	2672	2714	2672
q6	237	151	144	144
q7	978	631	608	608
q8	9241	1853	1902	1853
q9	6491	6452	6435	6435
q10	6988	2316	2302	2302
q11	466	256	263	256
q12	411	231	216	216
q13	17787	2940	2934	2934
q14	244	226	215	215
q15	551	508	505	505
q16	640	580	587	580
q17	958	531	529	529
q18	7266	6734	6676	6676
q19	1332	1045	1100	1045
q20	488	183	188	183
q21	4023	3169	3072	3072
q22	377	317	326	317
Total cold run time: 106493 ms
Total hot run time: 39830 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7214	7208	7186	7186
q2	324	232	230	230
q3	2867	2800	2914	2800
q4	2084	1812	1817	1812
q5	5629	5684	5600	5600
q6	224	143	142	142
q7	2158	1752	1832	1752
q8	3356	3528	3555	3528
q9	8893	8896	8878	8878
q10	3640	3552	3552	3552
q11	606	517	499	499
q12	800	644	635	635
q13	13709	3105	3027	3027
q14	324	263	282	263
q15	578	504	502	502
q16	672	664	626	626
q17	1828	1576	1579	1576
q18	7809	7576	7330	7330
q19	1662	1624	1537	1537
q20	2048	1834	1846	1834
q21	5455	5298	5251	5251
q22	675	560	575	560
Total cold run time: 72555 ms
Total hot run time: 59120 ms

@doris-robot
Copy link

doris-robot commented Dec 18, 2024

TPC-DS: Total hot run time: 189778 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7348e64c8e1d9492bd90e1339abf5ef37c5d0ef9, data reload: false

query1	979	387	373	373
query2	6534	2327	2281	2281
query3	6722	215	219	215
query4	34035	23425	23214	23214
query5	4376	486	460	460
query6	292	189	183	183
query7	4639	303	308	303
query8	324	274	235	235
query9	9605	2740	2748	2740
query10	472	255	252	252
query11	18110	15053	15137	15053
query12	157	103	102	102
query13	1665	435	414	414
query14	9529	7571	6868	6868
query15	277	182	180	180
query16	8029	422	427	422
query17	1641	564	559	559
query18	1840	294	300	294
query19	364	152	154	152
query20	122	110	113	110
query21	205	102	103	102
query22	4599	4382	4154	4154
query23	34403	33728	34025	33728
query24	11433	2470	2468	2468
query25	666	374	409	374
query26	1786	154	153	153
query27	2911	332	326	326
query28	8178	2427	2440	2427
query29	995	410	403	403
query30	296	154	146	146
query31	1050	808	818	808
query32	102	57	64	57
query33	761	299	291	291
query34	999	522	505	505
query35	865	716	764	716
query36	1073	929	948	929
query37	246	75	72	72
query38	4221	4001	4077	4001
query39	1467	1446	1471	1446
query40	281	103	98	98
query41	48	43	45	43
query42	115	100	102	100
query43	529	480	491	480
query44	1267	817	820	817
query45	182	165	180	165
query46	1151	714	704	704
query47	1930	1804	1820	1804
query48	417	317	326	317
query49	1280	401	420	401
query50	801	388	388	388
query51	7138	7076	7127	7076
query52	100	96	89	89
query53	260	180	191	180
query54	1251	421	426	421
query55	91	81	77	77
query56	252	248	246	246
query57	1253	1107	1088	1088
query58	233	214	222	214
query59	3097	2983	2894	2894
query60	258	248	247	247
query61	112	106	103	103
query62	867	678	655	655
query63	221	188	191	188
query64	5130	666	757	666
query65	3242	3195	3246	3195
query66	1384	310	315	310
query67	15955	15453	15515	15453
query68	5070	558	565	558
query69	435	248	255	248
query70	1154	1120	1116	1116
query71	353	255	252	252
query72	6381	4082	4121	4082
query73	765	361	367	361
query74	10327	8916	8830	8830
query75	3403	2634	2620	2620
query76	3107	1109	1177	1109
query77	508	274	274	274
query78	10179	9492	9347	9347
query79	1368	611	592	592
query80	1110	440	432	432
query81	529	227	231	227
query82	1090	126	121	121
query83	251	158	146	146
query84	241	78	78	78
query85	1413	392	305	305
query86	369	298	291	291
query87	4528	4517	4503	4503
query88	3616	2237	2202	2202
query89	403	293	289	289
query90	2022	189	191	189
query91	142	101	107	101
query92	64	53	52	52
query93	1127	546	541	541
query94	1047	280	288	280
query95	358	255	252	252
query96	613	279	287	279
query97	2830	2666	2667	2666
query98	219	196	194	194
query99	1561	1328	1306	1306
Total cold run time: 301396 ms
Total hot run time: 189778 ms

@doris-robot
Copy link

doris-robot commented Dec 18, 2024

TeamCity be ut coverage result:
Function Coverage: 38.88% (10134/26066)
Line Coverage: 29.78% (85173/285969)
Region Coverage: 28.84% (43693/151524)
Branch Coverage: 25.37% (22196/87482)
Coverage Report: http://coverage.selectdb-in.cc/coverage/7348e64c8e1d9492bd90e1339abf5ef37c5d0ef9_7348e64c8e1d9492bd90e1339abf5ef37c5d0ef9/report/index.html

@doris-robot
Copy link

doris-robot commented Dec 18, 2024

ClickBench: Total hot run time: 32.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7348e64c8e1d9492bd90e1339abf5ef37c5d0ef9, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.43	0.41	0.41
query6	1.16	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.53	0.50
query10	0.55	0.58	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.60	0.61
query14	2.72	2.75	2.89
query15	0.90	0.83	0.82
query16	0.39	0.38	0.38
query17	1.07	1.03	1.02
query18	0.22	0.21	0.21
query19	1.88	1.87	1.97
query20	0.01	0.01	0.02
query21	15.38	0.60	0.57
query22	2.44	3.06	1.93
query23	16.92	1.15	0.79
query24	2.86	0.99	1.27
query25	0.14	0.11	0.16
query26	0.51	0.15	0.14
query27	0.05	0.04	0.04
query28	10.85	1.10	1.07
query29	12.55	3.21	3.21
query30	0.25	0.06	0.06
query31	2.87	0.41	0.38
query32	3.25	0.46	0.47
query33	3.13	3.25	3.07
query34	16.93	4.45	4.45
query35	4.52	4.50	4.51
query36	0.69	0.47	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.03
query40	0.17	0.12	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 106.69 s
Total hot run time: 32.79 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 19, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 19, 2024

PR approved by anyone and no changes requested.

wangbo
wangbo approved these changes Dec 19, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit b1ccd36 into apache:master Dec 19, 2024
32 of 36 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 19, 2024
@xinyiZzz
[fix](memory) Fix adjust cache capacity (#45603)
6228386 
### What problem does this PR solve?

If the cache capacity adjustment is not completed within 500ms
(conf::memory_gc_sleep_time_ms), the next adjustment will be skipped.

In some scenarios, after Memory GC adjusts the cache capacity to 0, the
next adjustment to restore the cache capacity is skipped, the cache
capacity will remain at 0 for a long time.
@github-actions github-actions bot mentioned this pull request Dec 19, 2024
Merged
dataroaring pushed a commit that referenced this pull request Dec 19, 2024
@github-actions @xinyiZzz
branch-3.0: [fix](memory) Fix adjust cache capacity #45603 (#45665)
d84ee07 
Cherry-picked from #45603

Co-authored-by: Xinyi Zou <zouxinyi@selectdb.com>
BiteTheDDDDt pushed a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Feb 7, 2025
@xinyiZzz
[fix](memory) Fix adjust cache capacity apache#45603 (apache#3720)
0fab330
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@wangbo wangbo wangbo approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.4-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@xinyiZzz @Thearas @doris-robot @yiguolei @wangbo @dataroaring