Skip to content

[Metadata] Let entries expire in the metadata caches#14154

Merged
codelipenghui merged 2 commits intoapache:masterfrom
lhotari:lh-expire-entries-in-metadata-caches
Feb 8, 2022
Merged

[Metadata] Let entries expire in the metadata caches#14154
codelipenghui merged 2 commits intoapache:masterfrom
lhotari:lh-expire-entries-in-metadata-caches

Conversation

@lhotari
Copy link
Copy Markdown
Member

@lhotari lhotari commented Feb 7, 2022 •
edited
Loading

Motivation

  • There's essentially a memory leak in the Metadata caches since the entries never expire in the current solution.
  • refreshAfterWrite will never expire entries. After the given time, the next request to the entry will
    trigger a refresh in the background. The current entry will be used until the entry has been refreshed.

Modifications

  • expire after CACHE_REFRESH_TIME_MILLIS * 2

@lhotari
[Metadata] Let entries expire in the metadata caches
a183107 
- refreshAfterWrite will never expire entries. After the given time, the next request to the entry will
  trigger a refresh in the background. The current entry will be used until the entry has been refreshed.
  - documentation for Caffeine's refresh feature: https://github.com/ben-manes/caffeine/wiki/Refresh
@lhotari lhotari requested review from Jason918 and merlimat February 7, 2022 16:21
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 7, 2022

@lhotari:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

1 similar comment
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 7, 2022

@lhotari:Thanks for your contribution. For this PR, do we need to update docs?
(The PR template contains info about doc, which helps others know more about the changes. Can you provide doc-related info in this and future PR descriptions? Thanks)

@lhotari lhotari added the doc-not-needed Your PR changes do not impact docs label Feb 7, 2022
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 7, 2022

@lhotari:Thanks for providing doc info!

eolivelli
eolivelli approved these changes Feb 7, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@eolivelli eolivelli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change makes sense to me.

It will also allow to remove more quickly stale data probably

Jason918
Jason918 reviewed Feb 8, 2022
View reviewed changes
Copy link
Copy Markdown
Contributor

@Jason918 Jason918 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's better not to increase metadata load, how about

  • refresh after CACHE_REFRESH_TIME_MILLIS
  • expire after CACHE_REFRESH_TIME_MILLIS * 2

@lhotari
Copy link
Copy Markdown
Member Author

lhotari commented Feb 8, 2022

I think it's better not to increase metadata load, how about

  • refresh after CACHE_REFRESH_TIME_MILLIS
  • expire after CACHE_REFRESH_TIME_MILLIS * 2

@Jason918 thanks, I changed it in this way. PTAL

@codelipenghui codelipenghui merged commit 9165aed into apache:master Feb 8, 2022
michaeljmarshall
michaeljmarshall reviewed Feb 8, 2022
View reviewed changes
Copy link
Copy Markdown
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@codelipenghui codelipenghui added this to the 2.10.0 milestone Feb 21, 2022
Nicklee007 pushed a commit to Nicklee007/pulsar that referenced this pull request Apr 20, 2022
@lhotari
[Metadata] Let entries expire in the metadata caches (apache#14154)
5098824 
* [Metadata] Let entries expire in the metadata caches

- refreshAfterWrite will never expire entries. After the given time, the next request to the entry will
  trigger a refresh in the background. The current entry will be used until the entry has been refreshed.
  - documentation for Caffeine's refresh feature: https://github.com/ben-manes/caffeine/wiki/Refresh

* Address review comments
@Shawyeok Shawyeok mentioned this pull request May 19, 2023
Closed
@mattisonchao mattisonchao mentioned this pull request Jul 22, 2023
Merged
5 tasks
@codelipenghui codelipenghui mentioned this pull request Aug 30, 2023
Closed
14 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michaeljmarshall michaeljmarshall michaeljmarshall left review comments

@Jason918 Jason918 Jason918 approved these changes

@gaozhangmin gaozhangmin gaozhangmin approved these changes

@eolivelli eolivelli eolivelli approved these changes

@codelipenghui codelipenghui codelipenghui approved these changes

@nicoloboschi nicoloboschi nicoloboschi approved these changes

@mattisonchao mattisonchao mattisonchao approved these changes

@merlimat merlimat Awaiting requested review from merlimat

Assignees

@lhotari lhotari

Labels

doc-not-needed Your PR changes do not impact docs

Projects

None yet

Milestone

2.10.0

Development

Successfully merging this pull request may close these issues.

8 participants

@lhotari @Jason918 @gaozhangmin @eolivelli @codelipenghui @nicoloboschi @michaeljmarshall @mattisonchao