Skip to content

Cleanup segment metadata cache for a datasource if not in use#17853

Merged
gianm merged 1 commit intoapache:masterfrom
kfaraz:cleanup_unused_datasource_cache
Apr 1, 2025
Merged

Cleanup segment metadata cache for a datasource if not in use#17853
gianm merged 1 commit intoapache:masterfrom
kfaraz:cleanup_unused_datasource_cache

Conversation

@kfaraz
Copy link
Copy Markdown
Contributor

@kfaraz kfaraz commented Apr 1, 2025

Follow up to #17824

This patch adds the implementation suggested in this comment

Changes

  • Simplify the contract of DatasourceSegmentCache and SegmentMetadataCache
  • Hold a reference on a HeapMemoryDatasourceSegmentCache while it is being used by a transaction
  • Clean up unreferenced instances of HeapMemoryDatasourceSegmentCache in the sync thread
  • Fix up tests

Read-Write Transaction flow

  • Start JDBI transaction
  • Create instance of SqlSegmentMetadataTransaction
  • Within SegmentMetadataCache.writeCacheForDatasource():
    • Get or create a cache instance for the datasource
    • Acquire a reference to the cache instance
    • Acquire write lock on the cache instance
    • Perform the write operation
    • Invoke SqlSegmentMetadataTransaction.close()
    • Release write lock on the cache instance
    • Release the reference to the cache instance
  • Finish JDBI transaction

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@kfaraz kfaraz changed the title Cleanup unused segment metadata cache for a datasource if not in use Cleanup segment metadata cache for a datasource if not in use Apr 1, 2025
@cryptoe cryptoe added this to the 33.0.0 milestone Apr 1, 2025
@kfaraz kfaraz requested a review from gianm April 1, 2025 11:17
@gianm gianm merged commit 295e701 into apache:master Apr 1, 2025
75 checks passed
@kfaraz kfaraz deleted the cleanup_unused_datasource_cache branch April 1, 2025 11:37
@kfaraz
Copy link
Copy Markdown
Contributor Author

kfaraz commented Apr 1, 2025

Thanks a lot for the review, @gianm !!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants