Skip to content

Upgrade pending segments transactionally#15992

Closed
AmatyaAvadhanula wants to merge 3 commits intoapache:masterfrom
AmatyaAvadhanula:pending_segment_upgrade_in_transaction
Closed

Upgrade pending segments transactionally#15992
AmatyaAvadhanula wants to merge 3 commits intoapache:masterfrom
AmatyaAvadhanula:pending_segment_upgrade_in_transaction

Conversation

@AmatyaAvadhanula
Copy link
Copy Markdown
Contributor

The segment transactional replace action facilitates concurrent replace with streaming append by

  1. Upgrading pending segments in the metadata store
  2. Relaying this information to the streaming tasks so that they may update their internal state for queries

Presently both the steps happen outside the transaction in which the replacing segments are committed.
This PR aims to fix potential conflicts by ensuring that the pending segment upgrade happens in the same transaction in which the segments are committed.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

activeRealtimeSequencePrefixes = ImmutableSet.of();
} else {
activeRealtimeSequencePrefixes
= supervisorManager.getActiveRealtimeSequencePrefixes(activeSupervisorIdWithAppendLock.get());

Check warning

Code scanning / CodeQL

Dereferenced variable may be null

Variable [supervisorManager](1) may be null at this access as suggested by [this](2) null guard.
@AmatyaAvadhanula AmatyaAvadhanula marked this pull request as draft February 29, 2024 06:00
@AmatyaAvadhanula AmatyaAvadhanula marked this pull request as ready for review February 29, 2024 06:31
@AmatyaAvadhanula
Copy link
Copy Markdown
Contributor Author

This is being done as part of #16144

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants