Track IngestionState more accurately in realtime tasks.#16934
Merged
kfaraz merged 3 commits intoapache:masterfrom Aug 22, 2024
Merged
Track IngestionState more accurately in realtime tasks.#16934kfaraz merged 3 commits intoapache:masterfrom
kfaraz merged 3 commits intoapache:masterfrom
Conversation
Previously, SeekableStreamIndexTaskRunner set ingestion state to COMPLETED when it finished reading data from Kafka. This is incorrect. After the changes in this patch, the transitions go: 1) The task stays in BUILD_SEGMENTS after it finishes reading from Kafka, while it is building its final set of segments to publish. 2) The task transitions to SEGMENT_AVAILABILITY_WAIT after publishing, while waiting for handoff. 3) The task transitions to COMPLETED immediately before exiting, when truly done.
AmatyaAvadhanula
approved these changes
Aug 21, 2024
Contributor
Author
|
The CI failures were one flaky test case in |
kfaraz
approved these changes
Aug 22, 2024
hevansDev
pushed a commit
to hevansDev/druid
that referenced
this pull request
Aug 29, 2024
Previously, SeekableStreamIndexTaskRunner set ingestion state to COMPLETED when it finished reading data from Kafka. This is incorrect. After the changes in this patch, the transitions go: 1) The task stays in BUILD_SEGMENTS after it finishes reading from Kafka, while it is building its final set of segments to publish. 2) The task transitions to SEGMENT_AVAILABILITY_WAIT after publishing, while waiting for handoff. 3) The task transitions to COMPLETED immediately before exiting, when truly done.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previously, SeekableStreamIndexTaskRunner set ingestion state to COMPLETED when it finished reading data from Kafka. This is incorrect. After the changes in this patch, the transitions go:
The task stays in BUILD_SEGMENTS after it finishes reading from Kafka, while it is building its final set of segments to publish.
The task transitions to SEGMENT_AVAILABILITY_WAIT after publishing, while waiting for handoff.
The task transitions to COMPLETED immediately before exiting, when truly done.