SAMZA-2423: Heartbeat failure causes incorrect container shutdown #1240
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Symptom: When a container heartbeat fails, the container shutdown
sequence is triggered and the Container is never restarted.
Cause: When a container heartbeat fails, the container shutdown
sequence exists the Container with an exit code of `0` which
marks the container as `Completed` - preventing the JobCoordinator
from restarting the container.
Changes: The container can shutdown exceptionally in the following two ways:
1) Exception in the container
2) Heartbeat Expired
In both paths the ContainerLaunchUtil previously expected a
shared static variable to hold the exception. The change introduced
gets rid of the static variable and checks each path explicitly
and exits with code `1` in both cases.
Tests:
Contributor
|
Seems like the bug is we end up overwriting the Is that correct? |
Contributor
Author
|
@mynameborat Correct ! I updated the description to include this. |
Contributor
|
can we add a unit test for this? |
mynameborat
approved these changes
Dec 20, 2019
Contributor
mynameborat
left a comment
mynameborat
left a comment
There was a problem hiding this comment.
Looks like the class isn't easily unit testable. Let us get this merged.
Would be nice to follow up with a unit test for the entire util.
Contributor
Author
|
I agree, looks like there is no way to mock objects that |
rmatharu-zz
pushed a commit
to rmatharu-zz/samza
that referenced
this pull request
Jan 21, 2020
…ache#1240) Populate container exception from the listener only if it is null
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Symptom:
When a container heartbeat fails, the container shutdown
sequence is triggered and the Container is never restarted.
Cause:
When a container heartbeat fails, the container shutdown
sequence exists the Container with an exit code of
0whichmarks the container as
Completed- preventing the JobCoordinatorfrom restarting the container.
The bug is caused by
containerExceptionoverwritten with the valuereturned by
listener.getContainerExceptionwithout checking ifcontainerExceptionwas already set by the heartbeat monitorChanges:
Check the container exception if its been populated before populating it with the exception from listener.