Remove zombie ti from its executor #42932
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
boring-cyborg
bot
added
area:Scheduler
uranusjr
force-pushed
the
remove-zombie-from-executor
branch
from
4c7c211 to
acf738c
Compare
The tests were not set up correctly in the first place, likely a side effect from the per-ti-executor configuration changes (AIP-61). The tests are fixed to mock the correct component instead (ExecutorLoader), and checks added to assert ti cleanup in the executor.
uranusjr
force-pushed
the
remove-zombie-from-executor
branch
from
acf738c to
d02e583
Compare
Member
Author
|
Alright, things should be good now. I also identified some test setup issues when trying to add checks for this logic, and fixed them in this PR. See (edited) description above. |
ashb
approved these changes
Oct 15, 2024
dstandish
reviewed
Oct 15, 2024
uranusjr
force-pushed
the
remove-zombie-from-executor
branch
from
443349b to
c60c35e
Compare
uranusjr
added a commit
to astronomer/airflow
that referenced
this pull request
Oct 16, 2024
R7L208
pushed a commit
to R7L208/airflow
that referenced
this pull request
Oct 17, 2024
harjeevanmaan
pushed a commit
to harjeevanmaan/airflow
that referenced
this pull request
Oct 23, 2024
PaulKobow7536
pushed a commit
to PaulKobow7536/airflow
that referenced
this pull request
Oct 24, 2024
2 tasks
This was referenced Nov 5, 2024
ellisms
pushed a commit
to ellisms/airflow
that referenced
this pull request
Nov 13, 2024
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
More details to come. And tests.When a ti worker goes for too long without a heartbeat, the scheduler would identify it as a zombie, and stop it from running any longer. However, the scheduler does not perform housekeeping correctly, and in some cases the executor (of the ti) would still think the ti is still running.
This PR adds an explicit call to the executor to tell it the ti has been terminated and should be cleaned up.
Some tests were modified to check this cleanup is performing as expected. Those tests were actually not set up correctly, likely due to changes from AIP-61, mocking the wrong attribute and failing to actually assert the behaviour. They have been fixed to mock ExecutorLoader instead, which is the canonical executor source after AIP-61 implementation.