Fix task retries executing wrong method after deferred state #56731
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When a deferred task resumed and failed in its completion callback, retries would incorrectly re-execute the callback instead of restarting from the beginning. This caused retry attempts to fail repeatedly at the same step rather than attempting the full task execution from scratch. The fix ensures next_method and next_kwargs fields are properly cleared when tasks transition to retry, failed, or success states, preventing stale deferral parameters from persisting across task attempts. Fixes apache#56704
kaxil
added
the
backport-to-v3-1-test
amoghrajesh
approved these changes
Oct 16, 2025
tirkarthi
approved these changes
Oct 16, 2025
potiuk
reviewed
Oct 16, 2025
potiuk
approved these changes
Oct 16, 2025
Member
Author
|
the k8s failure will be fixed by #56733 |
github-actions bot
pushed a commit
that referenced
this pull request
Oct 16, 2025
…ate (#56731) When a deferred task resumed and failed in its completion callback, retries would incorrectly re-execute the callback instead of restarting from the beginning. This caused retry attempts to fail repeatedly at the same step rather than attempting the full task execution from scratch. The fix ensures next_method and next_kwargs fields are properly cleared when tasks transition to retry, failed, or success states, preventing stale deferral parameters from persisting across task attempts. Fixes #56704 (cherry picked from commit 16e9b35) Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
Backport successfully created: v3-1-test
|
kaxil
pushed a commit
that referenced
this pull request
Oct 16, 2025
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 17, 2025
…56731) When a deferred task resumed and failed in its completion callback, retries would incorrectly re-execute the callback instead of restarting from the beginning. This caused retry attempts to fail repeatedly at the same step rather than attempting the full task execution from scratch. The fix ensures next_method and next_kwargs fields are properly cleared when tasks transition to retry, failed, or success states, preventing stale deferral parameters from persisting across task attempts. Fixes apache#56704
abdulrahman305 bot
pushed a commit
to abdulrahman305/airflow
that referenced
this pull request
Oct 19, 2025
…56731) When a deferred task resumed and failed in its completion callback, retries would incorrectly re-execute the callback instead of restarting from the beginning. This caused retry attempts to fail repeatedly at the same step rather than attempting the full task execution from scratch. The fix ensures next_method and next_kwargs fields are properly cleared when tasks transition to retry, failed, or success states, preventing stale deferral parameters from persisting across task attempts. Fixes apache#56704
TyrellHaywood
pushed a commit
to TyrellHaywood/airflow
that referenced
this pull request
Oct 22, 2025
…56731) When a deferred task resumed and failed in its completion callback, retries would incorrectly re-execute the callback instead of restarting from the beginning. This caused retry attempts to fail repeatedly at the same step rather than attempting the full task execution from scratch. The fix ensures next_method and next_kwargs fields are properly cleared when tasks transition to retry, failed, or success states, preventing stale deferral parameters from persisting across task attempts. Fixes apache#56704
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When a deferred task resumed and failed in its completion callback, retries would incorrectly re-execute the callback instead of restarting from the beginning. This caused retry attempts to fail repeatedly at the same step rather than attempting the full task execution from scratch.
The fix ensures next_method and next_kwargs fields are properly cleared when tasks transition to retry, failed, or success states, preventing stale deferral parameters from persisting across task attempts.
Fixes #56704
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.