Ensure teardown tasks are executed when DAG run is set to failed

[jscheffl]

Related to Slack topic: https://apache-airflow.slack.com/archives/CCR6P6JRL/p1736440079894049

We noticed that if a DAG run is set to failed, all tasks are either set to failed or skipped. But if Teardown Tasks are used in a DAG, they are not executed. This could lead to infrastructure or external dependencies not properly cleaned-up.

This PR changes the behavior and does NOT fail/skip teardown tasks if a DAG is set to failed.
A side effect as consequence might be that the DAG is after the call NOT failed, else if it would set it to failed, then any teardown task (even if not skipped/failed) will not scheduled anymore.

[shahar1]

Nice catch!
One edge case that I thought about though* - what if the corresponding setup task hasn't finished running yet? (if such exists ofc)
For example:

If you set the DAG run to failed before the cluster was created, the delete_cluster should be skipped.

* Maybe the current architecture already handles, but it's worth checking before merging this PR.

[jscheffl]

Nice catch! One edge case that I thought about though* - what if the corresponding setup task hasn't finished running yet? (if such exists ofc) For example: If you set the DAG run to failed before the cluster was created, the delete_cluster should be skipped.

• Maybe the current architecture already handles, but it's worth checking before merging this PR.

Oh, yeah. Tested this explicitly. And actually the Scheduler had this feature before (added this to docs as well, was also new to me): If the setup task is skipped (which is the state when you mark as failed and it was not running) then the teardown is also skipped,

[jscheffl]

Added some docs as I thought about that slightly the behavior changes - to ensure it is properly documented.

[github-actions]

Backport successfully created: v2-10-test

Status	Branch	Result
✅	v2-10-test

[Lee-W]

just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?

[potiuk]

just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?

I'd say it's a bugfix. Not all "change behaviour" is "breaking change". Every bugfix is a "behavioural change" in essence, the important thing is what is the feature intention and whether the change is "fixing" things or "changing intention on how things should work". SemVer is all about intentions and not about whether behaviour changes or not - see https://semver.org/

In this case it seems reasonable to assume that this behaviour was intention of the "teardown" behaviour - so technically it's a bugfix (and it's been already backported to 2.10 it seems.

It might require a newsfragment, but IMHO - that's more than enough.

[jscheffl]

just notice this could potentially be a breaking change, but doesn't look like something we need a migration rule 🤔 would like to confirm whether we might want to have such a rule?

For sure it does not need migration rules. No code change needed.

Yeah and thought also a bit about if it is breaking, asked in #random / Slack for feedback and nobody except @dstandish was responding. Then after passing... I think it is really in the intend for why we have Setup/Teardown. Behaviour change is "just" because of the side effect that the DAG is not immediately failed if you set it to failed as Teardown need to be scheduled.

[rakesh-shah]

So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one.
thanks

[jscheffl]

So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one. thanks

This was fixed in 2.10.5 - See milestone flag on the right side. Also Release Notes help in this

[rakesh-shah]

So @jscheffl we are using 2.7.2 and we can see an issue where sometime when a task fails I am guessing it because of Task Kill might be due to Pod crash does not call failure callback could you please help me on this if this issue was also fixed or should I raise one. thanks

This was fixed in 2.10.5 - See milestone flag on the right side. Also Release Notes help in this

Yeah thanks for that @jscheffl I saw the release notes and it says it will work for teardown when the state is changed manually to failed

But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?

[jscheffl]

But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?

This sounds like a different use case. But you can test it and if you think it is a bug then please open a new issue for this as it seems to be unrelated. But please test with 2.11.0 or most recent 3.0

[rakesh-shah]

But @jscheffl I was thinking if it also covers when the issue appears due to Kill event during a pod crash which generates a SIGTERM ?

This sounds like a different use case. But you can test it and if you think it is a bug then please open a new issue for this as it seems to be unrelated. But please test with 2.11.0 or most recent 3.0

Sure @jscheffl sounds good will test it and revert