Fix logic to cancel the external job if the TaskInstance is not in a running or deferred state for BigQueryInsertJobOperator

[sunank200]

PR #38912 introduces a method for handling asyncio.CancelledError in a try/except block. However, this method is deemed unsafe, and it affects BigQueryInsertJobOperator operators, which enables external job cancellation if the triggerer restarts or crashes. This can cause weird behaviour like rescheduling deferred operators, as Airflow remains unaware of job cancellations.

As a workaround, capturing asyncio.CancelledError cancels the job only if the TaskInstance is not in a running or deferred state. This prevents premature external job termination.

More details at: #36090 (comment)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[sunank200]

@thesuperzapper please take a look

[Lee-W]

Just have a quick discussion with @sunank200. We'll yield the event if CancelError was raised , and handle the TaskInstance state check and cancelation in execute_complete

[sunank200]

Just have a quick discussion with @sunank200. We'll yield the event if CancelError was raised , and handle the TaskInstance state check and cancelation in execute_complete

This approach would not work as execute on complete is not called when the task is manually cancelled when inside triggerer

[pankajkoti]

LGTM. Some minor suggestions inline

[sunank200]

@thesuperzapper Please take a look I have tested this and it works fine.