Support for Kubernetes Executor side task attempt logging for failed tasks in case of task pods doesn't reach running state

[dirrao]

Right now, when the tasks fail due to pod launch failures or the pod is stuck in the pending phase, then the task logs from the UI are empty. It is very inconvenient for airflow consumers to debug it. They might not have access to the scheduler logs. We can push these failure reasons from the Kubernetes executor to task logs. So, that airflow consumers can able to see task failure reasons from the UI.

closes: #37435

[dirrao]

@jedcunningham / @hussein-awala
Can you review this MR when you are free?

[dirrao]

@romsharon98
Can you review it when you are free?

[dirrao]

@jedcunningham / @hussein-awala
Can you review this MR when you are free?

[dirrao]

@dstandish
With the changes introduced in PR #43183, is it still feasible to generate logs from locations other than the workers? If so, could you please provide a few references to explain this?

[dstandish]

@dstandish With the changes introduced in PR #43183, is it still feasible to generate logs from locations other than the workers? If so, could you please provide a few references to explain this?

Yes, you add a log record. Was done recently in an aws executor

[dstandish]

@dstandish With the changes introduced in PR #43183, is it still feasible to generate logs from locations other than the workers? If so, could you please provide a few references to explain this?

Ok following up with more specifics this morning.

Look at BaseExecutor.log_task_event

executors do not have access to a session, so from executor when we need to send this kind of message, we write the log records to a queue, and the scheduler consumes this queue and writes to db.

cc @potiuk

[potiuk]

Look at BaseExecutor.log_task_event

Nice.

[eladkal]

@dirrao what is the status of this PR?

[dirrao]

@dirrao what is the status of this PR?

I’ve implemented the changes on the Kubernetes executor to align with the new base executor feature. However, I’m uncertain about how to adapt these updates in the Kubernetes watcher code.

[amoghrajesh]

@dirrao why do you need to push anything to the watcher here?
You writing logs to the task logs should be sufficient right? or am missing something

[dstandish]

TaskContextLogger is removed from airflow; this PR needs to be updated to use the log table. The feature was added in this pr #40867 and you can see an example of its usage there.

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.