KPO - deferrable - Invalid kube-config file. Expected key contexts in kube-config

[raphaelauv]

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

5.2.0

Apache Airflow version

2.5.1

Operating System

linux 5.15.0-60-generic - Ubuntu 22.04

Deployment

Docker-Compose

Deployment details

No response

What happened

the exact same task in sync mode ( normal ) work but in async (deferable) it fail

airflow connection - kubernetes_default ->

{
    "conn_type": "kubernetes",
    "extra": "{\"extra__kubernetes__in_cluster\": false, \"extra__kubernetes__kube_config_path\": \"/opt/airflow/include/.kube/config\", \"extra__kubernetes__namespace\": \"default\", \"extra__kubernetes__cluster_context\": \"kind-kind\", \"extra__kubernetes__disable_verify_ssl\": false, \"extra__kubernetes__disable_tcp_keepalive\": false}"
}

from pendulum import today
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator

dag = DAG(
    dag_id="kubernetes_dag",
    schedule_interval="0 0 * * *",
    start_date=today("UTC").add(days=-1)
)

with dag:
    cmd = "echo toto && sleep 10 && echo finish"

    KubernetesPodOperator(
        task_id="task-a",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        name="airflow-test-pod",
        image="alpine:3.16.2",
        cmds=["sh", "-c", cmd],
        is_delete_operator_pod=True,
        deferrable=True,
        get_logs=True,
    )
    KubernetesPodOperator(
        task_id="task-B",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        name="airflow-test-pod",
        image="alpine:3.16.2",
        cmds=["sh", "-c", cmd],
        is_delete_operator_pod=True,
        get_logs=True,
    )

[2023-02-12, 09:53:24 UTC] {taskinstance.py:1768} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 611, in execute_complete
    raise AirflowException(event["message"])
airflow.exceptions.AirflowException: Invalid kube-config file. Expected key contexts in kube-config

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 630, in execute_complete
    self.post_complete_action(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 654, in post_complete_action
    self.cleanup(
  File "/home/airflow/.local/lib/python3.10/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 673, in cleanup
    raise AirflowException(
airflow.exceptions.AirflowException: Pod airflow-test-pod-vw8fxf25 returned a failure:

What you think should happen instead

deferrable KPO should work same as KPO and not fail

How to reproduce

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[raphaelauv]

maybe same problem than this one -> astronomer/astronomer-providers#745

[hussein-awala]

This exception is raised by the kubernetes client, the version used in your Airflow worker may be not compatible with the kubectl version which you used to generate the config file, can you check what are the used version for kubernetes client and kubectl? can you add the config file to check why it's not parsed by the client?

[raphaelauv]

if I set the parameter config_file the deferable task then work

    KubernetesPodOperator(
        task_id="task-a",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        config_file="/opt/airflow/include/.kube/config",
        deferrable=True,
        ...

hello @hussein-awala, the config file is not the problem , it's the deferrable code not fully using the kubernetes_conn_id , (exactly like the issue I shared in my previous message)

[hussein-awala]

Indeed, I think there is a missing check for this path when you provide it in connection, I will try to add a unit test and maybe refacto my PR, can you please test it on your dag if this is possible?