SparkKubernetesOperator fails after upgrade from 2.8.1 to 2.8.2

[gbloisi-openaire]

Apache Airflow version

2.8.2

If "Other Airflow 2 version" selected, which one?

2.8.3rc1

What happened?

I'm running a spark-pi example using the SparkKubernetesOperator:

    task_id='spark_pi_submit',
    namespace='lot1-spark-jobs',
    application_file="/example_spark_kubernetes_operator_pi.yaml",
    kubernetes_conn_id="kubernetes_default",
    do_xcom_push=True,
    in_cluster=True,
    delete_on_termination=True,
    dag=dag
)

It was running fine on 2.8.1. After upgrading to airflow 2.8.2 I got the following error:

│     kube_client=self.client,                                                                                                                                                              │
│                 ^^^^^^^^^^^                                                                                                                                                               │
│   File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__                                                                                                                    │
│     val = self.func(instance)                                                                                                                                                             │
│           ^^^^^^^^^^^^^^^^^^^                                                                                                                                                             │
│   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 250, in client                                           │
│     return self.hook.core_v1_client                                                                                                                                                       │
│            ^^^^^^^^^                                                                                                                                                                      │
│   File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__                                                                                                                    │
│     val = self.func(instance)                                                                                                                                                             │
│           ^^^^^^^^^^^^^^^^^^^                                                                                                                                                             │
│   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 242, in hook                                             │
│     or self.template_body.get("kubernetes", {}).get("kube_config_file", None),                                                                                                            │
│        ^^^^^^^^^^^^^^^^^^                                                                                                                                                                 │
│   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 198, in template_body                                    │
│     return self.manage_template_specs()                                                                                                                                                   │
│            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                                                   │
│   File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py", line 127, in manage_template_specs                            │
│     template_body = _load_body_to_dict(open(self.application_file))                                                                                                                       │
│                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                                                        │
│ FileNotFoundError: [Errno 2] No such file or directory: 'apiVersion: "sparkoperator.k8s.io/v1beta2"\nkind: SparkApplication\nmetadata:\n  name: spark-pi\n  namespace: lot1-spark-jobs\ns │
│ [2024-03-10T10:29:15.613+0000] {taskinstance.py:1149} INFO - Marking task as UP_FOR_RETRY. dag_id=spark_pi, task_id=spark_pi_submit, execution_date=20240310T102910, start_date=20240310T │

It looks like self.application_file eventually contains the content of the file it point to.

I suspect it was caused by changes introduced by PR-22253. I'm quite new to Airflow and Python but my guess is that "application_file" property hasn't to be managed as a template_property since template representations where moved to template_body.

What you think should happen instead?

No response

How to reproduce

Given my understanding of the issue, a very simple example of SparkKubernetesOperator using application_file property should reproduce this issue.

Operating System

kind kubernetes

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

cc: @hamedhsn ? Can you please take a look? It seems related to your change

As a workaround @gbloisi-openaire -> you can easily downgrade cncf.kubernetes provider to previous version and continue using Airlfow 2.8.2. In Airflow Providers can be upgraded and downloaded separately from airflow.

If you are using container image - you can build your own image and downgrade providers there: https://airflow.apache.org/docs/docker-stack/build.html#example-of-customizing-airflow-provider-packages

[hamedhsn]

@gbloisi-openaire can you pls post the content of your example_spark_kubernetes_operator_pi.yaml file
I assume the file exists at /example_spark_kubernetes_operator_pi.yaml

[gbloisi-openaire]

Plese find it below. It is an adaption from tests/system/providers/cncf/kubernetes/example_spark_kubernetes_spark_pi.yaml. I removed license's comments after a few tries to have confirmation that the error was reporting the content of this file.

Also my DAG is an adaption from the provided tests/system/providers/cncf/kubernetes/example_spark_kubernetes.py.
So far I got it working successfully by rewriting the file content as a template_spec object and by removing the do_xcom_push=True and the SparkKubernetesSensor task.

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: lot1-spark-jobs
spec:
  type: Scala
  mode: cluster
  image: "apache/spark:v3.1.3"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///opt/spark/examples/jars/spark-examples_2.12-3.1.3.jar"
  sparkVersion: "3.1.3"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/tmp"
        type: Directory
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.1.3
    serviceAccount: spark
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 3.1.3
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"

[Taragolis]

This cause by multiple changes and some of them mutable exclusive or conflicts to the Airflow Core

1. Expects that application_file is always is file
2. application_file is templated_field, which could be both values or template file. If it a file it return content of the file, rather than path.
3. Move operation around templated fields into the execute method: Fix rendering SparkKubernetesOperator.template_body #37271 which fix SparkKubernetesOperator not rendering template correctly #37261

If value expected non-templated file I would recommend to use LiteralValue (see: #35017, this not documented well) wrapper instead

[renanxx1]

I had the same issue described in the post after upgrading from 2.8.1 to 2.8.2. Will it be fixed in 2.8.3 version ?

[potiuk]

I had the same issue described in the post after upgrading from 2.8.1 to 2.8.2. Will it be fixed in 2.8.3 version ?

Well. The merge happened 6 hours ago and 2.8.3 has been released yesterday - so - obviously - no.

But this is a provider fix - and you can both - downgrade provider to older version that had no problems and upgrade the provider when it gets released after the fix. So you do not have to wait - you can downgrade the provider now - and then when we release an RC candidate for new providers - in a few days you can take RC for a spin and test it.

Can we count on your help testing it when RC is out @renanxx1 ?