Skip to content

spark kubernetes operator arguments description reordering #41372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eladkal merged 3 commits into from
Aug 12, 2024
Merged

spark kubernetes operator arguments description reordering #41372

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
2d566f3
spark kubernetes operator arguments description reordering
dirrao Aug 10, 2024
28d9461
spark kubernetes operator arguments description reordering
dirrao Aug 11, 2024
7dc7c74
Merge branch 'main' into spark_operator_documentation
dirrao Aug 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 9 additions & 7 deletions airflow/providers/cncf/kubernetes/operators/spark_kubernetes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,24 +48,26 @@ class SparkKubernetesOperator(KubernetesPodOperator):
For more detail about Spark Application Object have a look at the reference:
https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/v1beta2-1.3.3-3.1.1/docs/api-docs.md#sparkapplication

:param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
:param kubernetes_conn_id: the connection to Kubernetes cluster
:param image: Docker image you wish to launch. Defaults to hub.docker.com,
:param code_path: path to the spark code in image,
:param namespace: kubernetes namespace to put sparkApplication
:param cluster_context: context of the cluster
:param application_file: yaml file if passed
:param name: name of the pod in which the task will run, will be used (plus a random
suffix if random_name_suffix is True) to generate a pod id (DNS-1123 subdomain,
containing only [a-z0-9.-]).
:param application_file: filepath to kubernetes custom_resource_definition of sparkApplication
:param template_spec: kubernetes sparkApplication specification
:param get_logs: get the stdout of the container as logs of the tasks.
:param do_xcom_push: If True, the content of the file
/airflow/xcom/return.json in the container will also be pushed to an
XCom when the container completes.
:param success_run_history_limit: Number of past successful runs of the application to keep.
:param delete_on_termination: What to do when the pod reaches its final
state, or the execution is interrupted. If True (default), delete the
pod; if False, leave the pod.
:param startup_timeout_seconds: timeout in seconds to startup the pod.
:param log_events_on_failure: Log the pod's events if a failure occurs
:param reattach_on_restart: if the scheduler dies while the pod is running, reattach and monitor
:param delete_on_termination: What to do when the pod reaches its final
state, or the execution is interrupted. If True (default), delete the
pod; if False, leave the pod.
:param kubernetes_conn_id: the connection to Kubernetes cluster
"""

template_fields = ["application_file", "namespace", "template_spec"]
Expand Down