Skip to content

Enhance SparkKubernetesOperator #29977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
potiuk merged 1 commit into from
Mar 14, 2023
Merged

Enhance SparkKubernetesOperator #29977

potiuk merged 1 commit into from
Mar 14, 2023

Conversation

@blcksrx
Copy link
Contributor

@blcksrx blcksrx commented Mar 8, 2023

Enhancement on SparkKubernetesOperator:

  • log the spark driver output to the airflow
  • provide on_kill method
  • re-write SparkKubernetesOperator tests in pytest
  • generalise and simplify create_custom_object and get_custom_object on KubernetesHook:
    • they are not only about SparkApplication
    • (In my opinion) deleting previous object is out of scope this method and airflow

closes: #21325

@blcksrx blcksrx requested a review from jedcunningham as a code owner March 8, 2023 13:58
@boring-cyborg boring-cyborg bot added provider:cncf-kubernetes Kubernetes (k8s) provider related issues area:providers labels Mar 8, 2023
@blcksrx blcksrx force-pushed the spark_k8s_operator branch 6 times, most recently from 994fb03 to af1afd4 Compare March 9, 2023 15:03
@blcksrx blcksrx force-pushed the spark_k8s_operator branch from af1afd4 to 78450c7 Compare March 10, 2023 08:50
@eladkal eladkal changed the title enhance spark_k8s_operator Enhance SparkKubernetesOperator Mar 10, 2023
@potiuk
Copy link
Member

potiuk commented Mar 14, 2023

Looks cool.

@potiuk potiuk merged commit 9a4f674 into apache:main Mar 14, 2023
@eladkal eladkal mentioned this pull request Apr 2, 2023
Closed
64 tasks
@hanna-liashchuk hanna-liashchuk mentioned this pull request May 10, 2023
Closed
2 tasks
jamescnowell added a commit to jamescnowell/airflow that referenced this pull request Jun 5, 2023
@jamescnowell
Revert "enhance spark_k8s_operator (apache#29977)"
f2e650c 
This reverts commit 9a4f674.

Based on the discussion found
[here](apache#31183), previous
changes
to the Spark K8s Operator broke existing functionality and did not
update the documentation for the newly enabled functionality.

The Spark Sensor no longer works, XCOM no longer works on the Operator
itself, and the Operator does not fail when the Spark job fails.

Rather than attempt to fix or resolve the current implementation,
I am reverting to the existing, documented implementation.

I would propose creating a _new_ Operator with alternative
functionality (one which does not need a Sensor, copies logs, etc.)
if that is desired.
@assaf-xm assaf-xm mentioned this pull request Jul 5, 2023
Open
2 tasks
@Taragolis Taragolis mentioned this pull request Oct 17, 2023
Closed
2 tasks
@romsharon98 romsharon98 mentioned this pull request Nov 25, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@potiuk potiuk potiuk approved these changes

@jedcunningham jedcunningham Awaiting requested review from jedcunningham

Assignees

No one assigned

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

on_kill method for SparkKubernetesOperator

2 participants

@blcksrx @potiuk