KubernetesResourceOperator - KubernetesDeleteResourceOperator & KubernetesCreateResourceOperator

[raphaelauv]

Add an abstract KubernetesResourceOperator

and two concrete operators : KubernetesDeleteResourceOperator and KubernetesCreateResourceOperator

missing documentation and tests

example of use

dag = DAG(
    dag_id="resource_dag",
    schedule_interval="0 0 * * *",
    start_date=today("UTC").add(days=-1)
)

pvc_conf = """
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {pvc_name}
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: 5Gi
"""
pvc_conf_all = """
apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: toto
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    name: tata
"""

with dag:
    a = KubernetesCreateResourceOperator(
        task_id="create_pvc_1",
        yaml_conf=pvc_conf.format(pvc_name="toto"),
    )
    b = KubernetesCreateResourceOperator(
        task_id="create_pvc_2",
        yaml_conf=pvc_conf.format(pvc_name="tata"),
    )

    c = KubernetesDeleteResourceOperator(
        task_id="delete_pvc",
        yaml_conf=pvc_conf_all,
    )

    a >> b >> c

[jedcunningham]

I'm hesitant to build create/deletes for every single resource type... Especially if you are going to ask them to give you it all via yaml anyways.

Check out create_from_yaml instead. We could probably use this to build a generic create/delete?

[raphaelauv]

missing delete_from_yaml : kubernetes-client/python#940

[raphaelauv]

inspired from tomplus/kubernetes_asyncio#239

[potiuk]

errors to fix :(

[raphaelauv]

hey @potiuk I think I fixed the issues related to the PR , could you have a second look , thanks 👍

[raphaelauv]

@jedcunningham thanks for your first review , could you have a second look to the PR , thanks

[potiuk]

That one looks good to me. @jedcunningham ?

[raphaelauv]

just rebased on main branch

[potiuk]

Tests failing :(

[raphaelauv]

flaky test related to #30831

[potiuk]

LGTM - @jedcunningham ?

[raphaelauv]

@eladkal thank again for your review , I adressed all your remarks , could you please re-review , thanks

[raphaelauv]

@jedcunningham could you review again , thank you

[hussein-awala]

Looks good! LGTM

[eladkal]

@jedcunningham any further comments?

[raphaelauv]

@potiuk could we merge ? Thanks