[AIRFLOW-3412] Fix kubernetes executor to delete pods after termination

[float34]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-3412
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

• Here are some details about my PR, including screenshots of any UI changes:
  • We need to pass 'try_number' parameter during the pod creation in kubernetes_executor.py, otherwise '_labels_to_key' function will always raise KeyError, making self.result_queue empty.
  • As a result '_change_state' function in kubernetes_executor.py will never be called and pods will keep hanging after termination even if other pod deletion conditions are met.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  • Not sure how to write one for this particular case, maybe someone will help?

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes flake8

[skudriashev]

Please, keep trailing comma.

[skudriashev]

Ditto.

[tiagovrtr]

Any idea on what could have stalled test_volume_mount? I am getting flooded with stale pods because of this issue

[dmateusp]

I deployed v-1-10-stable in my Kubernetes cluster and reproduced the same issue, this PR solved it on my side as well

[kyle-hamlin]

Can we get this merged? This is pretty critical for Airflow to work properly in Kubernetes.

[singhania]

I am facing one issue where KubernetesExecutor is not launching pods as soon as reaches parallelism even though pods are completed.

[dmateusp]

I think we're observing the same issue @singhania I'll try to reproduce tomorrow morning

[singhania]

One more observation, If i don't set parallelism then also after some 30 completed pods it stops creating new pods.
I have opened an issue on Airflow jira board but no idea about the priority.
https://issues.apache.org/jira/browse/AIRFLOW-3652

After debugging found that key is not getting created properly and hence running task are not getting popped after state change.

[ckljohn]

I found a similar PR #4163. Are they fixing the same thing?

[feng-tao]

PTAL @dimberman

[singhania]

I have created a PR #4471 which along with this PR will solve the issue of pods not being created (https://issues.apache.org/jira/browse/AIRFLOW-3652).
Would really like to get it reviewed as we are blocked by this issue.

[dmateusp]

hi there, wanted to let you know I've cherry picked this PR and I have been running it in prod for 2 weeks in here https://github.com/dmateusp/incubator-airflow/tree/v1-10-stable-AIRFLOW-3022-AIRFLOW-3412-AIRFLOW-3652

[spinus]

Looks like another PR was merged with similar function (https://github.com/apache/airflow/pull/4163/files), probably that can be closed, correct?

[float34]

@spinus Yes, I'll close that now as irrelevant.