Add time to live for finished jobs in Kubernetes

[potiuk]

When you upgrade/change job in K8S that has been finished and not manually removed, this leads to "Field is immutable" error.

This is a known kubernetes issue:

kubernetes/kubernetes#89657

And there are some workarounds (manually removing the job for example), but the only good solution is possible only in K8S 1.23+ with ttlSecondsAfterFinished set for the job, so that K8S can auto
clean it.

This PR adds it conditionally for K8S >= 1.23

Fixes: #27561

---

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

Hey @dstandish @jedcunningham -> WDYT?

[hussein-awala]

It's an interesting change, but there's still a chance of getting the same error if for some reason we run the jobs twice before the TTL ends, on top of that we can add {{ .Release.Revision }} as suffix for job name -> metadata.name: {{ .Release.Name }}-<job name>-{{ .Release.Revision }}, and keep the cleanup task for garbage collection.

[potiuk]

It's an interesting change, but there's still a chance of getting the same error if for some reason we run the jobs twice before the TTL ends, on top of that we can add {{ .Release.Revision }} as suffix for job name -> metadata.name: {{ .Release.Name }}-<job name>-{{ .Release.Revision }}, and keep the cleanup task for garbage collection.

Thought about it, but I am not sure if there is another reason why we wanted to keep the job names "static" - @jedcunningham @dstandish ?

[potiuk]

Closing in favour of #29314