Improve annotations customization in the chart

[FloChehab]

Description

As discussed in #13616 (comment), we would need to improve the customization of annotations across all objects of the chart.

Use case / motivation

We often need to paremetrize the annotations and we would rather do it in one PR with all the objects rather than one small PR everytime.

Are you willing to submit a PR?

@mik-laj I am ready to be assigned.

Related Issues
Not to my knowledge.

---

What need to be tackled before starting dev:

• On what objects do we want annotations?
• What terminology do we choose in the values.yaml? (annotations or extraAnnotations or both or something else?)
• Have a table with the mapping of each object and the value path for the annotation parametrization?
• Should we drop the value airflowPodAnnotations ?

[FloChehab]

First:

Objects that should have parametrizable annotations:

• Pod (inside deployment and StatefulSet),
• Service,
• ServiceAccount,
• Ingress.

Is this list too much ? Am I missing something?

[FloChehab]

(I have added more info in the issue description, feel free to edit)

[mik-laj]

@dimberman @XD-DENG @potiuk Do you have any thoughts on this? I saw that these changes often come back in different forms, and I think we can do it once, but good.

[XD-DENG]

Absolutely a good idea 👍

[potiuk]

Absolutely !

[mik-laj]

We have a ticket that improves the annotation handling for SA a bit. Anyone willing to do a review and say if this change is sufficient for this ticket as well?
#11769

[FloChehab]

Hello,

Regarding the terminology

What terminology do we choose in the values.yaml? (annotations or extraAnnotations or both or something else?)

I checked the chart and extraAnnotations is only used when there are other annotations related to prometheus that are already present (for the pgbouncer-service & the statsd-service). My position on this would be to remove such annotations related to prometheus and use the annotations terminology everywhere.

[FloChehab]

Hello, just wanted to let you now that I should have time to work on this issue in couple of weeks.

[dimberman]

@FloChehab awesome! Please let us know when it's ready to review :)