Alert only if certain time passed

[mubarak-j]

I’m trying to avoid receiving false alerts/spam-like notifications when applications' health status change briefly e.g degraded/unknown <-> healthy often within a minute.

I tried to add the following time condition to alert if the app has been in the current state for 5 mins or more but it’s not working, or maybe I’m using the wrong field app.status.operationState.startedAt here?

    trigger.on-health-degraded: |
      - description: Application has degraded
        oncePer: app.status.operationState.syncResult.revision
        send:
        - app-health-degraded
        when: app.status.health.status == 'Degraded' and time.Now().Sub(time.Parse(app.status.operationState.startedAt)).Minutes() >= 5

I'm curious if anyone solved this issue using argocd-notifications?

[mubarak-j]

many folks reported this issue in slack as well https://cloud-native.slack.com/archives/C01UKS2NKK3/p1657184741979379

@pasha-codefresh @RaviHari @alexmt is something you can help with?

[mubarak-j]

Probably related to argoproj/argo-cd#12169

[svghadi]

I believe the newly introduced field app.status.health.lastTransitionTime should help with this use-case. This field stores the timestamp of when last health status was changed.

[Andiesam]

Hey! Has anyone been able to get this to work with app.status.health.lastTransitionTime ? I also believe this is the key to solving the issue, but using it with the other functions doesn’t seem to work for me.
I have tried the following configurations:

trigger.on-health-degraded: |
    - description: Application health has degraded
      when: app.status.health.status in ['Degraded'] and app.metadata.name not in ['BrokenAppIcanIgnore'] and time.Parse(app.status.health.lastTransitionTime).Add(2 * time.Minute).Before(time.Now())
      send:
      - on-health-degraded`

and this at some point

trigger.on-health-degraded: |
        - description: Application health has degraded
          when: app.status.health.status in ['Degraded']  and app.metadata.name not in ['BrokenAppIcanIgnore']  and  time.Now().Sub(time.Parse(app.status.health.lastTransitionTime)).Minutes() >= 2
          send:
          - on-health-degraded

[orYoffe]

Also interested.
Issue with app.status.health.lastTransitionTime is that the notification trigger doesn't validate again if we check with it that time passed IIRC.
And checking whether the app was changed is not great either because it can be affected by other things or other apps as well.

So ideally an example of how one could could set a notification that would be validating on an interval and not only based on changes or you could configure the validation to run a certain amount of time after the last update, like adding validateAfter: 10m or something, unless there's already a way to do this without some prometheus rule.

and if I'm already wishing for things, having the notifications auto resolve would be amazing ❤