Adds: Active time interval

[dubyte]

This is a PR that implements active time interval as in: #2778

The point is to use active_time_interval to express that alertmanager only sent alerts in the interval mentioned.
so in other words active_time_interval mutes in the complement of mute_time_interval

[dubyte]

Example:

  routes:
  - receiver: devs
    mute_time_intervals:
      - offhours
      - holidays
    continue: true
  - receiver: monitoring
    active_time_intervals: 
      - offhours
      - holidays

time_intervals:
- name: offhours
  time_intervals:
  # Monday to Friday 5pm - 9:00 in UTC
  - weekdays: ['monday:friday']
    times:
    - start_time: '00:00'
      end_time: '13:00'
    - start_time: '21:00'
      end_time: '24:00'
  - weekdays: ['saturday', 'sunday']
- name: holidays
  time_intervals:
  - months: ['december']
    days_of_month: ['24']
  - months: ['december']
    days_of_month: ['25:31']
  - months: ['january']
    days_of_month: ['1:2']
  - ...

[simonpasquier]

I agree that it makes sense.

[simonpasquier]

a few nits, otherwise lgtm

[dubyte]

Thank you @simonpasquier

[simonpasquier]

looks good to me (minus 2 last nits). I'd ask @roidelapluie to give it a review to make sure that I haven't missed anything critical. Thanks!

[simonpasquier]

could you also rebase on top of the main branch (it should fix the CI failure with mixins)?

[xkilian]

An issue has been identified related to this PR, @simonpasquier @roidelapluie , during testing with @dubyte

If an active alert is muted, the only effect currently, is to mute notifications.
But the alert is still considered active, which causes duplicate alertes to be created in the alertmanager/karma UI and alertmanager API. This is due to the use of the "continue" keywork and the fact intervals are evaluated at the notification phase. There is no way to distinguish between a real alert and a muted alert.

Suggested paths to correct this:

• Add a new possible state: active, silenced, inhibited, muted
  • change the state of the alert to muted
  • add the muted state to the @State LABEL.
  • add muted to the API filters when querying for alerts
• MAYBE: muted and unmuted alerts that are equivalent should be treated as the same alert for repeat-interval
• alerts should not clear/trigger at each time_interval cross.
• alertmanager UI should permit to filter between active/silenced/inhibited/muted alerts

Or

• add a LABEL to the alert, to indicated this alert is muted (like state=muted or muted=true); This could be a temporary fix while solution 1 is fleshed out.

Or

• consider active_interval and mute_interval in the matching stage and not in the notification processing stage. Though this may lead to clearing+re-firing alerts with no way to deal with them. So I do not think this is the right path.

[dubyte]

Hi
Agreed that there is no way to know if the notification is muted -for certain receivers- or not, so I propose the next: #2832

If all agreed I would like to move the discussion about the api change to that feature request. So we can close this PR.
The tests show that the feature works for mute and active time intervals. So we are not blocked.

Thank you

[simonpasquier]

But the alert is still considered active, which causes duplicate alertes to be created in the alertmanager/karma UI and alertmanager API. This is due to the use of the "continue" keywork and the fact intervals are evaluated at the notification phase. There is no way to distinguish between a real alert and a muted alert.

Correct but the state of an alert (muted/active) is always relative to an alert group. I agree with @dubyte that it should be addressed at the API/UI level.

[simonpasquier]

@benridley would you have time to review this PR by any chance (at least the UX)?

[benridley]

Besides the minor typo above, this looks great. Thanks @dubyte, I think this will be a nice addition to AM.

[dubyte]

Thanks @benridley

[simonpasquier]

sorry @dubyte I had 2 last pending comments that I forgot to submit. Hopefully they are minor and easy to fix :)

[simonpasquier]

👍

[bollmann]

Hi,

I'm trying to use this feature and I'm defining my time intervals and corresponding routes as follows:

    time_intervals:
      - name: workinghours
        time_intervals:
          - times:
              - start_time: 08:00
                end_time: 18:00
            weekdays:
              - monday
              - tuesday
              - wednesday
              - thursday
              - friday
...
    route:
      routes:
        - receiver: oncall-pager
          matchers:
            - severity="workinghours"
          active_time_intervals:
            - workinghours

Furthermore, I have a prometheus rule with label severity="workinghours" defined. With this configuration, I would expect my prometheus rule to only be active during workinghours, i.e., Monday to Friday from 8am until 6pm. However, for some reason my prometheus rule also fires outside of the hours 8am until 6pm. That is, it seems to be active even when it should not be.

Did I do something wrong here in the configuration?

[xkilian]

Here is a working example for you.

It is a best practice to define your alertes with severity levels. We use 0-5, 5 being the most critical.
The group label defines the administrative group responsible for handling the alert.

So based on the group, severity and time of day the alert will be routed to the appropriate receiver configuration. We use distinct receiver configurations as the email template (in our case) need to be different for each case. Ex. Offhours is an email to a specific box. Critical daytime hours will open a ticket automatically with a ticketing app. And severity 3 will be handled with an email regardless of time of day.

What you were doing wrong is not setting the mute_time_intervals. Which is when to NOT send a notification (ex. off-hours), Then the active_time_interval indicates what to actually DO during offhours. This is documented, but may not be clear at first read.

    - matchers:
        - group= system-admins
      receiver: receiver-system-admins
      routes:
        - matchers:
            - severity = 3
          receiver: receiver-system-admins-email
        - matchers:
            - "severity =~ 4|5"
          receiver: receiver-system-admins-critical-daytime
          mute_time_intervals: 
            - offhours
            - holidays
          continue: true
        - matchers:
            - "severity =~ 4|5"
            - env = PR
          receiver: receiver-system-admins-critical-offhours
          active_time_intervals: 
            - offhours
            - holidays       

Hope this helps

[xkilian]

As a bonus, you can see how to do a regex OR statement for the severity (had me stumped for a while) with the new matchers.