[Bug]: ‘alert -P’ gets an Unknown if everything is fine

[wattebausch]

Please try to fill out as much of the information below as you can. Thank you!

• Yes, I've searched similar issues on GitHub and didn't find any.

Which version contains the bug?

0.2.0 & development

Describe the bug

Hello there

We use check_prometheus in the ‘alert -P’ method. We only want to output errors. If we run without ‘-P’ the SMS/email will be too long with over 100 checks when an error occurs.

We have compared version ‘v0.2.0’ and ‘development’. We noticed that the versions behave differently with ‘alert’. However, both are the same, if our Prometheus is all set to ‘Inactive’, i.e. OK, then we get an ‘UNKNOWN’ back

Can you check if this is the same for you?
Thanks for your great work

How to recreate the bug?

Version: 0.2.0 (Check against a cluster without alerts)

./check_prometheus_v0.2.0_Linux_x86_64 -H metrics.clean-cluster.example.com -p 80 alert 
[OK] - Alerts inactive | total=123 firing=0 pending=0 inactive=123

./check_prometheus_v0.2.0_Linux_x86_64 -H metrics.clean-cluster.example.com -p 80 alert -P
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive
 | 

# echo $?
3

Version: 0.2.0 (check against a cluster with problem)

./check_prometheus_v0.2.0_Linux_x86_64 -H metrics.some-probelm-cluster.example.com -p 80 alert 
[CRITICAL] - 126 Alerts: 1 Firing - 0 Pending - 125 Inactive
 \_[OK] [FluxHelmReleaseNotReady] is inactive
 \_[OK] [FluxGitRepositorySyncFailed] is inactive
...
 \_[CRITICAL] [Watchdog] is firing - value: 1.00
 \_[OK] [InfoInhibitor] is inactive
..
 \_[OK] [K3sCertificateExpiration] is inactive
 | total=126 firing=1 pending=0 inactive=125

./check_prometheus_v0.2.0_Linux_x86_64 -H metrics.some-probelm-cluster.example.com -p 80 alert -P
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
 \_[CRITICAL] [Watchdog] is firing - value: 1.00
 | 

Version: development (Check against a cluster without alerts)

./check_prometheus_develop -H metrics.clean-cluster.example.com -p 80 alert 
[OK] - 125 Alerts: 0 Firing - 0 Pending - 125 Inactive
\_ [OK] [FluxHelmReleaseNotReady] is inactive
\_ [OK] [FluxGitRepositorySyncFailed] is inactive
\_ [OK] [InfoInhibitor] is inactive
..
\_ [OK] [K3sCertificateExpiration] is inactive
|total=125 firing=0 pending=0 inactive=125

./check_prometheus_develop -H metrics.clean-cluster.example.com -p 80 alert -P
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

# echo $?
3

Version: development (check against a cluster with problem)

./check_prometheus_develop -H metrics.some-probelm-cluster.example.com  -p 80 alert 
[CRITICAL] - 126 Alerts: 1 Firing - 0 Pending - 125 Inactive
\_ [OK] [FluxHelmReleaseNotReady] is inactive
\_ [OK] [FluxGitRepositorySyncFailed] is inactive
..
\_ [CRITICAL] [Watchdog] is firing - value: 1.00
\_ [OK] [InfoInhibitor] is inactive
..
\_ [OK] [K3sCertificateExpiration] is inactive
|total=126 firing=1 pending=0 inactive=125

./check_prometheus_develop -H metrics.some-probelm-cluster.example.com  -p 80 alert -P
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00

[RincewindsHat]

Hi @wattebausch,
To summarize: You would like to only see Problems in the Output and not alerts which are OK?
Did I read that correctly?

[wattebausch]

Yes.
if I have understood it correctly, alert -P should be. But as I see it, the exit code is 3 (Unknown) and not 0 (OK)
-P, --problems Display only alerts which status is not inactive/OK

./check_prometheus_develop -H metrics.clean-cluster.example.com -p 80 alert -P
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

# echo $?
3

./check_prometheus_v0.2.0_Linux_x86_64 -H metrics.clean-cluster.example.com -p 80 alert -P
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive
 | 

# echo $?
3

[martialblog]

Hi, right now that is just the logic of the plugin.

I think the reasoning behind it was, if there's nothing there, we cannot determine the status. But I agree that it is somewhat odd behavior. If there are no alert rule defined it would also return UNKNOWN.

We can improve that, just gotta keep all possible scenarios in mind for example:

	All Alerts	--problems	--name
No Alerts in API	OK	OK	Unknown, since we cannot determine the status of the wanted alert
Alerts in API	Worst State	Worst State	Unknown, since we cannot determine the status of the wanted alert

What does everybody think?

[martialblog]

@wattebausch Thanks again for your feedback. It's very helpful!

[RincewindsHat]

Hi, right now that is just the logic of the plugin.

I think the reasoning behind it was, if there's nothing there, we cannot determine the status. But I agree that it is somewhat odd behavior. If there are no alert rule defined it would also return UNKNOWN.

We can improve that, just gotta keep all possible scenarios in mind for example:
All Alerts --problems --name
No Alerts in API OK OK Unknown, since we cannot determine the status of the wanted alert
Alerts in API Worst State Worst State Unknown, since we cannot determine the status of the wanted alert

What does everybody think?

Without knowing much about prometheus this sounds about right, when checking for alerts it is probably OK if there are none.

[martialblog]

@wattebausch I reworked the logic a bit, you can check out the new code here #66

This should be more expected and flexible behavior

[wattebausch]

@wattebausch I reworked the logic a bit, you can check out the new code here #66

This should be more expected and flexible behavior

Check my Version

git checkout
Your branch is up to date with 'origin/no-alerts-logic'.

./check_prometheus --version
check_prometheus version development
commit: HEAD
date: latest

i don't know whether the first four lines should appear in the help. FYI

./check_prometheus alert --help
Checks the status of a Prometheus alert and evaluates the status of the alert:
firing = 2
pending = 1
inactive = 0

Usage:
  check_prometheus alert [flags]

Examples:
...
...
Flags:
  -h, --help                     help for alert
  -n, --name strings             The name of one or more specific alerts to check.
                                 This parameter can be repeated e.G.: '--name alert1 --name alert2'
                                 If no name is given, all alerts will be evaluated
  -T, --no-alerts-state string   State to assign when no alerts are found (0, 1, 2, 3, OK, WARNING, CRITICAL, UNKNOWN). If not set this defaults to OK (default "OK")
  -P, --problems                 Display only alerts which status is not inactive/OK. Note that in combination with the --name flag this might result in no alerts being displayed

Global Flags:
  -b, --bearer string      Specify the Bearer Token for server authentication (CHECK_PROMETHEUS_BEARER)
...

Unfortunately, the behaviour is still the same. As far as I can see, I have compiled the correct version
Should the ‘Inactive’ not be the number of checks. in my case 125?

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P -T 0
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P -T OK
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P -T 1
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P -T CRITICAL
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P --no-alerts-state OK
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P --no-alerts-state 0
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

./check_prometheus -H metrics.clean-cluster.example.com -p 80 alert -P -T OK ; echo $?
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

3

Here I check whether, if there is ‘Firing’, it does not mistakenly go to OK. perfect, works as it should.

./check_prometheus -H metrics.some-probelm-cluster.example.com -p 80 alert -P -T OK
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00

./check_prometheus -H metrics.some-probelm-cluster.example.com -p 80 alert -P -T 0
[CRITICAL] - 1 Alerts: 1 Firing - 0 Pending - 0 Inactive
\_ [CRITICAL] [Watchdog] is firing - value: 1.00

[martialblog]

Thanks for the feedback, I'll have a another look

[martialblog]

Hi,

these are the results when checking Prometheus with the new logic:

# Starting Prometheus without any alert rules defined:

go run main.go alert
[OK] - No alerts defined

go run main.go alert -T 3
[UNKNOWN] - No alerts defined

go run main.go alert -T 2
[CRITICAL] - No alerts defined

go run main.go alert -T foo
[UNKNOWN] - invalid value for --no-alerts-state: foo (*errors.errorString)

# When an alert is defined:

go run main.go alert -T 2
[OK] - 1 Alerts: 0 Firing - 0 Pending - 1 Inactive
\_ [OK] [NodeHasMemoryPressure] is inactive

# When you are looking for a specific alert and alerts are defined:
# These counts only include the names that match the --name parameters

go run main.go alert -T 2 --name foo
[UNKNOWN] - 0 Alerts: 0 Firing - 0 Pending - 0 Inactive

go run main.go alert -T 1 --name foo --name NodeHasMemoryPressure
[OK] - 1 Alerts: 0 Firing - 0 Pending - 1 Inactive
\_ [OK] [NodeHasMemoryPressure] is inactive
|total=1 firing=0 pending=0 inactive=1

# When you are looking for a specific alert and no alerts are defined:

go run main.go alert -T 2 --name foo
[UNKNOWN] - No such alert defined

As far as I can tell, this is the behavior we want.