Skip to content

Logging improvements #4113

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SuperQ merged 3 commits into from
Nov 14, 2025
Merged

Logging improvements #4113

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0207611
ref(webhook): return error if unable to parse group key
tjhop Nov 6, 2024
aaa6726
fix(silence): log errors at ERROR level instead of INFO
tjhop Nov 6, 2024
a64bfe2
chore(notify): make logging more consistent, converge on `group_key`
tjhop Nov 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 7 additions & 6 deletions notify/discord/discord.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -95,10 +95,11 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
return false, err
}

n.logger.Debug("extracted group key", "key", key)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")
Copy link
Contributor

@siavashs siavashs Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we move this inside notify.ExtractGroupKey() instead of repeating the same line of code in all integrations?

Copy link
Contributor Author

@tjhop tjhop Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds reasonable, but I'm trying to think through the most sane way to do that. Currently, notify.ExtractGroupKey() has the following signature:

alertmanager/notify/util.go

Lines 159 to 166 in 29d491e

// ExtractGroupKey gets the group key from the context.
func ExtractGroupKey(ctx context.Context) (Key, error) {
key, ok := GroupKey(ctx)
if !ok {
return "", fmt.Errorf("group key missing")
}
return Key(key), nil
}

Do we want to update the function signature to accept a logger as well?
Alternatively, I could use the top level slog.Debug() function inside of notify.ExtractGroupKey() if we want to shim a call in to main somewhere after we initialize the logger to slog.SetDefault(logger).

Thoughts?

Copy link
Contributor

@siavashs siavashs Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd say skip it, not worth the effort.

Copy link
Contributor Author

@tjhop tjhop Nov 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Still a good point, maybe a better option will present in the future


alerts := types.Alerts(as...)
data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
data := notify.GetTemplateData(ctx, n.tmpl, as, logger)
tmpl := notify.TmplText(n.tmpl, data, &err)
if err != nil {
return false, err
Expand All @@ -109,22 +110,22 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
return false, err
}
if truncated {
n.logger.Warn("Truncated title", "key", key, "max_runes", maxTitleLenRunes)
logger.Warn("Truncated title", "max_runes", maxTitleLenRunes)
}
description, truncated := notify.TruncateInRunes(tmpl(n.conf.Message), maxDescriptionLenRunes)
if err != nil {
return false, err
}
if truncated {
n.logger.Warn("Truncated message", "key", key, "max_runes", maxDescriptionLenRunes)
logger.Warn("Truncated message", "max_runes", maxDescriptionLenRunes)
}

content, truncated := notify.TruncateInRunes(tmpl(n.conf.Content), maxContentLenRunes)
if err != nil {
return false, err
}
if truncated {
n.logger.Warn("Truncated message", "key", key, "max_runes", maxContentLenRunes)
logger.Warn("Truncated message", "max_runes", maxContentLenRunes)
}

color := colorGrey
Expand Down Expand Up @@ -160,7 +161,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
if _, err := netUrl.Parse(n.conf.AvatarURL); err == nil {
w.AvatarURL = n.conf.AvatarURL
} else {
n.logger.Warn("Bad avatar url", "key", key)
logger.Warn("Bad avatar url", "key", key)
}
}

Expand Down
1 change: 1 addition & 0 deletions notify/jira/jira.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
}

logger := n.logger.With("group_key", key.String())
logger.Debug("extracted group key")

var (
alerts = types.Alerts(as...)
Expand Down
5 changes: 3 additions & 2 deletions notify/msteams/msteams.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,9 +85,10 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
return false, err
}

n.logger.Debug("extracted group key", "key", key)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
data := notify.GetTemplateData(ctx, n.tmpl, as, logger)
tmpl := notify.TmplText(n.tmpl, data, &err)
if err != nil {
return false, err
Expand Down
5 changes: 3 additions & 2 deletions notify/msteamsv2/msteamsv2.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,9 +109,10 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
return false, err
}

n.logger.Debug("extracted group key", "key", key)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
data := notify.GetTemplateData(ctx, n.tmpl, as, logger)
tmpl := notify.TmplText(n.tmpl, data, &err)
if err != nil {
return false, err
Expand Down
7 changes: 4 additions & 3 deletions notify/opsgenie/opsgenie.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,9 +132,10 @@ func (n *Notifier) createRequests(ctx context.Context, as ...*types.Alert) ([]*h
if err != nil {
return nil, false, err
}
data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

n.logger.Debug("extracted group key", "key", key)
data := notify.GetTemplateData(ctx, n.tmpl, as, logger)

tmpl := notify.TmplText(n.tmpl, data, &err)

Expand Down Expand Up @@ -174,7 +175,7 @@ func (n *Notifier) createRequests(ctx context.Context, as ...*types.Alert) ([]*h
default:
message, truncated := notify.TruncateInRunes(tmpl(n.conf.Message), maxMessageLenRunes)
if truncated {
n.logger.Warn("Truncated message", "alert", key, "max_runes", maxMessageLenRunes)
logger.Warn("Truncated message", "alert", key, "max_runes", maxMessageLenRunes)
}

createEndpointURL := n.conf.APIURL.Copy()
Expand Down
5 changes: 3 additions & 2 deletions notify/pagerduty/pagerduty.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -308,17 +308,18 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
if err != nil {
return false, err
}
logger := n.logger.With("group_key", key)

var (
alerts = types.Alerts(as...)
data = notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
data = notify.GetTemplateData(ctx, n.tmpl, as, logger)
eventType = pagerDutyEventTrigger
)
if alerts.Status() == model.AlertResolved {
eventType = pagerDutyEventResolve
}

n.logger.Debug("extracted group key", "key", key, "eventType", eventType)
logger.Debug("extracted group key", "eventType", eventType)

details := make(map[string]string, len(n.conf.Details))
for k, v := range n.conf.Details {
Expand Down
14 changes: 7 additions & 7 deletions notify/pushover/pushover.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,10 +72,10 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
if !ok {
return false, fmt.Errorf("group key missing")
}
data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

// @tjhop: should this use `group` for the keyval like most other notify implementations?
n.logger.Debug("extracted group key", "incident", key)
data := notify.GetTemplateData(ctx, n.tmpl, as, logger)

var (
err error
Expand Down Expand Up @@ -113,7 +113,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)

title, truncated := notify.TruncateInRunes(tmpl(n.conf.Title), maxTitleLenRunes)
if truncated {
n.logger.Warn("Truncated title", "incident", key, "max_runes", maxTitleLenRunes)
logger.Warn("Truncated title", "incident", key, "max_runes", maxTitleLenRunes)
}
parameters.Add("title", title)

Expand All @@ -130,7 +130,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)

message, truncated = notify.TruncateInRunes(message, maxMessageLenRunes)
if truncated {
n.logger.Warn("Truncated message", "incident", key, "max_runes", maxMessageLenRunes)
logger.Warn("Truncated message", "incident", key, "max_runes", maxMessageLenRunes)
}
message = strings.TrimSpace(message)
if message == "" {
Expand All @@ -141,7 +141,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)

supplementaryURL, truncated := notify.TruncateInRunes(tmpl(n.conf.URL), maxURLLenRunes)
if truncated {
n.logger.Warn("Truncated URL", "incident", key, "max_runes", maxURLLenRunes)
logger.Warn("Truncated URL", "incident", key, "max_runes", maxURLLenRunes)
}
parameters.Add("url", supplementaryURL)
parameters.Add("url_title", tmpl(n.conf.URLTitle))
Expand All @@ -167,7 +167,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
}
u.RawQuery = parameters.Encode()
// Don't log the URL as it contains secret data (see #1825).
n.logger.Debug("Sending message", "incident", key)
logger.Debug("Sending message", "incident", key)
resp, err := notify.PostText(ctx, n.client, u.String(), nil)
if err != nil {
return true, notify.RedactURL(err)
Expand Down
15 changes: 9 additions & 6 deletions notify/rocketchat/rocketchat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,7 +140,14 @@ func getToken(c *config.RocketchatConfig) (string, error) {
func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error) {
var err error

data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
key, err := notify.ExtractGroupKey(ctx)
if err != nil {
return false, err
}
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

data := notify.GetTemplateData(ctx, n.tmpl, as, logger)
tmplText := notify.TmplText(n.tmpl, data, &err)
if err != nil {
return false, err
Expand All @@ -152,11 +159,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)

title, truncated := notify.TruncateInRunes(title, maxTitleLenRunes)
if truncated {
key, err := notify.ExtractGroupKey(ctx)
if err != nil {
return false, err
}
n.logger.Warn("Truncated title", "key", key, "max_runes", maxTitleLenRunes)
logger.Warn("Truncated title", "max_runes", maxTitleLenRunes)
}
att := &Attachment{
Title: title,
Expand Down
16 changes: 10 additions & 6 deletions notify/slack/slack.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,8 +93,16 @@ type attachment struct {
// Notify implements the Notifier interface.
func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error) {
var err error

key, err := notify.ExtractGroupKey(ctx)
if err != nil {
return false, err
}
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

var (
data = notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
data = notify.GetTemplateData(ctx, n.tmpl, as, logger)
tmplText = notify.TmplText(n.tmpl, data, &err)
)
var markdownIn []string
Expand All @@ -107,11 +115,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)

title, truncated := notify.TruncateInRunes(tmplText(n.conf.Title), maxTitleLenRunes)
if truncated {
key, err := notify.ExtractGroupKey(ctx)
if err != nil {
return false, err
}
n.logger.Warn("Truncated title", "key", key, "max_runes", maxTitleLenRunes)
logger.Warn("Truncated title", "max_runes", maxTitleLenRunes)
}
att := &attachment{
Title: title,
Expand Down
19 changes: 11 additions & 8 deletions notify/telegram/telegram.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,27 +64,30 @@ func New(conf *config.TelegramConfig, t *template.Template, l *slog.Logger, http
}

func (n *Notifier) Notify(ctx context.Context, alert ...*types.Alert) (bool, error) {
key, ok := notify.GroupKey(ctx)
if !ok {
return false, fmt.Errorf("group key missing")
}

logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

var (
err error
data = notify.GetTemplateData(ctx, n.tmpl, alert, n.logger)
data = notify.GetTemplateData(ctx, n.tmpl, alert, logger)
tmpl = notify.TmplText(n.tmpl, data, &err)
)

if n.conf.ParseMode == "HTML" {
tmpl = notify.TmplHTML(n.tmpl, data, &err)
}

key, ok := notify.GroupKey(ctx)
if !ok {
return false, fmt.Errorf("group key missing")
}

messageText, truncated := notify.TruncateInRunes(tmpl(n.conf.Message), maxMessageLenRunes)
if err != nil {
return false, err
}
if truncated {
n.logger.Warn("Truncated message", "alert", key, "max_runes", maxMessageLenRunes)
logger.Warn("Truncated message", "max_runes", maxMessageLenRunes)
}

n.client.Token, err = n.getBotToken()
Expand All @@ -101,7 +104,7 @@ func (n *Notifier) Notify(ctx context.Context, alert ...*types.Alert) (bool, err
if err != nil {
return true, err
}
n.logger.Debug("Telegram message successfully published", "message_id", message.ID, "chat_id", message.Chat.ID)
logger.Debug("Telegram message successfully published", "message_id", message.ID, "chat_id", message.Chat.ID)

return false, nil
}
Expand Down
2 changes: 1 addition & 1 deletion notify/victorops/victorops.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@ func (n *Notifier) createVictorOpsPayload(ctx context.Context, as ...*types.Aler

stateMessage, truncated := notify.TruncateInRunes(stateMessage, maxMessageLenRunes)
if truncated {
n.logger.Warn("Truncated state_message", "incident", key, "max_runes", maxMessageLenRunes)
n.logger.Warn("Truncated state_message", "group_key", key, "max_runes", maxMessageLenRunes)
}

msg := map[string]string{
Expand Down
7 changes: 4 additions & 3 deletions notify/webex/webex.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,10 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
return false, err
}

n.logger.Debug("extracted group key", "key", key)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
data := notify.GetTemplateData(ctx, n.tmpl, as, logger)
tmpl := notify.TmplText(n.tmpl, data, &err)
if err != nil {
return false, err
Expand All @@ -87,7 +88,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)

message, truncated := notify.TruncateInBytes(message, maxMessageSize)
if truncated {
n.logger.Debug("message truncated due to exceeding maximum allowed length by webex", "truncated_message", message)
logger.Debug("message truncated due to exceeding maximum allowed length by webex", "truncated_message", message)
}

w := webhook{
Expand Down
7 changes: 3 additions & 4 deletions notify/webhook/webhook.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,12 +82,11 @@ func (n *Notifier) Notify(ctx context.Context, alerts ...*types.Alert) (bool, er

groupKey, err := notify.ExtractGroupKey(ctx)
if err != nil {
// @tjhop: should we `return false, err` here as we do in most
// other Notify() implementations?
n.logger.Error("error extracting group key", "err", err)
return false, err
}

n.logger.Debug("extracted group key", "key", groupKey.String())
logger := n.logger.With("group_key", groupKey)
logger.Debug("extracted group key")

msg := &Message{
Version: "4",
Expand Down
8 changes: 5 additions & 3 deletions notify/wechat/wechat.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,10 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
return false, err
}

n.logger.Debug("extracted group key", "key", key)
data := notify.GetTemplateData(ctx, n.tmpl, as, n.logger)
logger := n.logger.With("group_key", key)
logger.Debug("extracted group key")

data := notify.GetTemplateData(ctx, n.tmpl, as, logger)

tmpl := notify.TmplText(n.tmpl, data, &err)
if err != nil {
Expand Down Expand Up @@ -174,7 +176,7 @@ func (n *Notifier) Notify(ctx context.Context, as ...*types.Alert) (bool, error)
if err != nil {
return true, err
}
n.logger.Debug(string(body), "incident", key)
logger.Debug(string(body))

var weResp weChatResponse
if err := json.Unmarshal(body, &weResp); err != nil {
Expand Down
3 changes: 1 addition & 2 deletions silence/silence.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -555,8 +555,7 @@ Loop:
return
}
if err := runMaintenance(doMaintenance); err != nil {
// @tjhop: this should probably log at error level
s.logger.Info("Creating shutdown snapshot failed", "err", err)
s.logger.Error("Creating shutdown snapshot failed", "err", err)
}
}

Expand Down
Loading