Skip to content

Support setting the Gossip cluster "label"#3254

Closed
greed42 wants to merge 1 commit intoprometheus:mainfrom
greed42:gossip-label
Closed

Support setting the Gossip cluster "label"#3254
greed42 wants to merge 1 commit intoprometheus:mainfrom
greed42:gossip-label

Conversation

@greed42
Copy link

@greed42 greed42 commented Feb 15, 2023

This is an alternate mechanism for isolating Alertmanager clusters without having to set up the right components of TLS.

It should solve issues such as #2250, although enabling this feature will lead to loss of non-persisted state. (For example, if you rely on alertmanager cluster peering to maintain silences instead of using persistent volume storage in Kubernetes.) The Gossip label serves as the "cluster ID" idea mentioned in #2250 (comment).

You can enable with the command-line flag, --cluster.gossip-label; any non-empty string will form an effective namespace for gossip communication.

If you use Prometheus Operator, you can set the ALERTMANAGER_CLUSTER_GOSSIP_LABEL environment variable (as Prometheus Operator does not have a way of adding additional command-line flags). You would need to modify your Alertmanager object something like:

kind: Alertmanager
...
spec:
  ...
  containers:
    - name: alertmanager
      env:
        - name: ALERTMANAGER_CLUSTER_GOSSIP_LABEL
          value: infrastructure-eu-west-2
  ...

This is low-security mechanism, suitable for use with Alertmanager configuration where anyone can add or remove a silence. It protects against surprising cluster expansion due to IP:port re-use.

This is an alternate mechanism for isolating Alertmanager clusters without having to set up the right components of TLS.

It should solve issues such as <prometheus#2250>, although enabling this feature will lead to loss of non-persisted state. (For example, if you rely on alertmanager cluster peering to maintain silences instead of using persistent volume storage in Kubernetes.) The Gossip label serves as the "cluster ID" idea mentioned in <prometheus#2250 (comment)>.

You can enable with the command-line flag, `--cluster.gossip-label`; any non-empty string will form an effective namespace for gossip communication.

If you use Prometheus Operator, you can set the `ALERTMANAGER_CLUSTER_GOSSIP_LABEL` environment variable (as Prometheus Operator does not have a way of adding additional command-line flags). You would need to modify your Alertmanager object something like:

```
kind: Alertmanager
...
spec:
  ...
  containers:
    - name: alertmanager
      env:
        - name: ALERTMANAGER_CLUSTER_GOSSIP_LABEL
          value: infrastructure-eu-west-2
  ...
```

This is low-security mechanism, suitable for use with Alertmanager configuration where anyone can add or remove a silence. It protects against surprising cluster expansion due to IP:port re-use.

Signed-off-by: Graham Reed <greed@7deadly.org>
@simonpasquier
Copy link
Member

Superseded by #3354

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants