Add AWS SNS receiver

[treid314]

Work for issue: #2559

Work Remaining:

• Support AWS ARN auth
• Get AWS keys from environment variables
• Properly truncate SNS messages
• Add email templates
• Document new config options for SNS

Signed-off-by: Tyler Reid tyler.reid@grafana.com

[roidelapluie]

Yes, preferably SIGv4 would move to its own go module in common.

[pracucci]

Thanks @treid314 for working on this! I left some comments to hopefully help you progressing on it.

[pracucci]

How is this test expected to work? We can't really push to SNS endpoint.

[treid314]

This test has been used for my local testing and will be removed.

[pracucci]

Does the AWS SDK allows you to mock a service (SNS in this case)? Can we actually write a unit test on Notify()?

[treid314]

There's not a built in mocking in the AWS SDK. If I re-work the the way we create the publish input and move it to a function we could check that logic against the publish input we expect to generate. But I'm not seeing a way to test Notify() directly.

[pracucci]

Good job! I think we're very close. I left few more comments. Mostly nits.

[pracucci]

Good job! The logic LGTM. From my perspective the only missing thing is getting the sigv4 config merged into prometheus/common and vendor it here.

[cstyan]

Looks sane to me overall, though I've never used AWS SNS.

@simonpasquier @w0rm can either of you take a look?

EDIT: meant to comment, not approve

[roidelapluie]

did you see my comment about the notify method? can we split it to make it more readable/reviewable?

[treid314]

@roidelapluie Thank you for the review! I updated the notify method to make it more readable. Let me know if you have any other comments!

[roidelapluie]

Thanks! There would be some clean-up to be done on the imports and one comment does not start with a capital letter but it has no consequences on the code, so let's merge this and create a followup PR later.

[kevinayres]

This is great! You guys rock! Can't wait to try it.

[just1900]

Hi, @treid314 sorry to bother you, I'm wondering if there is a way to send alerts to sns without ak/sk?

[jeromeinsf]

@just1900 SNS does not support unauthenticated calls against its APIs

[just1900]

@just1900 SNS does not support unauthenticated calls against its APIs

Thanks for your reply, I'm currently running alertmanager under eks, I mean using the identity token instead of ak/sk directly.

[jeromeinsf]

I see, so https://docs.aws.amazon.com/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up-enable-IAM.html would not help?

[just1900]

I've configured serviceaccount with IAM role. This does work finally :)
I dive a little bit into the implementation of sns notifier and figure out how to send sns request with identity token. The final config would be like

  sns_configs:
  - topic_arn: arn:aws:sns:us-east-1:xxx:am-test
#    api_url: https://sns.us-east-1.amazonaws.com # do not set this
    sigv4:
      region: us-east-1
#      role_arn: <role> # do not set this

The first thing that confuse me is that if setting the role_arn, aws sdk will try to assume the role of role_arn with a AssumeRoleWithWebIdentity role, and return an unauthorized error. I'm not sure how to bypass this.

alertmanager/notify/sns/sns.go

Lines 119 to 130 in dbc8618

	stsSess, err = session.NewSessionWithOptions(session.Options{
	Config: aws.Config{
	Region: aws.String(n.conf.Sigv4.Region),
	Credentials: creds,
	},
	Profile: n.conf.Sigv4.Profile,
	})
	if err != nil {
	return nil, err
	}
	}
	creds = stscreds.NewCredentials(stsSess, n.conf.Sigv4.RoleARN)

Then I get role_arn commented, however, alertmanager will use the api_url as an endpoint when retrieving sts credentials.

alertmanager/notify/sns/sns.go

Lines 102 to 106 in dbc8618

	sess, err := session.NewSessionWithOptions(session.Options{
	Config: aws.Config{
	Region: aws.String(n.conf.Sigv4.Region),
	Endpoint: aws.String(tmpl(n.conf.APIUrl)),
	},

which may cause the AssumeRoleWithWebIdentity not work as documented in aws/aws-sdk-go#3972.

IMO, we can optimize the code here like aws/aws-sdk-go#3972 (comment) to gain a better user experience.