Add logging during awskms auto-unseal by tvoran · Pull Request #9794 · hashicorp/vault

tvoran · 2020-08-21T00:54:10Z

Adds debug and warn logging around AWS credential chain generation, specifically to help users understand why auto-unseal isn't working on AWS, by logging which role is being used in the case of a webidentity token.

Also in the case of webidentity, this tests retrieving credentials, and logs a warning if it has trouble, in which case it falls through to using creds from the instance metadata, which may still fail, but with a warning logged at least users can tell why the webidentity auth failed.

Depends on hashicorp/go-kms-wrapping#22

Example logging with level=trace when auto-unseal fails:

before:


Error parsing Seal configuration: error fetching AWS KMS wrapping key information: AccessDeniedException: User: arn:aws:sts::000000000000:assumed-role/eksctl-tvoran-eks-dev-nodegroup-s-NodeInstanceRole-000000000000/i-0000000000000 is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:us-west-2:000000000000:key/00000000-00000000-00000000-00000000
    status code: 400, request id: 00000000-00000000-00000000-00000000
stream closed

after:


2020-08-20T20:33:47.809Z [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-08-20T20:33:47.812Z [DEBUG] service_registration.kubernetes: "namespace": "default"
2020-08-20T20:33:47.812Z [DEBUG] service_registration.kubernetes: "pod_name": "vault-0"
2020-08-20T20:33:47.812Z [DEBUG] seal.awskms: adding web identity provider: roleARN=arn:aws:iam::000000000000:role/eksctl-my-cluster-addon-iamserviceaccoun-Role1-0000000000000
2020-08-20T20:33:48.798Z [WARN]  seal.awskms: error assuming role: roleARN=arn:aws:iam::000000000000:role/eksctl-my-cluster-addon-iamserviceaccoun-Role1-0000000000000 tokenPath=/var/run/secrets/eks.amazonaws.com/serviceaccount/token sessionName= err="WebIdentityErr: failed to retrieve credentials
caused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity
    status code: 403, request id: 00000000-00000000-00000000-00000000"
Error parsing Seal configuration: error fetching AWS KMS wrapping key information: AccessDeniedException: User: arn:aws:sts::000000000000:assumed-role/eksctl-tvoran-eks-dev-nodegroup-s-NodeInstanceRole-0000000000000/i-0000000000000000 is not authorized to perform: kms:DescribeKey on resource: arn:aws:kms:us-west-2:000000000000:key/00000000-00000000-00000000-00000000
    status code: 400, request id: 00000000-00000000-00000000-00000000
stream closed

Adds debug and warn logging around AWS credential chain generation, specifically to help users debugging auto-unseal problems on AWS, by logging which role is being used in the case of a webidentity token. Adds a deferred call to flush the log output as well, to ensure logs are output in the event of an initialization failure.

calvn · 2020-08-31T20:46:18Z

 				SecretAccessKey: c.SecretKey,
 				SessionToken:    c.SessionToken,
 			}})
+		c.Logger.Debug("added static credential provider", "AccessKey", c.AccessKey)


c.Logger could potentially be nil if none is passed into CredentialsConfig so these calls would panic. Let's wrap logger calls with a nil check.

Sure, added in 98adda6

calvn · 2020-09-01T22:25:28Z

 	}

-	creds, err := RetrieveCreds(m["aws_access_key_id"], m["aws_secret_access_key"], m["aws_security_token"])
+	creds, err := RetrieveCreds(m["aws_access_key_id"], m["aws_secret_access_key"], m["aws_security_token"], hclog.Default())


This does not derive any of the logging properties from the main command, so it might result in different format, log level, etc.

How would we get the logging properties from the main command here? I haven't found a way to do that.

The log would be from the client (result from a vault login command) for this particular case so it might be fine, though hclog.Default() logs up to Info only, so it would skip any Debug output.

I'm not seeing anywhere else in the vault login command that verbosity levels are set, so in 2271d6c I added a log_level option to the aws cli login command, and used that to configure the logger passed to RetrieveCreds(). Or maybe it's preferable to just pass a null logger here?

Used to setup the logger for use in GenerateCredentialChain()

…ging

calvn · 2020-09-28T20:47:15Z

 	github.com/hashicorp/go-gcp-common v0.6.0
 	github.com/hashicorp/go-hclog v0.14.1
-	github.com/hashicorp/go-kms-wrapping v0.5.12
+	github.com/hashicorp/go-kms-wrapping v0.5.15


We should be using v0.5.16 right?

Yep, just fixed in e4f2e2b

* adds error handling for control groups to api service as post request middleware * updates kv list route to use api service * updates kv config route to use api service * updates kv secrets overview route to use api service * updates kv secret details route to use api service * adds kv form * updates kv metadata details route to use api service * updates kv paths and version history routes to use api service * refactors kv-data-fields component to form component * updates kv secret create route to use api service * updates kv secret edit route to use api service * updates kv metadata edit route to use api service * adds waitFor to async middleware in api service to attempt to fix race conditions in tests * adds kvMetadata path to capabilities path map * fixes kv list item delete test selector * removes kv models, adapters and serializers * removes store from kv addon * removes ember data related test helpers from kv-run-commands * updates comments that referred to kv ember data models * updates kv-page-header tests * updates model-form-fields test to use totp-key model rather than kv/data * removes another reference to kv/data model from path-help test * fixes kv v2 workflow create tests * fixes issue returning metadata for secret when latest version is deleted * decodes uri in path returned by api service parseError method * fixes kv v2 edge cases tests * fixes issue deleteing control group token in api service * decodes url for control group token lookup in api service * fixes version history linked block link * defaults cas to 0 when creating new secret * removes log * adds ember-template-lint to kv engine * more test fixes * updates kv-suggestion-input component to use api service * removes kv metadata model reference from quick actions card * fixes sync destination sync secrets tests * updates kv helpers from classic format * updates kv helpers imports * reverts to use secret.version in details edit route * fixes isDeleted import in kv version history test * adds waitFor to api service parseError method * reverts removing async from addQueryParams api method * attempts to fix test flakiness requesting custom metadata from data endpoint * more tweaks to requesting metadata from data * adds waitFor to requestData method Co-authored-by: Jordan Reimer <zofskeez@gmail.com>

tvoran requested a review from a team August 21, 2020 00:54

tvoran mentioned this pull request Aug 27, 2020

Add a logger to awskms hashicorp/go-kms-wrapping#22

Merged

calvn reviewed Aug 31, 2020

View reviewed changes

calvn reviewed Sep 1, 2020

View reviewed changes

tvoran added 3 commits September 2, 2020 23:12

check for nil logger and update go-kms-wrapping

98adda6

Added log_level option to aws cli login

2271d6c

Used to setup the logger for use in GenerateCredentialChain()

Merge remote-tracking branch 'origin/master' into VAULT-92/awskms-log…

5bcd860

…ging

tvoran mentioned this pull request Sep 16, 2020

Adding logger to CredentialsConfig #9969

Merged

Merge remote-tracking branch 'origin/master' into VAULT-92/awskms-log…

4226eba

…ging

calvn approved these changes Sep 28, 2020

View reviewed changes

calvn added this to the 1.6 milestone Sep 28, 2020

Merge remote-tracking branch 'origin/master' into VAULT-92/awskms-log…

62dfab2

…ging

calvn reviewed Sep 28, 2020

View reviewed changes

add go-kms-wrapping@v0.5.16, go mod tidy, vendor

e4f2e2b

tvoran merged commit 10c0ada into master Sep 28, 2020

tvoran deleted the VAULT-92/awskms-logging branch September 28, 2020 21:06

pcman312 mentioned this pull request Sep 30, 2020

DBPW - Revert AutoMTLS #10065

Merged

tvoran mentioned this pull request Dec 7, 2020

Vault not booting on AWS using awskms seal hashicorp/vault-helm#368

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add logging during awskms auto-unseal#9794

Add logging during awskms auto-unseal#9794
tvoran merged 7 commits into
masterfrom
VAULT-92/awskms-logging

tvoran commented Aug 21, 2020

Uh oh!

calvn Aug 31, 2020

Uh oh!

tvoran Sep 3, 2020

Uh oh!

calvn Sep 1, 2020

Uh oh!

tvoran Sep 3, 2020

Uh oh!

calvn Sep 3, 2020

Uh oh!

tvoran Sep 16, 2020

Uh oh!

calvn Sep 28, 2020

Uh oh!

tvoran Sep 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

tvoran commented Aug 21, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants