Skip to content

Implement Secret storage backend as gRPC server#644

Merged
pkosiec merged 10 commits intocapactio:mainfrom
pkosiec:secret-storage-backend
Feb 28, 2022
Merged

Implement Secret storage backend as gRPC server#644
pkosiec merged 10 commits intocapactio:mainfrom
pkosiec:secret-storage-backend

Conversation

@pkosiec
Copy link
Copy Markdown
Collaborator

@pkosiec pkosiec commented Feb 24, 2022

Description

Changes proposed in this pull request:

  • Add Protocol Buffers schema
  • Generate gRPC client + server code from Protocol Buffers
  • Implement gRPC server for secret storage backend
    • Support initially just AWS Secrets Manager. Tested also dotenv provider. The vault provider needs to be tested
    • Add tests for the gRPC server with fake secret provider

TODO

Notes

Lint failing because of timeout. Don't worry about it - The job will be green after review 🙂

Testing

To regenerate the gRPC code, run:

SKIP_DEPS_INSTALLATION=false make gen-grpc-resources

To run example calls against the gRPC service:

  1. Create AWS security credentials with SecretsManagerReadWrite policy.

  2. Export environment variables:

     export AWS_ACCESS_KEY_ID="{accessKey}"
     export AWS_SECRET_ACCESS_KEY="{secretKey}"
  3. Run the server:

    APP_SUPPORTED_PROVIDERS=dotenv,aws_secretsmanager APP_LOGGER_DEV_MODE=true go run ./cmd/secret-storage-backend/main.go
  4. Now let's use Go generated gRPC client to to see how the server behaves. Run the example pkg/hub/api/grpc/storage_backend/example_test.go.

    To run it, you can e.g. change the func ExampleNewStorageBackendClient() { to func TestE2EScenario(t *testing.T) { and run it as an usual test.

  5. Change the line 12 of this test and run it again:

    provider := "aws_secretsmanager"

Related issue(s)

Resolves #637

@pkosiec pkosiec added enhancement New feature or request WIP Work in progress area/hub Relates to Hub labels Feb 24, 2022
@pkosiec pkosiec changed the title Implement Secret storage backend Implement Secret storage backend as gRPC server Feb 24, 2022
@pkosiec pkosiec force-pushed the secret-storage-backend branch from b5f2b6c to 795dd28 Compare February 24, 2022 08:24
@pkosiec pkosiec force-pushed the secret-storage-backend branch from 8af5f25 to 2c71d10 Compare February 24, 2022 13:22
@pkosiec pkosiec marked this pull request as ready for review February 24, 2022 13:41
@pkosiec pkosiec force-pushed the secret-storage-backend branch from 2c71d10 to 6474795 Compare February 24, 2022 13:57
Comment thread go.sum
Comment thread go.sum
@pkosiec pkosiec removed the WIP Work in progress label Feb 24, 2022
@pkosiec pkosiec added the WIP Work in progress label Feb 24, 2022
Copy link
Copy Markdown
Collaborator

@mszostok mszostok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I really enjoyed reviewing your PR, really nice work 👍

Comment thread cmd/secret-storage-backend/README.md Outdated
Comment thread cmd/secret-storage-backend/README.md
Comment thread cmd/secret-storage-backend/README.md Outdated
Comment thread cmd/secret-storage-backend/README.md
Comment thread cmd/secret-storage-backend/main.go Outdated
Comment thread internal/secret-storage-backend/server_test.go Outdated
Comment thread internal/secret-storage-backend/server_test.go Outdated
Comment thread pkg/hub/api/grpc/storage_backend.proto Outdated
Comment thread pkg/hub/api/grpc/storage_backend/example_test.go Outdated
Comment thread internal/secret-storage-backend/server.go Outdated
Copy link
Copy Markdown
Collaborator

@mszostok mszostok left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tutti bueno 😄 🚀

italian-hand-fancy

Comment thread pkg/hub/api/grpc/storage_backend/client_test.go
@@ -0,0 +1,315 @@
package secretstoragebackend
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably even earlier, as using the dotenvwill be useful in e2e tests, so we will need to add some padding to other task to figure this out or you can create a follow-up task to enable the Secret Store storage "consumption" by Local Hub.

Comment thread pkg/hub/api/grpc/storage_backend/client_test.go Outdated
@pkosiec
Copy link
Copy Markdown
Collaborator Author

pkosiec commented Feb 28, 2022

Integration tests failed during Capact iunstallation, but the previous run was successful (https://github.com/capactio/capact/actions/runs/1900095690) - and also this PR doesn't change anything related to integration tests, so I'm merging this without retry.

@pkosiec pkosiec merged commit f51af88 into capactio:main Feb 28, 2022
@pkosiec pkosiec deleted the secret-storage-backend branch February 28, 2022 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/hub Relates to Hub enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement AWS Secrets Manager storage backend logic

2 participants