Infrastructure rockets: the way to extend Booster infrastructure in AWS. (Also transformed the static site deployment feature into the first rocket) by javiertoledo · Pull Request #414 · boostercloud/booster

javiertoledo · 2020-09-13T23:38:45Z

In this PR we introduce the concept of "rockets" as the way to extend Booster. It's likely that we start to use this pattern more and more to make the framework more configurable and easier to extend.

Some notes and ideas to take into account before reviewing this PR:

Infrastructure rockets must be loaded dynamically to avoid importing them in the user's project, as that would require them to include the whole CDK in the deployed lambdas.
I expect this change to slightly accelerate the deployments in the framework integration tests.
One key of this change is removing the static page deployment feature from the framework completely and move it to a separate project that does not interfere with the development of the framework.
The next logical change would be removing the plugin from this repository and move it to its own one, but we'll keep the first plugins in the Booster monorepo until the feature is stabilized.
If we want to provide a way to extend the runtime too, rocket developers will need to provide two packages, one that's loaded in the context of the user's project, the other, loaded dynamically on deploy time (as the one in this PR).
Rockets are dependent on a specific provider because different providers have different ways to extend the infrastructure. In this PR we only changed the AWS provider to have a constructor function that accepts the plugins.
One great thing about this implementation is that it makes it relatively easy for advanced users to manipulate the project stack adding any resources they want with the AWS CDK.

TODOs

Deployment of resources defined by the plugin
Nuking resources deployed by the plugin
Implement tests for the plugins load process in AWS and AWS infra packages
Implement tests for the passing of the plugins through the different layers of AWS infrastructure package
Implement tests for deploying the resources defined by the plugin
Implement tests for destroying the resources defined by the plugin
Implement tests for ApplicationStackBuilder management of plugins
Add documentation about infrastructure plugins
~~- [ ] Implement an integration test with a simpler "reference rocket" that demonstrates that the stack extensions are deployed.~~

packages/framework-provider-aws-infrastructure/src/index.ts

packages/framework-provider-aws-infrastructure/src/infraestructure-plugin.ts

packages/framework-provider-aws-infrastructure/src/infrastructure/index.ts

packages/framework-provider-aws-infrastructure/test/infrastructure/deploy.test.ts

...s/framework-provider-aws-infrastructure/test/infrastructure/stacks/application-stack.test.ts

alvaroloes · 2020-09-14T15:22:21Z

It is a great start Javi! I like the approach.

I see that plugins in this PR can only change/add/modify infrastructure. I envision that plugins would also need to add functionality to the provider runtime and the framework core.

I would try to put together a way of allowing plugins in the three Booster areas (core, provider runtime, provider infrastructure) based on the approach you have followed here (although not in such detail to begin with)

javiertoledo · 2020-09-14T15:53:50Z

I think it would be nice to avoid implementing core or runtime plugins until we actually have a scenario where that's needed. Maybe a good one would be moving the current auth implementation to a plugin, but I think that we would need to agree on how we want to manage the auth tokens in a more general way in Booster first.

NickSeagull · 2020-10-05T15:50:36Z

Hello, how's this going? I'm super excited about this feature 😄

javiertoledo · 2020-10-05T16:49:57Z

Going fine, just completing the tests and docs :-D

juanjoman

LGTM!! Just left some nit-picky comments 👏

packages/framework-provider-aws-infrastructure/test/infrastructure/nuke.test.ts

NickSeagull

Looks awesome, I'm hyped to start using this!

Wanted to create a Discord bot using Booster, but then realized that I would need a webhook rocket 👀

packages/framework-provider-aws-infrastructure/src/infrastructure/stacks/application-stack.ts

NickSeagull · 2020-10-14T13:47:47Z

packages/framework-provider-aws-infrastructure/src/rockets/infrastructure-rocket.ts

+  unmountStack: (utils: RocketUtils) => void
+}
+
+// Separate function to make it easier to mock in tests


I'd add something like "not type safe, take care" 😜

What do you mean Nick?

NickSeagull · 2020-10-14T13:48:33Z

packages/framework-provider-aws-infrastructure/test/index.test.ts

+      })
+
+      describe('nuke', () => {
+        xit('initializes nuke with no rockets')


Is this a typo? 👀

Never seen this xit. Also, I see the test is not implemented. Maybe it has something to do with that...

It means "pending test" and mocha shows it as pending in the test list with a different color when they're run 😎

NickSeagull · 2020-10-14T13:49:40Z

packages/framework-provider-aws-infrastructure/test/infrastructure/deploy.test.ts

-      // Monkey patching this private function to be able to spy it
-      const fakeGetStackServiceConfiguration = fake(Infrastructure.__get__('getStackServiceConfiguration'))
-      Infrastructure.__set__('getStackServiceConfiguration', fakeGetStackServiceConfiguration)
+      spy(StackServiceConfiguration, 'getStackServiceConfiguration')


You could save this into a const and then use it on line 56

packages/framework-provider-aws-infrastructure/test/infrastructure/nuke.test.ts

packages/framework-provider-aws/src/index.ts

…proach for infrastructure extensibility via plugins

Co-authored-by: Nick Tchayka <nick@booster.cloud>

…o the stack builder in the AWS infrastructure package

- Renamed plugins as "rockets", the official name for Booster plugins - Added support for rockets unmounting in the nuke process - Created tests for rockets passing through the nuke functions - Added missing tests for the whole nuke process - Moved the nuke operation to its own file - Created a rocket utils object that include the same tools available in Booster

javiertoledo · 2020-10-15T20:33:53Z

As there are some known issues in the way integration tests run with breaking changes, and given the size of this PR, I think that it makes sense to release this functionality now and implement an integration test in a future PR, so I'm setting this one ready for review!

alvaroloes

Finishe review!! I just left two more questions

packages/rocket-static-sites-aws-infrastructure/src/static-website-stack.ts

alvaroloes · 2020-10-28T17:29:08Z

packages/rocket-static-sites-aws-infrastructure/src/static-website-stack.ts

+      const staticSiteBucket = new Bucket(stack, 'staticWebsiteBucket', {
+        websiteIndexDocument: indexFile,
+        websiteErrorDocument: errorFile,
+        bucketName: params.bucketName,


I see the bucketName does not have a default name, but I think we could provide one and the user would not need to specify it.

What do you think about using config.applicationName + static-site-bucket?

This would require many changes because in the current implementation we don't have access to config. I've created issue #482 to tackle this.

…ebsite-feature-into-a-plugin

Co-authored-by: Álvaro López Espinosa <alvaro@booster.cloud> Co-authored-by: Nick Tchayka <nick@booster.cloud>

Co-authored-by: Nick Tchayka <nick@booster.cloud>

…n the static website rocket

…hub.com:boostercloud/booster into feat/moved-static-website-feature-into-a-plugin

javiertoledo · 2020-11-02T19:10:48Z

Integration tests failed, I think it's because this branch doesn't have the changes in #476, so I'd wait for that PR to be merged before trying this again.

…ebsite-feature-into-a-plugin # Conflicts: # packages/framework-integration-tests/integration/fixtures/cart-demo/src/config/config.ts

javiertoledo · 2020-11-04T21:20:29Z

packages/framework-integration-tests/integration/cli/cli.project.integration.ts

This is crazy slow. For some reason, I was forced to rebuild all modules to make the project compile, but with this change, we're using the code under development instead of the libraries installed from npm. I don't know if there are other places in the code that would need to use this trick. It's very likely that we could optimize this in the future, but for this PR, this is more than enough...

javiertoledo · 2020-11-04T21:21:32Z

packages/cli/src/templates/project/config-ts.ts

 Booster.configure('production', (config: BoosterConfig): void => {
  config.appName = '{{{ projectName }}}'
-  config.provider = Provider
+  config.provider = Provider()


I was forced to make all providers export a function to avoid breaking the project generator. Having a heterogeneous interface like the one I proposed makes code generation way harder.

javiertoledo requested review from NickSeagull, alvaroloes, charlietfe, itrion and rdiaz82 September 13, 2020 23:38

NickSeagull suggested changes Sep 14, 2020

View reviewed changes

javiertoledo force-pushed the feat/moved-static-website-feature-into-a-plugin branch from 3b32cb7 to ba4ffca Compare September 23, 2020 19:28

javiertoledo changed the title ~~Possible approach for infrastructure plugins~~ Moved static website into a plugin and created basic plugin capabilities Sep 30, 2020

javiertoledo changed the title ~~Moved static website into a plugin and created basic plugin capabilities~~ Moved static website into a plugin and created basic infrastructure plugin capabilities Sep 30, 2020

javiertoledo force-pushed the feat/moved-static-website-feature-into-a-plugin branch from ba4ffca to 2042a9d Compare September 30, 2020 19:44

javiertoledo changed the title ~~Moved static website into a plugin and created basic infrastructure plugin capabilities~~ Infrastructure rockets to extend Booster infrastructure in AWS. Moved static website into the first rocket. Oct 9, 2020

juanjoman approved these changes Oct 13, 2020

View reviewed changes

NickSeagull approved these changes Oct 14, 2020

View reviewed changes

Javier Toledo and others added 10 commits October 15, 2020 19:57

Moved the static website feature into a plugin. Drafted a possible ap…

81b43b0

…proach for infrastructure extensibility via plugins

Fix typo in an error message

f7951df

Co-authored-by: Nick Tchayka <nick@booster.cloud>

Applied some changes suggested in code review

11cd88f

Added tests for the plugin load process

9ae63d6

Fixed linter issues

9c830fc

Ooops

0f14fd9

Fixed broken interfaces after rebase

82e853d

Renamed plugin as rocket, implemented tests for passing the plugins t…

dad870b

…o the stack builder in the AWS infrastructure package

Added docs for rockets functionality

f42e702

javiertoledo force-pushed the feat/moved-static-website-feature-into-a-plugin branch from 32b5d49 to f42e702 Compare October 15, 2020 20:20

javiertoledo marked this pull request as ready for review October 15, 2020 20:30

javiertoledo linked an issue Oct 27, 2020 that may be closed by this pull request

Create a reference rocket implementation to add integration tests for rocket stack extension deployments in AWS #464

Closed

alvaroloes reviewed Oct 28, 2020

View reviewed changes

alvaroloes approved these changes Oct 28, 2020

View reviewed changes

Javier Toledo and others added 8 commits November 2, 2020 17:08

Merge remote-tracking branch 'public/master' into feat/moved-static-w…

650183b

…ebsite-feature-into-a-plugin

Fixed TOC in docs/README.md

6c23f11

Apply suggestions from code review

639de1e

Co-authored-by: Álvaro López Espinosa <alvaro@booster.cloud> Co-authored-by: Nick Tchayka <nick@booster.cloud>

Made mountStack optional

0cb09a9

Typo in packages/framework-provider-aws/src/index.ts

600baf8

Co-authored-by: Nick Tchayka <nick@booster.cloud>

Rename localDistPath with a more generic rootPath config option i…

2f491c7

…n the static website rocket

Merge branch 'feat/moved-static-website-feature-into-a-plugin' of git…

bec343c

…hub.com:boostercloud/booster into feat/moved-static-website-feature-into-a-plugin

Fix static site rocket tests

fb79d91

javiertoledo mentioned this pull request Nov 2, 2020

Write documentation for the static site rocket #483

Open

boostercloud locked as resolved and limited conversation to collaborators Nov 2, 2020

boostercloud unlocked this conversation Nov 2, 2020

Merge remote-tracking branch 'public/master' into feat/moved-static-w…

143daec

…ebsite-feature-into-a-plugin # Conflicts: # packages/framework-integration-tests/integration/fixtures/cart-demo/src/config/config.ts

javiertoledo commented Nov 4, 2020

View reviewed changes

boostercloud locked as resolved and limited conversation to collaborators Nov 4, 2020

boostercloud unlocked this conversation Nov 4, 2020

boostercloud locked as resolved and limited conversation to collaborators Nov 9, 2020

Fixed local integration tests. Applied minor code improvements

4fa3f89

javiertoledo force-pushed the feat/moved-static-website-feature-into-a-plugin branch from 71e04f5 to 4fa3f89 Compare November 9, 2020 12:35

boostercloud unlocked this conversation Nov 9, 2020

boostercloud locked as too heated and limited conversation to collaborators Nov 9, 2020

boostercloud unlocked this conversation Nov 9, 2020

boostercloud locked and limited conversation to collaborators Nov 9, 2020

boostercloud unlocked this conversation Nov 9, 2020

javiertoledo merged commit b1c0a4e into master Nov 9, 2020

javiertoledo deleted the feat/moved-static-website-feature-into-a-plugin branch November 9, 2020 15:16

Conversation

javiertoledo commented Sep 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alvaroloes commented Sep 14, 2020

Uh oh!

javiertoledo commented Sep 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NickSeagull commented Oct 5, 2020

Uh oh!

javiertoledo commented Oct 5, 2020

Uh oh!

juanjoman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

NickSeagull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

javiertoledo commented Oct 15, 2020

Uh oh!

alvaroloes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javiertoledo commented Nov 2, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

javiertoledo commented Sep 13, 2020 •

edited

Loading

javiertoledo commented Sep 14, 2020 •

edited

Loading