Harder, Better, Faster, Stronger CLI integration tests by javiertoledo · Pull Request #476 · boostercloud/booster

javiertoledo · 2020-10-25T12:10:52Z

Description

We have been experiencing reliability issues with CLI integration tests. They were directly changing the files from the sample project in the package framework-integration-tests, introducing issues that then affected deployment tests and others, as they all rely on these files to be unaltered.

You may want to listen to the OST of this PR while reviewing it 😎

Changes

In this PR, I've reworked the CLI integration tests with the hopes of solving the mentioned reliability issues and make the tests faster the process. To do that I've done the following:

Removed file management from setup.ts (Tests should be self-contained)
Each test file creates a sandbox folder where they can mess up the file system in any way they want without altering future tests. For commands that require an existing project, there's a new createSandboxProject function that creates a new copy of the sample project in the integration tests package.
Replaced exec calls with the promise-wrapped version from package child-process-promise for consistency and code simplicity.
Created a loadFixture helper and moved calls to low-level fs functions to fileHelper.ts to make tests code more readable
Replaced exact string comparations to matchers in many expect blocks to make them a bit more resilient to changes.
prompt interaction in project generator tests now waits for the actual prompt instead of writing the next thing every second.
Lots of changes to make the coder easier to understand by humans (esp. regexes)
Bonus: Added a VSCode launch descriptor to ease debugging CLI integration tests.
Bonus: Fixed the CLI config to keep a consistent context for skipGit and skipInstall flags (functionally there's no difference)

Checks

Project Builds
Project passes tests and checks
~~-Updated documentation accordingly~~ (not needed)

Additional notes

This change doesn't guarantee that other integration tests are reliable, I'll tackle that in future PRs.

NickSeagull

LGTM!

packages/framework-integration-tests/integration/cli/cli.event.integration.ts

packages/framework-integration-tests/integration/cli/cli.project.integration.ts

NickSeagull · 2020-10-27T14:57:20Z

packages/framework-integration-tests/integration/helper/fileHelper.ts


-export const removeFolders = (paths: Array<string>): Array<Promise<void>> =>
-  paths.map((path: string) => rmdir(path, { recursive: true }))
+export const createFolder = (folder: string): void => {


Maybe it'd be better to call this ensureFolderExists? As it doesn't create it if it exists

That makes sense...

…tern and stronger

…p function somewhere else :-#

javiertoledo · 2020-11-02T19:45:03Z

Ooops, AWS tests failing, I'll investigate...

charlietfe

Nothing to add, much cleaner and easy to understand tests. I did a nitty picky comment that you can omit it if you want.

So let's 🚢 🇮🇹 dude!

charlietfe · 2020-11-03T07:03:08Z

packages/framework-integration-tests/integration/cli/cli.command.integration.ts


-describe('Command', () => {
-  const cliPath = path.join('..', 'cli', 'bin', 'run')
+  const cliPath = path.join('..', '..', 'cli', 'bin', 'run')


Nit-picky, it looks like the path segments never change in cli tests. Does it make sense to use a placeholder constant with the segments and add that const to the fileHelper?

charlietfe · 2020-11-03T07:06:46Z

packages/framework-integration-tests/integration/cli/cli.command.integration.ts

    it('should create a new command', async () => {
      const expectedOutputRegex = new RegExp(
-        /(.+) boost (.+)?new:command(.+)? (.+)\n- Verifying project\n(.+) Verifying project\n- Creating new command\n(.+) Creating new command\n(.+) Command generated!\n/
+        ['boost new:command', 'Verifying project', 'Creating new command', 'Command generated'].join('(.|\n)*')


Oh god, what we had here, clearness to the rescue!

adriangmweb

It looks great! Much cleaner and easier to understand now.

Integrations tests are failing when trying to create a scheduled command, but anyway LGTM! 🚀

- Moved it to the AWS provider integration tests - Introduced a specific event - Used the CartReadModel, which wasn't restricted to a specific user role

…call

javiertoledo requested review from NickSeagull and alvaroloes October 25, 2020 12:11

javiertoledo self-assigned this Oct 25, 2020

javiertoledo added feature-request New feature or request hacktoberfest-accepted size: L Tasks that imply many files. labels Oct 25, 2020

javiertoledo mentioned this pull request Oct 25, 2020

feat(cli): ask to override files #475

Merged

3 tasks

javiertoledo force-pushed the fix/integration-tests branch from 05438fe to 95cd20e Compare October 25, 2020 12:38

NickSeagull approved these changes Oct 27, 2020

View reviewed changes

javiertoledo mentioned this pull request Oct 27, 2020

Simpler checkers #478

Merged

3 tasks

This was linked to issues Oct 27, 2020

CLI integration tests delete files from the repo #460

Closed

Integration test for CLI don't work #449

Closed

This was referenced Oct 27, 2020

Integration test for CLI don't work #449

Closed

CLI integration tests delete files from the repo #460

Closed

Javier Toledo added 3 commits November 2, 2020 18:49

Reworked project generator integration tests to make them faster, bet…

aa5220f

…tern and stronger

Sandboxed CLI integration tests for improved reliability

10479c5

Ooops, accidentally broke all other integration tests moving the slee…

a30b973

…p function somewhere else :-#

javiertoledo force-pushed the fix/integration-tests branch from 9439470 to a30b973 Compare November 2, 2020 18:49

javiertoledo mentioned this pull request Nov 2, 2020

Infrastructure rockets: the way to extend Booster infrastructure in AWS. (Also transformed the static site deployment feature into the first rocket) #414

Merged

8 tasks

javiertoledo requested review from adriangmweb and charlietfe November 2, 2020 19:10

boostercloud locked as resolved and limited conversation to collaborators Nov 2, 2020

boostercloud unlocked this conversation Nov 2, 2020

charlietfe approved these changes Nov 3, 2020

View reviewed changes

adriangmweb approved these changes Nov 3, 2020

View reviewed changes

Javier Toledo added 3 commits November 3, 2020 20:41

Less verbose waitForIt (too much logging makes things slow)

578e0bd

Added a launch config to run end-to-end tests in the VSCode debugger

9d8c73c

Reworked schedule commands integration test

adc641a

- Moved it to the AWS provider integration tests - Introduced a specific event - Used the CartReadModel, which wasn't restricted to a specific user role

boostercloud locked as resolved and limited conversation to collaborators Nov 3, 2020

Give enough time to the Scheduled Command tests to receive the first …

3c2997d

…call

boostercloud unlocked this conversation Nov 3, 2020

boostercloud locked as resolved and limited conversation to collaborators Nov 3, 2020

Fix small issue with long delays in waitForIt

3340a60

boostercloud unlocked this conversation Nov 3, 2020

boostercloud locked as resolved and limited conversation to collaborators Nov 3, 2020

javiertoledo merged commit a696750 into master Nov 3, 2020

javiertoledo deleted the fix/integration-tests branch November 3, 2020 22:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Harder, Better, Faster, Stronger CLI integration tests#476

Harder, Better, Faster, Stronger CLI integration tests#476
javiertoledo merged 8 commits intomasterfrom
fix/integration-tests

javiertoledo commented Oct 25, 2020

Uh oh!

NickSeagull left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

NickSeagull Oct 27, 2020

Uh oh!

javiertoledo Oct 27, 2020

Uh oh!

javiertoledo commented Nov 2, 2020

Uh oh!

charlietfe left a comment

Uh oh!

charlietfe Nov 3, 2020

Uh oh!

charlietfe Nov 3, 2020

Uh oh!

adriangmweb left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

javiertoledo commented Oct 25, 2020

Description

Changes

Checks

Additional notes

Uh oh!

NickSeagull left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

NickSeagull Oct 27, 2020

Choose a reason for hiding this comment

Uh oh!

javiertoledo Oct 27, 2020

Choose a reason for hiding this comment

Uh oh!

javiertoledo commented Nov 2, 2020

Uh oh!

charlietfe left a comment

Choose a reason for hiding this comment

Uh oh!

charlietfe Nov 3, 2020

Choose a reason for hiding this comment

Uh oh!

charlietfe Nov 3, 2020

Choose a reason for hiding this comment

Uh oh!

adriangmweb left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

NickSeagull left a comment •

edited

Loading