Ability to load fixture state for e2e tests #6215

whymarrh · 2019-02-27T17:47:37Z

This PR adds the ability for us to write e2e tests that use fixture data for both the extension's background state and the current state of the blockchain (via Ganache). Fortunately we compute the initial Redux store contents (the front-end state) from the initial background state so loading the background state from a fixture is currently synonymous with loading the Redux state.

Overview

As outlined in #6154:

We need to create infrastructure and tooling that allows us to run our e2e tests from an existing state.

We need a solution for the background that allows us to dynamically load state during test runs.

Once this task is complete, our e2e tests will be able to start the app at a state given in [JSON] file (plus [Ganache]) and run tests from there.

This PR introduces a way to write and organize fixtures into a set of directories that contain both the background and Ganache state as well as a load function that takes a path to a fixture directory, loads that state and runs a set of assertions in that context.

By example:

describe('MetaMask Browser Extension from existing state', function () {
  it('displays the lock screen', async () => {
    await load('onboarded', async ({driver}) => {
      const button = await driver.findElement('.unlock-page__title')
      await button.click()
    })
  })
})

This approach allows us to start the app in a given state and make assertions from there. The implementation (detailed below) allows us to execute each of the tests in parallel. The arguments passed to the callback currently include a wrapper around a WebDriver instance that exposes helper functions similar to those we've already been using (e.g. functions handling timeouts and retries). The helper functions on the WebDriver wrapper can be expanded to include any sugar/convenience methods we'd like (see also #6155, #6156).

The implementation

The background loads its state using the LocalStore wrapper for the storage.local API:

metamask-extension/app/scripts/background.js

Lines 182 to 186 in 160f196

    
           // read from disk 
        
           // first from preferred, async API: 
        
           versionedData = (await localStore.get()) || 
        
                           diskStore.getState() || 
        
                           migrator.generateInitialState(firstTimeState)

Background

The e2e tests now spin up a local HTTP server that provides fixture state to clients at pre-determined URLs. The extension now has an e2e flag set at build time that is used to set the application down e2e-specific flows which we're using in the background to load the initial state instead from the network (from said server). This setup keeps everything local but does introduce network latency where there wasn't any before only during e2e tests (a small cost). When built without the flag set the application behaves no differently.

Each time the load function is called the fixture server is pointed at a new fixture directory. This requires that the startup of each test be sequential but allows the tests to run in parallel after the initial state is loaded (i.e. all of the assertions, the majority of the test).

Ganache

Instead of using ganache-cli from the command line, the e2e tests now start ganache's server during each load invocation. At the same time the tests spin up a local server for the fixture data, we also start a ganache-cli server instance with the options outlined in the ${fixturesDirectory}/db.js file (note that this is a JS file instead of JSON to be able to things like path.resolve). Once the server is up and running, we can interact with the RPC endpoint. Because the server is started on any available port, a ganache reference is passed through to the tests to allow them to setup the RPC endpoint as needed.

The state of this PR (tasks)

The following task list will be updated as this PR is updated:

Ability to load background state in
- Chrome
- Firefox
Ability to load Ganache state

Testing this PR (no pun intended)

This PR includes an example spec file (e2e/tests/example.spec.js) which outlines the API future tests can be written with. (It might be deleted before this is merged?) To run the example spec file:

Firefox (the default browser):

npx mocha --no-timeouts e2e/tests/example.spec.js

Firefox, explicitly:

BROWSER=firefox npx mocha --no-timeouts e2e/tests/example.spec.js

Chrome:

BROWSER=chrome npx mocha --no-timeouts e2e/tests/example.spec.js

Alternative approaches to pre-loading background state

Briefly on the alternative approaches we've considered for this functionality:

Editing the extension source files before loading it

(A recap of a discussion in Slack.)

The old e2e tests did mutate background state by prepending the background file with a global variable assignment, setting the METAMASK_TEST_CONFIG:

metamask-extension/test/e2e/func.js

Lines 28 to 37 in cb2698d

    
           async function createModifiedTestBuild ({ browser, srcPath }) { 
        
             // copy build to test-builds directory 
        
             const extPath = path.resolve(`test-builds/${browser}`) 
        
             await fs.ensureDir(extPath) 
        
             await fs.copy(srcPath, extPath) 
        
             // inject METAMASK_TEST_CONFIG setting default test network 
        
             const config = { NetworkController: { provider: { type: 'localhost' } } } 
        
             await prependFile(`${extPath}/background.js`, `window.METAMASK_TEST_CONFIG=${JSON.stringify(config)};\n`) 
        
             return { extPath } 
        
           }

This approach doesn't support the ability for us to run tests in parallel. We could update it to include writing out a separate build of the extension to disk N times for N tests and edit each differently but that would be a lot of I/O during what is already an already slow test suite.

Preloading the extension state before loading the extension

The ideal approach to loading background would be to have the browser know about the state before the extension is installed. This is easily possible with Firefox but less easy to do with Chrome. Firefox stores extension state in a single text file on disk whereas Chrome stores its extension state across multiple binary leveldb databases.^[3]

kumavis · 2019-02-27T23:24:26Z

We have an API available for testing/tooling in the ui set the locale, could use it to override the state.

If we give it more powerful features like that we should consider deleting the api object after some time out, so it can only be used at boot and not abused at runtime

danjm · 2019-02-28T19:58:08Z

e2e/webdriver/chrome.js

This is an OS, and possibly individual system, dependent path

My bad, good catch

whymarrh · 2019-03-13T16:50:44Z

This PR currently supports the ability to load background state in both Firefox and Chrome

whymarrh · 2019-03-13T16:51:32Z

app/scripts/background.js

This needs to be conditional on some environment variable

But I think this can be merged in to the destination branch (e2e-2.0) for now

But I think this can be merged in to the destination branch (e2e-2.0) for now

We should create an issue or some sort of hard reminder to fix this before merging the destination branch to develop

Created #6324

whymarrh · 2019-03-14T02:24:14Z

77eb21a adds the ability to start Ganache from options (which includes an existing db path) during e2e tests. The Ganache wrapper (i.e., helper functions) hasn't been fleshed out yet since I don't have a good sense of what its API should be—that'll be something that comes out of #6157 as well. What we can do is interact with Ganache's RPC endpoint on some port.

danjm · 2019-03-18T15:58:45Z

app/scripts/lib/network-store.js

Why does this do nothing but return an empty promise?

Because it's ephemeral—during the lifetime of the app we don't need to bother saving the state that's set here

danjm · 2019-03-19T23:56:07Z

ui/app/store.js

Should this be treated as true in all cases?

Yup, removed. Good catch, this was for testing.

danjm · 2019-03-19T23:56:54Z

ui/index.js

Do we definitely want 'debug' in all cases?

Yup, removed. Good catch, this was for testing.

danjm · 2019-03-19T23:58:29Z

app/scripts/lib/network-store.js

Can 'http://localhost:12345/state.json' be specified dynamically, from an env variable or at least stored in a descriptive constant?

I'll extract this into a constant, good point. I think I'll leave this hard-coded though as this is something that we need to share and keep the same between the app (which could be pre-built into dist/) and the e2e logic. It's possible to have it dynamically set but I feel like this is easy enough to start with and should we need to we can make it dynamic later.

danjm · 2019-03-19T23:59:47Z

e2e/fixtures/index.js

Can these properties be specified dynamically, from an env variable or at least stored in a descriptive constant?

Constants for these as well

danjm · 2019-03-20T00:09:35Z

e2e/index.js

Why do we need the reference counter? Is it to handle multiple simultaneous calls of the fixture server, such that only one instance of the server can ever be running at a time?

Yup. We can only have (and only need) the single server on port 12345 so we want to make sure it's only started once and that it's shutdown when the tests end. I'm open to better ways of doing this.

danjm · 2019-03-20T00:16:43Z

e2e/index.js

Can we give load a slightly more unique name so that it is easy to grep for its uses in the codebase?

That's... a really good suggestion. I'll do withFixtures?

Update eth-contract-metadata dependency and sub-dependencies

The constants are mirrored between the e2e logic (`e2e/`) and the app logic in the ReadOnlyNetworkStore (`app/scripts/lib/network-store.js`) as they need to be the same in both places. These could eventually be made dynamic but the current build process isn't kind to that idea (in that the app is compiled and run separately from the e2e tests, which aren't compiled at all).

whymarrh requested review from chikeichan, danjm and tmashuang February 27, 2019 17:47

danjm self-assigned this Feb 27, 2019

danjm reviewed Feb 28, 2019

View reviewed changes

kumavis mentioned this pull request Mar 12, 2019

Build - unify build process #6276

Closed

4 tasks

whymarrh force-pushed the e2e-load-fixtures branch 4 times, most recently from a651560 to a206ded Compare March 13, 2019 16:48

whymarrh commented Mar 13, 2019

View reviewed changes

whymarrh force-pushed the e2e-load-fixtures branch from a206ded to 77eb21a Compare March 14, 2019 02:19

whymarrh marked this pull request as ready for review March 14, 2019 02:25

whymarrh and others added 3 commits March 14, 2019 12:52

Centre the notification in the current window (MetaMask#6307)

ea34d3b

Fixes popups not showing when screen size is odd (MetaMask#6312)

98699fb

Fix typos in English messages (MetaMask#6317)

00c52d5

danjm suggested changes Mar 20, 2019

View reviewed changes

whymarrh force-pushed the e2e-load-fixtures branch from 4963f2a to 7799da2 Compare March 20, 2019 11:31

whymarrh mentioned this pull request Mar 20, 2019

Add environment variable indicating E2E tests mode #6324

Closed

Add rollback script, move auto-changelog script (MetaMask#6252)

8230071

danjm approved these changes Mar 20, 2019

View reviewed changes

whymarrh added 6 commits March 20, 2019 17:01

Bump gaba version to avoid broken eth-contract-metadata

a702ade

Merge pull request MetaMask#6327 from MetaMask/eth-contract-metadata

fc534b8

Update eth-contract-metadata dependency and sub-dependencies

Add a ReadOnlyNetworkStore for pulling state from the network

cbbba64

Load background state fixtures in e2e tests

b29f5eb

Load Ganache state with options during e2e tests

0815ae5

Reject Ganache#quit if the server is not running

b2d6359

whymarrh added 2 commits March 20, 2019 17:38

Rename load as withFixtures for e2e test fixtures

38b19bc

whymarrh force-pushed the e2e-load-fixtures branch from 7799da2 to 1ab9cb5 Compare March 20, 2019 20:11

whymarrh merged commit 5dae079 into MetaMask:e2e-2.0 Mar 21, 2019

whymarrh deleted the e2e-load-fixtures branch March 21, 2019 14:46

This was referenced Apr 2, 2019

Allow loading of ganache, background and redux states on e2e test runs #6154

Closed

Add initial e2e specs using new test infrastructure #6387

Closed

whymarrh mentioned this pull request Jul 18, 2019

End-to-end tests "2.0" #6873

Closed

Gudahtt mentioned this pull request Jan 6, 2021

Provide metamask specific abstractions to our e2e tests, creating a library that speeds up e2e test writing work #6159

Closed

	// read from disk
	// first from preferred, async API:
	versionedData = (await localStore.get()) \|\|
	diskStore.getState() \|\|
	migrator.generateInitialState(firstTimeState)

	async function createModifiedTestBuild ({ browser, srcPath }) {
	// copy build to test-builds directory
	const extPath = path.resolve(`test-builds/${browser}`)
	await fs.ensureDir(extPath)
	await fs.copy(srcPath, extPath)
	// inject METAMASK_TEST_CONFIG setting default test network
	const config = { NetworkController: { provider: { type: 'localhost' } } }
	await prependFile(`${extPath}/background.js`, `window.METAMASK_TEST_CONFIG=${JSON.stringify(config)};\n`)
	return { extPath }
	}

Ability to load fixture state for e2e tests #6215

Ability to load fixture state for e2e tests #6215

Uh oh!

Conversation

whymarrh commented Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

The implementation

Background

Ganache

The state of this PR (tasks)

Testing this PR (no pun intended)

Alternative approaches to pre-loading background state

Editing the extension source files before loading it

Preloading the extension state before loading the extension

Uh oh!

kumavis commented Feb 27, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whymarrh commented Mar 13, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

whymarrh commented Mar 14, 2019

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

whymarrh commented Feb 27, 2019 •

edited

Loading