System Test Suite

[joepetrowski]

The Vision

History

We have always had a mix of tools for testing system parachains prior to releases. However, these have always had major problems or tradeoffs, for example:

• Requiring beefy hardware to run multi-chain dev networks;
• Not using the actual production state;
• Hard-to-read Yaml, JS, macro syntaxes.

In Parity's latest releases, prior to handing over the runtimes to the Fellowship, we used a combination of Rust-based XCM-Emulator tests and JavaScript integration tests. The Emulator tests were meant to run in CI and are easy to work with for testing XCM interactions. The JS tests served as a final check prior to publishing a release, but had their own weaknesses too.

Acala has a tool called Chopsticks that fetches the on-chain state and starts a development network based on it. This is quite powerful and should be part of our testing future.

Motivation

The objective of this effort is to come up with a test suite such that the Fellowship would feel comfortable whitelisting a runtime upgrade with the tested upgrades. There are a few design constraints to consider:

1. Anyone should be able to run the test suite.
2. Parity's polkadot-sdk repo shall not depend on the runtimes maintained by the Fellowship.
3. The tools are well-documented and can be used by parachain developers to write their own tests.
4. When a test fails, tools should support debugging.
5. Tests should be runnable in CI such that failures can be detected in PRs and not at the moment of a release.

Tools

I envision a mix of both Emulator and Chopsticks.

Emulator is useful for things like ensuring that XCM programs will work properly when executed in the context of each chain. The tests run in a single process and allow for easy debugging.

• xcm-emulator #50

Chopsticks is better suited to ensuring that runtime upgrades will work given real network conditions (for example, that a migration that iterates through some keys will not go overweight).

Both of these tools will need some modifications to be ideal for these purposes. They are both opinionated toward the people who made them. We should make them more general to work for these objectives. IMO, Parity should contribute directly to Chopsticks.

Test Coverage

We should have a set of tests related strictly to the Polkadot system (Relay Chain plus system paras). This set should include expected interactions with the chains from non-system chains.

Of course, any parachain should be able to write tests specific to their own chain. However, I would not think that these tests failing would prevent a system upgrade. Of course, they should be open for discussion and perhaps there is a reasonable solution.

The Plan

Plan to plan:

1. Decide what test coverage we need.
2. Decide the best tools to achieve it - probably Chopsticks and Emulator.
3. Decide on repo architecture to facilitate anyone running it.

Currently, starting this effort is @joepetrowski, @0xmovses, and @NachoPal. Would be happy to have more people involved in designing this test suite (cc @ggwpez, @bkchr).

This should become a Fellowship RFC once we have a more detailed plan.

[0xmovses]

Some questions and thoughts from my side.

1. Good CLI interface is key here, cargo-nextest is great for this.

2. As Chopsticks and XCM Emulator are written in different languages (Typescript, Rust), it makes sense initially to keep them in separate repositories.

   • Ideally however, we would want to bring them together so one testing suite could support all the functionality a developer might need, forking mainnet state, XCM unit tests, Chopstick parachain integration tests. Aiming for something like Foundry comes to mind.
   • To jump between repositories (and languages) in order to run a whole test suite seems cumbersome.
   • It might be possible to use Assembly Script somewhere within Chopsticks to build a wasm binary from the Chopstick test files. XCM Emulator also would build a wasm binary...

3. If we wanted one power-testing tool with all of this functionality out-of-the-box it might be best to iterate towards that gradually, so that any blockers or unknowns would not stall this project entirely. So, breaking the work into larger sub-issues and starting to submit PRs to the respective repositories, before considering how we might create one TS / Rust testing-monorepo, if we wanted to do that. These could be sub-issues could be:

   • Update Chopsticks to make functionality more generalised to be used across the network.
   • Add Debugging Info to Chopsticks
   • Improve CLI for Chopsticks.
   • Create XCM Emulator Repo, add Polkadot System Tests.
   • Improve CLI for Emulator (clap?)
   • Use cargo-nextest

4. I think we have decided to keep maintaining Chopsticks in Typescript. But it's interesting to consider the upside of a Rust rewrite. Would a rewrite to rust help testing for all developers across the network? Would this lift be worth it?

Pros:

• One rust monorepo for all testing tools, two binaries XCM-Emulator, Chopsticks-Rusty
• Speed
• Infra portability
• Ease of CLI development and general maintainability.

Cons:

• Most polkadot devs will probably want to write tests in TS.
• Removes the option for TS for polkadot devs to write in TS

5. The Polkadot System Tests (rust, XCM Emulator Tests), should build as a binary which any developer can easily run locally and on their own CI infra. The purpose of this repo is to provide the tooling and framework to write new XCM Emulator tests for their project, but also to run the default polkadot-system-tests out of the box. They get the tooling and the essential tests in one bin.

[xlc]

A reminder that we already have this https://github.com/AcalaNetwork/e2e-tests/

Also I can see it could be helpful to rewrite Chopsticks with Rust, it is not really relevant at all. If you want to write the tests with Rust, use subxt.

But happy to see extra contributions to Chopsticks anyway. For example, the core executor is currently running in wasm by nodejs which could be changed to native (while maintaining wasm compatibility for browser execution) for better performance.

[NachoPal]

I tried to address point 3 from The Plan (Decide on repo architecture to facilitate anyone running it) in regards xcm-emulator tests.

Here I created a POC of a new repository where everybody (not only System Parachains team) can declare their chains to be used by themselves and others for their networks declarations. Once a network is defined, teams can add their integrations tests.

Some blockers/issues I encountered:

• I had to address this issue: [xcm-emulator] Allow Chains and Network imports from integration-tests-common crate polkadot-sdk#1383 to be able to declare chains in a different crate the network is declared. This unlocks single chain declaration for sharing it among different networks.
• There was not an easy way of updating workspace crate dependencies to follow specific versions. To solve it I opened this PR. This is essential to be able to easily align the repo with whatever crate versions fellows/runtimes is using.
• Some crates were not being released (E.g. xcm-emulator). Talked to @Morganamilo to release them.
  • integration-test-common still needs to be released, but some previous work is needed to remove runtimes dependencies from the crate. Because of these runtimes dependencies, I couldn't import yet runtimes from fellows/runtimes as it was causing conflicts.

Some design decisions:

• Each network integration-tests are in their own workspace. The reason for it is to allow teams to follow their own updating pace without blocking others. The only pieces that should always be up to date (members of the repo's workspace) are chais and networks as they could be potentially used by others.

[0xmovses]

Chopsticks Upgrade Proposal

I looked at what updates would be required to implement these type of e2e tests in a Chopsticks repo.

1. Networks to be an updatable struct in memory.

• Currently the way to configure networks is in Chopsticks is to write your own network.ts file within the networks/ dir. At the moment all of these are then fed into a const all, this is too opinionated for general usage. Instead initialize the networks object, populate it with network config data either upon init, or upon test(s) teardown from a json / yml file.

2. Refactor query api.

• Currently all methods for querying are in the form of queryTokenBalance, queryBalance, instead we should have a Query object with associated methods, the object can have its own state which will be useful for the tests. And this provides a common api.

3. Create exhaustive list of query helper methods, with TS Docs.

• This will allow devs to get the full functionality of polkadot-js with docs built in their IDE. The current list of query methods is there only for what Acala needed at the time.

4. Refactor extrinsics api.

• similarly to query, the extrinsic helper functions currently have many hardcoded values and are opinionated about args, network config etc. All of this can be more generalised with a ExtrinsicCall object, a builder (or the TS equivalent) pattern for initialising values, and again some state that can be mutated throughout the test.

Keeping it light for now, but these initial changes will move the repo into a place where its more user friendly for developers.

[NachoPal]

@0xmovses Chopsticks tests api and how Chopsticks deploys a local testnet is tightly coupled? Or it would be possible to deploy a testnet with Chopsticks and use a different interface to interact with the testnet?

[xlc]

@0xmovses

Networks to be an updatable struct in memory.

Can you give some use cases of be able to dynamically add networks? The current approach ensures the type checker can catch typo of network names.

Anyway we can refactor it into two layers, one that's super flexible that works for all the scenarios and on top of it I can recreate the current interface to keep the static assertions.

Refactor query api.
Create exhaustive list of query helper methods, with TS Docs.
Refactor extrinsics api.

Yeah those will be nice. Just make sure don't over-engineering it. What we have currently already supports many parachains so they are generic enough and not just for Acala.

@NachoPal as I stated before, Chopsticks implemented the standard Substrate RPC and bunch more dev RPCs that you can easily invoke them with polkadot.js or whatever library you want. So no you don't have to use the code in our e2e repo. You can write the test in Rust or whatever suits you the best. The benefits of using our e2e tests is that it is already implemented and maybe it will save you some time from reinvent the wheel. There are many considerations that you may have not aware of when developing test suites against live networks, which is a moving target and we have taken care most of them. For example, you don't want to hardcode the weights and fees in tests because they change all the time but you still want to assert them don't change too much unexpectedly.

[bkchr]

I had to address this issue:
paritytech/polkadot-sdk#1383 to be able to declare chains in a different crate the network is declared. This unlocks single chain declaration for sharing it among different networks.
There was not an easy way of updating workspace crate dependencies to follow specific versions. To solve it I opened paritytech/diener#42. This is essential to be able to easily align the repo with whatever crate versions fellows/runtimes is using.

This is one of the main reasons I don't want to have tests in Rust. Yes Rust is nice and we all love it. However, for what we want to achieve here, it is not flexible enough and you will need to write too much code. I mean having different runtime versions with different types and whatever. We should just reuse the tooling that already exists there.

What I would like to see is that runtimes are not represented as repos/src files/whatever and instead are just wasm files. Runtimes compiled at a certain point. Just working against wasm files will also make a lot of things more easy. You can just drop them and rerun the test etc. Wasm files also give you the opportunity to write test across the entire ecosystem to ensure that you can send a XCM from statemint to acala to whatever.

I also think these tests should be as easy as possible to write. This means they should be in typescript, not using any magic YAML configs that make your eyes bleed when you read it or whatever kind of magic languages exists out there.

If chopsticks can give you RPC access to a wasm file, you should be able to write quite a lot of tests before you hit any blocker.

[xlc]

Chopsticks supports run devnet from a chain spec genesis file