Pluggable Consensus Refactoring

[rphmeier]

Substrate is designed to have pluggable consensus, where the same core codebase can accommodate many different possible consensus engines, which provide block production and finality.

Consensus engines will have two components:

• A client component, which actually performs consensus work
• A runtime component, which manages authorities, epochs, and signals changes.

Consensus engines are identified by a unique type ConsensusEngineId: [u8; 4]. We should create a registry of these somewhere.

There are 3 relevant digest item types to consensus:

• PreRuntime(ConsensusEngineId, Vec<u8>) (Inherent digests #2372): the client component ensures that a block carries a PreRuntime digest, which contains information about the provenance of the block. To be interpreted by the runtime component. Examples of data a PreRuntime digest may contain are the author's public key or a VRF proof of slot ownership.
• Consensus(ConsensusEngineId, Vec<u8>): the client code for a given ConsensusEngineId will watch the header-chain for matching Consensus digests. It will attempt to interpret the data contained within as some kind of signal for change (of authority sets or similar). It is expected that the runtime component of the consensus engine would be the only one issues such digests.
• Seal(ConsensusEngineId, SealSignature): A seal that proves provenance of the block. These are applied after the runtime. Each consensus engine may provide up to one seal. These are verified before the import process. SealSignature is a generic type which would typically be Vec<u8> outside of runtimes and some known struct within (again, to avoid having specific cryptography usages taint the chain definition).

Outstanding issues:

Assumption of AuthorityId and AuthorityIdFor (#2802)

Assuming that the authority ID type is fixed or that it exists at all is wrong. All consensus engine crypto should be done internally. Caching of authority set changes for light clients should be done via the light client cache.

• The type AuthorityId on Digest should be removed.
• The DigestItem::AuthoritiesChange should be phased out in favor of Consensus digests.
• The substrate-consensus::ImportQueue and Verifier interfaces should be changed so that items can be written to the light client cache rather than returning Vec<AuthorityIdFor>

killing srml-consensus in favor of consensus-specific modules (#2802)

It's too early to be trying to unify the behavior of all consensus engines on the runtime side. We're generally opting for consensus engines to have a specific runtime component which knows how and when to signal things to the client code.

Fork choice

cc @gnunicorn

Different engines will have different ways of choosing which fork is the best. There are three situations where we need to update our "best" block:

• Upon import of a new block.
• Upon finalization of a new block (see Reorg best chain on finality if diverged #1442)
• Upon external stimulus (generally not needed, but for Polkadot's consensus we have custom voting rules where we'll receive erasure-coded pieces that drive our fork choice)

It should not be the responsibility of the Client to manage fork-choice, although we would like any metadata necessary to drive a fork-choice to be present in a unified database with the client (ideally architected in an efficient manner).

Fork choice rules are currently implemented with the SelectChain abstraction.

• Take SelectChain parameter in client when importing a block
• Take SelectChain parameter in client when finalizing a block
• Make a global SelectChain parameter available to all consensus engines when doing a hybrid set up.