RCP-240731A: Improvements to consensus ordering of state processing

[dr-orlovsky]

authors: @dr-orlovsky, @zoedberg
layers: consensus
breaking: consensus, api, serialization

Background

RGB contracts have global state; which is assembled from the contributions of individual operations (genesis, state transitions and state extensions). Since schema defines a maximum number of global state items of each type, the ordering of this processing affects what is reported to the clients. The global state is also accessible via AluVM op codes (CnC, LdC) and can be used in validating consequent operations. Thus, the order in which operations are processed during the validation and state computing affects consensus and validity of future operations. This order is named "consensus ordering" (of operations or global state).

At this moment the ordering happens according to the following ruleset:

• genesis always goes first;
• we order the rest of operations basing on the mining height of a witness bitcoin transaction committing to the operation data;
• if the operation is not mined (i.e. its witness transaction is in mempool, RBF, lightning channel), it is ordered after all mined operations;
• state extensions, which doesn't have direct witness transaction are ordered using witness transaction of a first state transition which closes a single-use seal defined in the extension; and they always are processed before that state transition;
• if two or more witness transactions are mined in the same block, they are ordered by a lexicographic ordering of their txids;
• we always use block timestamp and not block height, such that we can support multiple layer 1 (bitcoin, liquid) with different block production times (and incomparable heights);
• for multiple off chain transactions (mempool, LN channel) we use a dedicated parameter priority provided as a part of their off chain status, such that HTLCs can be indicated to be processed after the commitment transaction.

Motivation

The provided ruleset has one inconsistency: in lightning channels we order HTLCs after commitment using a dedicated off chain priority field, which will not be reflected if these transactions get mined, when they will get re-ordered basing on their txids, such that there is no guarantee that the state of HTLCs will not come before the commitment transaction (which may render it invalid)

Proposal

It is proposed to:

• remove priority field from off chain transaction ordering;
• add new nonce field to state transactions and state transitions, such their operation id commits to it
• use that nonce for ordering operations with witnesses mined in the same block
• the value of nonce must be a 64-bit integer
• add operation type as a factor of ordering, assigning its higher priority than nonce.

This will allow to order graphs of off chain transactions in a consistent way, preserved even when they are mined.

Rationale

• 64-bit nonce. The max number of eltoo channel updates is 2^48 - 1. Since all updates are added to the graph, plus we need to order HTLCs (and we may have thousands of them), we need 64-bit nonce. Even though not of all of commitment transactions would be published, some may, and all of them must be ordered appropriately.
• operation type used in ordering. If we have two different state transitions of different mined into the same block, which (by mistake) have the same nonce, we still can order them using type. This allows usage of type id value as a way of setting strong ordering overriding nonce values (for instance token issues can be always ordered before spendings).

[zoedberg]

I'm not sure about this. First of all the statement:

there is no guarantee that the state of HTLCs will not come before the commitment transaction (which may render it invalid)

state transitions are explicit in which are the RGB inputs and associated UTXOs so I don't see how HTLCs could come before the commitment transactions.

Then, about this statement:

This will allow to order graphs of off chain transactions in a consistent way,

I'm not sure we actually need ordering offchain transactions. All commitment TXs will be spending the funding TX and RGB should not care about the order of these commitment TXs, it should be agnostic about the LN.
Moreover as we discussed in RGB-WG/rgb-std#238 the optimal solution for LN would be to avoid consuming the fascia for every channel update, since that would make the stash grow a lot over time. Initially you proposed to go with the suboptimal solution (i.e. consuming the fascia at every channel update) because it seemed faster to implement, but now it seems it's requiring a lot of work too. So I would reconsider the idea and instead implement the optimal solution (i.e. deterministically generate state transitions matching previous state). With that in place all the discussion about offchain TXs ordering should become irrelevant.

[dr-orlovsky]

state transitions are explicit in which are the RGB inputs and associated UTXOs so I don't see how HTLCs could come before the commitment transactions.

basing on the current RGB Core code :)

I'm not sure we actually need ordering offchain transactions.

Consensus uses this information, so it is needed. Otherwise the system won't work and will be buggy/attackable. This is not a question

it should be agnostic about the LN.

yes, it is agnostic. It has nothing to do with LN specifically, just any off chain transaction graph.

With that in place all the discussion about offchain TXs ordering should become irrelevant.

This is unrelated. It is relevant notwithstanding anything we do with specific LN details.

[zoedberg]

basing on the current RGB Core code :)

What do you mean?

Consensus uses this information, so it is needed. Otherwise the system won't work and will be buggy/attackable. This is not a question

IMO it would be useful if you shared more information on this, what attack/bug would be possible?

[dr-orlovsky]

What do you mean?

I mean that RGB Core uses consensus ordering inside AluVM and for organizing global state reported to the clients. It is part of the existing code base.

IMO it would be useful if you shared more information on this, what attack/bug would be possible?

I gave a full description in the proposal, "Motivation" section. Once mined, the ordering of the state processing may change, leading to invalid contracts which were already accepted (or resulting from an uncooperatively-closed channels)

[dr-orlovsky]

Consensus ordering was introduced in RGB-WG/rgb-core#117 (those days global state was named metadata) and also described in the glossary https://github.com/orgs/RGB-WG/discussions/52 (see "consensus order").

[zoedberg]

I think there's been a misundestanding. Let's recap the conversation about HTLC ordering:

there is no guarantee that the state of HTLCs will not come before the commitment transaction (which may render it invalid)

state transitions are explicit in which are the RGB inputs and associated UTXOs so I don't see how HTLCs could come before the commitment transactions.

basing on the current RGB Core code :)

What do you mean?

I mean that RGB Core uses consensus ordering inside AluVM and for organizing global state reported to the clients. It is part of the existing code base.

From this discussion it seems RGB core is already able to order transactions, therefore I fail to understand the "Motivation" part of this RFC.

I understand ordering of onchain transactions is not the same of ordering offchain transactions, where you cannot use the height to order TXs. But 1) this could happen also onchain, think about CPFP and 2) it's always possible to order TXs following the inputs/outputs chain (in other words TXs can be ordered by which inputs are spending which outputs). So to me it's impossible that HTLC TX can "come before the commitment transaction" since the input of the HTLC TX is one of the UTXOs created by the commitment TX.

Once mined, the ordering of the state processing may change, leading to invalid contracts which were already accepted (or resulting from an uncooperatively-closed channels)

As just explained, I don't see how the order may change. But maybe I'm failing to understand what you're trying to say. Please try to explain this again, giving more details

[dr-orlovsky]

From this discussion it seems RGB core is already able to order transactions, therefore I fail to understand the "Motivation" part of this RFC.

It is not. It has no idea of bitcoin transaction ordering, and should not has. Otherwise it has to be re-designed embedding (partially) Bitcoin Core in itself.

That is what I say: the current RGB Core code has no idea of bitcoin transaction ordering. Since there is no 1-to-1 relation between bitcoin transaction and state transitions, even if it had, it wouldn't help.

So the issue is the state transitions ordering. And without the proposed consensus rules there is no way how to order them to match the lightning channel structure (and some other cases, so this is lightning-agnostic problem, just highlighted by lightning case)

[dr-orlovsky]

I understand ordering of onchain transactions

You constantly confuse bitcoin transactions with RGB state transitions. They are not matching each other, not 1-to-1.

[zoedberg]

I think I understood the issue. In RGB-WG/rgb-core#117 you say:

the order of state transition follows the bitcoin consensus ordering of witness transaction, i.e. using block height and position in the block of the witness transaction

therefore RGB doesn't order onchain TXs based on inputs/outputs relations but based on TX height and TX position in the block, which means that it's impossible with current code to order offchain transactions, since they don't have a height nor position in the block.

Now I understand the necessity to find a solution to order offchain TXs.

In the nonce solution, though, I'm worried that we could have issues in restoring the used nonce in case of re-construction of a previous state (e.g. in LN when an attacker publishes a TX associated to an old channel update). But maybe I didn't understand the proposal. Could you please expand this part:

add new nonce field to state transactions and state transitions, such their operation id commits to it

?

Would the nonce be a random number or a number the dev will be able to set?
How would that work in case multiple transitions need to be "committed to" (spent)?

[dr-orlovsky]

therefore RGB doesn't order onchain TXs based on inputs/outputs relations but based on TX height and TX position in the block,

Correct. The problem is that bitcoin transactions and RGB state transitions have many-to-many relationship, and one can't deterministically/unambiguously reconstruct an ordering for RGB state transitions just from how blockchain transactions are ordered - not even touching the case of mempool and state channels. And RGB state transitions are DAG, meaning that there is no linear order which can be always build from the DAG itself (which is a 2D net). We can't use a flow of seals (both in RGB transitions and bitcoin transactions) due to this reason: you have a DAG of seals and no linear ordering is possible. Using witness transaction ordering is the only thing which may work, but still we need something on top of it to solve ambiguity (since there is no bitcoin ordering for witness transactions in the same block).

Now I understand the necessity to find a solution to order offchain TXs.

Without that solution contract has no defined state, and all contract state introspections op-codes (reading global state) are impossible - and without those op codes and global state no DeFi apps can be made with RGB.

In the nonce solution, though, I'm worried that we could have issues in restoring the used nonce in case of re-construction of a previous state (e.g. in LN when an attacker publishes a TX associated to an old channel update).

The nonce becomes part of the state transitions, it is there - no need to restore anything:

add new nonce field to state transactions and state transitions, such their operation id commits to it

Would the nonce be a random number or a number the dev will be able to set?

Consensus doesn't put any requirements on the value of the nonce other than it is used for the ordering. I doubt any application can use random numbers to do the ordering :)

How would that work in case multiple transitions need to be "committed to" (spent)?

I do not understand the way you use the terminology. How you relate commitments and spends??

[dr-orlovsky]

Maybe it would be easier to read how the ordering works from the code itself: https://github.com/RGB-WG/rgb-core/blob/d06c627c846e9111a940aa84a8d5a670317d1f40/src/vm/contract.rs#L364-L404

since this structure has automatic Ord derive, the moment you put RGB operations into a BTreeMap<OpOrd, OpRef> you get consensus ordering automatically. The order of the enum variants and fields inside them is the order in which the values are compared by rust - as described in the doc comment.

The nonce value is taken from the operations: https://github.com/RGB-WG/rgb-core/blob/d06c627c846e9111a940aa84a8d5a670317d1f40/src/operation/operations.rs#L229-L231; each operation has nonce in its own data: https://github.com/RGB-WG/rgb-core/blob/d06c627c846e9111a940aa84a8d5a670317d1f40/src/operation/operations.rs#L385 (nonce of genesis is always u8::MAX: https://github.com/RGB-WG/rgb-core/blob/d06c627c846e9111a940aa84a8d5a670317d1f40/src/operation/operations.rs#L520-L521)

Operation ids commit to the nonce value alongside all other data they have, so the nonce can't be changed: https://github.com/RGB-WG/rgb-core/blob/d06c627c846e9111a940aa84a8d5a670317d1f40/src/operation/commit.rs#L310

[zoedberg]

I don't have the time to look at the code right now, I will do it on Monday. I'm not sure about this though:

The nonce becomes part of the state transitions, it is there - no need to restore anything:

it might become an issue in the way we think LN should work. In LN we would like to be able to avoid saving a bunch of data to reconstruct the RGB information to re-color previous commitment TXs. The desired workflow would be that we just save the RGB amounts that were assigned to the commitment TX vouts and then deterministically recreate the Fascia that was alredy created in the past but not stored nor consumed. This way we would reduce the space required for each channel update at a minimum. If we are not able to reconstruct the nonce deterministically then I'm afraid we will not be able to achieve our desired behavior.