From 4b50346b0ff4abfc15fa9cdc7b3ca8824f76c7c6 Mon Sep 17 00:00:00 2001 From: Bastien Teinturier Date: Mon, 9 Dec 2019 11:19:40 +0100 Subject: [PATCH 1/5] Bolt 1: Specify that extensions to existing messages must use TLV The spec already prepared a hook to add additional information to existing messages (additional bytes at the end of a message must be ignored). Since we're using TLV in many places, it would make sense to use that optional additional space at the end of each message to allow an optional tlv stream. We keep the "additional bytes must be ignored" rule for what happens after that tlv stream, giving us extra flexibility to add another type of optional content to existing messages later if needed. --- 01-messaging.md | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/01-messaging.md b/01-messaging.md index 40d1909d0..5ab01ee73 100644 --- a/01-messaging.md +++ b/01-messaging.md @@ -22,6 +22,7 @@ All data fields are unsigned big-endian unless otherwise specified. * [The `ping` and `pong` Messages](#the-ping-and-pong-messages) * [Appendix A: BigSize Test Vectors](#appendix-a-bigsize-test-vectors) * [Appendix B: Type-Length-Value Test Vectors](#appendix-b-type-length-value-test-vectors) + * [Appendix C: Message Extension](#appendix-c-message-extension) * [Acknowledgments](#acknowledgments) * [References](#references) * [Authors](#authors) @@ -37,6 +38,7 @@ After decryption, all Lightning messages are of the form: 1. `type`: a 2-byte big-endian field indicating the type of message 2. `payload`: a variable-length payload that comprises the remainder of the message and that conforms to a format matching the `type` +3. `extension`: an optional [TLV stream](#type-length-value-format) The `type` field indicates how to interpret the `payload` field. The format for each individual type is defined by a specification in this repository. @@ -44,12 +46,17 @@ The type follows the _it's ok to be odd_ rule, so nodes MAY send _odd_-numbered A sending node: - MUST NOT send an evenly-typed message not listed here without prior negotiation. + - MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation. + - If it doesn't include `extension` fields: + - MUST omit the `extension` TLV stream entirely. A receiving node: - upon receiving a message of _odd_, unknown type: - MUST ignore the received message. - upon receiving a message of _even_, unknown type: - MUST fail the channels. + - upon receiving a message with an `extension` containing an _even_, unknown type: + - MUST fail the channels. The messages are grouped logically into five groups, ordered by the most significant bit that is set: @@ -62,9 +69,11 @@ The messages are grouped logically into five groups, ordered by the most signifi The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes. A node: - - MUST ignore any additional data within a message beyond the length that it expects for that type. + - MUST ignore any additional data within a message beyond the `extension`. - upon receiving a known message with insufficient length for the contents: - MUST fail the channels. + - upon receiving an invalid `extension`: + - MUST fail the channels. - that negotiates an option in this specification: - MUST include all the fields annotated with that option. - When defining custom messages: @@ -88,6 +97,9 @@ The _it's ok to be odd_ rule allows for future optional extensions without negotiation or special coding in clients. The "ignore additional data" rule similarly allows for future expansion. +The _extension_ field lets senders include additional TLV data by leveraging +the "ignore additional data" rule. + Implementations may prefer to have message data aligned on an 8-byte boundary (the largest natural alignment requirement of any type here); however, adding a 6-byte padding after the type field was considered @@ -879,6 +891,25 @@ failure: 1. Invalid stream: 0xffffffffffffffffff 00 00 00 2. Reason: valid TLV records but invalid ordering +## Appendix C: Message Extension + +This section contains examples of valid and invalid extensions on the `init` +message. The base `init` message (without extensions) for these examples is +`0x001000000000` (all features turned off). + +The following `init` messages are valid: + +- `0x001000000000`: no extension provided +- `0x0010000000000601012a030104`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`) +- `0x0010000000000601012a03010400000000`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`) and additional bytes that must be ignored + +The following `init` messages are invalid: + +- `0x00100000000000`: the extension contains 0 records (this is invalid because it would break the canonical encoding of the message) +- `0x0010000000002a`: the extension is present but truncated +- `0x0010000000000302012a`: the extension contains unknown _even_ TLV records +- `0x00100000000006010101010102`: the extension TLV stream is invalid (duplicate TLV record type `0x01`) + ## Acknowledgments [ TODO: (roasbeef); fin ] From 2f2d0f74f9e022cbf427ab33ba3df0fa1eb06e48 Mon Sep 17 00:00:00 2001 From: Bastien Teinturier Date: Tue, 10 Dec 2019 10:36:45 +0100 Subject: [PATCH 2/5] Remove TLV stream length-prefix. Clarify that extension bytes are included in signatures. --- 01-messaging.md | 23 +++++++++++------------ 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/01-messaging.md b/01-messaging.md index 5ab01ee73..1a05b24a7 100644 --- a/01-messaging.md +++ b/01-messaging.md @@ -69,7 +69,6 @@ The messages are grouped logically into five groups, ordered by the most signifi The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes. A node: - - MUST ignore any additional data within a message beyond the `extension`. - upon receiving a known message with insufficient length for the contents: - MUST fail the channels. - upon receiving an invalid `extension`: @@ -94,11 +93,9 @@ Length is limited to 65535 bytes by the cryptographic wrapping, and messages in the protocol are never more than that length anyway. The _it's ok to be odd_ rule allows for future optional extensions -without negotiation or special coding in clients. The "ignore -additional data" rule similarly allows for future expansion. - -The _extension_ field lets senders include additional TLV data by leveraging -the "ignore additional data" rule. +without negotiation or special coding in clients. The _extension_ field +similarly allows for future expansion by letting senders include additional +TLV data. Implementations may prefer to have message data aligned on an 8-byte boundary (the largest natural alignment requirement of any type here); @@ -900,15 +897,17 @@ message. The base `init` message (without extensions) for these examples is The following `init` messages are valid: - `0x001000000000`: no extension provided -- `0x0010000000000601012a030104`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`) -- `0x0010000000000601012a03010400000000`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`) and additional bytes that must be ignored +- `0x00100000000001012a030104`: the extension contains two _odd_ TLV records (with types `0x01` and `0x03`) The following `init` messages are invalid: -- `0x00100000000000`: the extension contains 0 records (this is invalid because it would break the canonical encoding of the message) -- `0x0010000000002a`: the extension is present but truncated -- `0x0010000000000302012a`: the extension contains unknown _even_ TLV records -- `0x00100000000006010101010102`: the extension TLV stream is invalid (duplicate TLV record type `0x01`) +- `0x00100000000001`: the extension is present but truncated +- `0x00100000000002012a`: the extension contains unknown _even_ TLV records (assuming that TLV type `0x02` is unknown) +- `0x001000000000010101010102`: the extension TLV stream is invalid (duplicate TLV record type `0x01`) + +Note that when messages are signed, the _extension_ is part of the signed bytes. +Nodes should store the _extension_ bytes even if they don't understand them to +be able to correctly verify signatures. ## Acknowledgments From 5c4ef38a3ab39641e8af9062b0459870ec9bdbfc Mon Sep 17 00:00:00 2001 From: Bastien Teinturier Date: Tue, 10 Dec 2019 11:09:56 +0100 Subject: [PATCH 3/5] Promote optional message fields to mandatory. This applies to: - option_upfront_shutdown_script: if you're not interested, just set the length to 0 - channel_reestablish commitment points: it makes sense to always include those regardless of whether `option_dataloss_protect` or `option_static_remotekey` are set No need to change the `channel_update`'s `htlc_maximum_msat` because the `message_flags` encode its presence/absence. It can still be either included or omitted without causing issues to the extension stream. --- 02-peer-protocol.md | 37 +++++++++++++++---------------------- 1 file changed, 15 insertions(+), 22 deletions(-) diff --git a/02-peer-protocol.md b/02-peer-protocol.md index 1f4f0a5ee..2b31b883b 100644 --- a/02-peer-protocol.md +++ b/02-peer-protocol.md @@ -123,8 +123,8 @@ the funding transaction and both versions of the commitment transaction. * [`point`:`htlc_basepoint`] * [`point`:`first_per_commitment_point`] * [`byte`:`channel_flags`] - * [`u16`:`shutdown_len`] (`option_upfront_shutdown_script`) - * [`shutdown_len*byte`:`shutdown_scriptpubkey`] (`option_upfront_shutdown_script`) + * [`u16`:`shutdown_len`] + * [`shutdown_len*byte`:`shutdown_scriptpubkey`] The `chain_hash` value denotes the exact blockchain that the opened channel will reside within. This is usually the genesis hash of the respective blockchain. @@ -198,10 +198,7 @@ The sending node: - MUST set `first_per_commitment_point` to the per-commitment point to be used for the initial commitment transaction, derived as specified in [BOLT #3](03-transactions.md#per-commitment-secret-requirements). - MUST set `channel_reserve_satoshis` greater than or equal to `dust_limit_satoshis`. - MUST set undefined bits in `channel_flags` to 0. - - if both nodes advertised the `option_upfront_shutdown_script` feature: - - MUST include either a valid `shutdown_scriptpubkey` as required by `shutdown` `scriptpubkey`, or a zero-length `shutdown_scriptpubkey`. - - otherwise: - - MAY include a`shutdown_scriptpubkey`. + - MUST include either a valid `shutdown_scriptpubkey` as required by `option_upfront_shutdown_script`, or a zero-length `shutdown_scriptpubkey`. The sending node SHOULD: - set `to_self_delay` sufficient to ensure the sender can irreversibly spend a commitment transaction output, in case of misbehavior by the receiver. @@ -284,8 +281,8 @@ funding transaction and both versions of the commitment transaction. * [`point`:`delayed_payment_basepoint`] * [`point`:`htlc_basepoint`] * [`point`:`first_per_commitment_point`] - * [`u16`:`shutdown_len`] (`option_upfront_shutdown_script`) - * [`shutdown_len*byte`:`shutdown_scriptpubkey`] (`option_upfront_shutdown_script`) + * [`u16`:`shutdown_len`] + * [`shutdown_len*byte`:`shutdown_scriptpubkey`] #### Requirements @@ -1120,8 +1117,8 @@ messages are), they are independent of requirements here. * [`channel_id`:`channel_id`] * [`u64`:`next_commitment_number`] * [`u64`:`next_revocation_number`] - * [`32*byte`:`your_last_per_commitment_secret`] (option_data_loss_protect,option_static_remotekey) - * [`point`:`my_current_per_commitment_point`] (option_data_loss_protect,option_static_remotekey) + * [`32*byte`:`your_last_per_commitment_secret`] + * [`point`:`my_current_per_commitment_point`] `next_commitment_number`: A commitment number is a 48-bit incrementing counter for each commitment transaction; counters @@ -1129,7 +1126,6 @@ are independent for each peer in the channel and start at 0. They're only explicitly relayed to the other node in the case of re-establishment, otherwise they are implicit. - ### Requirements A funding node: @@ -1171,16 +1167,14 @@ The sending node: next `revoke_and_ack` message it expects to receive. - if `option_static_remotekey` applies to the commitment transaction: - MUST set `my_current_per_commitment_point` to a valid point. - - otherwise, if it supports `option_data_loss_protect`: + - otherwise: - MUST set `my_current_per_commitment_point` to its commitment point for - the last signed commitment it received from its channel peer (i.e. the commitment_point + the last signed commitment it received from its channel peer (i.e. the commitment_point corresponding to the commitment transaction the sender would use to unilaterally close). - - if `option_static_remotekey` applies to the commitment transaction, or the sending node supports `option_data_loss_protect`: - - if `next_revocation_number` equals 0: - - MUST set `your_last_per_commitment_secret` to all zeroes - - otherwise: - - MUST set `your_last_per_commitment_secret` to the last `per_commitment_secret` - it received + - if `next_revocation_number` equals 0: + - MUST set `your_last_per_commitment_secret` to all zeroes + - otherwise: + - MUST set `your_last_per_commitment_secret` to the last `per_commitment_secret` it received A node: - if `next_commitment_number` is 1 in both the `channel_reestablish` it @@ -1221,10 +1215,9 @@ A node: - MUST NOT broadcast its commitment transaction. - SHOULD fail the channel. - otherwise: - - if `your_last_per_commitment_secret` does not match the expected values: + - if `your_last_per_commitment_secret` does not match the expected values: - SHOULD fail the channel. - - otherwise, if it supports `option_data_loss_protect`, AND the `option_data_loss_protect` - fields are present: + - otherwise, if it supports `option_data_loss_protect`: - if `next_revocation_number` is greater than expected above, AND `your_last_per_commitment_secret` is correct for that `next_revocation_number` minus 1: From d5a3832b208c1ed513aa09e0b76038f372187fea Mon Sep 17 00:00:00 2001 From: t-bast Date: Tue, 4 Feb 2020 13:31:38 +0100 Subject: [PATCH 4/5] Bolt1: clean-up post-IRC meeting @TheBlueMatt made a point that always parsing a TLV stream even when you don't expect anything is cumbersome. If you don't expect any extension, you may ignore the tlv stream. @rustyrussell noticed an unnecessary requirement now that TLV stream isn't length-prefixed. --- 01-messaging.md | 34 +++++++++++++++------------------- 1 file changed, 15 insertions(+), 19 deletions(-) diff --git a/01-messaging.md b/01-messaging.md index 1a05b24a7..e7812b6be 100644 --- a/01-messaging.md +++ b/01-messaging.md @@ -44,20 +44,6 @@ The `type` field indicates how to interpret the `payload` field. The format for each individual type is defined by a specification in this repository. The type follows the _it's ok to be odd_ rule, so nodes MAY send _odd_-numbered types without ascertaining that the recipient understands it. -A sending node: - - MUST NOT send an evenly-typed message not listed here without prior negotiation. - - MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation. - - If it doesn't include `extension` fields: - - MUST omit the `extension` TLV stream entirely. - -A receiving node: - - upon receiving a message of _odd_, unknown type: - - MUST ignore the received message. - - upon receiving a message of _even_, unknown type: - - MUST fail the channels. - - upon receiving a message with an `extension` containing an _even_, unknown type: - - MUST fail the channels. - The messages are grouped logically into five groups, ordered by the most significant bit that is set: - Setup & Control (types `0`-`31`): messages related to connection setup, control, supported features, and error reporting (described below) @@ -68,11 +54,9 @@ The messages are grouped logically into five groups, ordered by the most signifi The size of the message is required by the transport layer to fit into a 2-byte unsigned int; therefore, the maximum possible size is 65535 bytes. -A node: - - upon receiving a known message with insufficient length for the contents: - - MUST fail the channels. - - upon receiving an invalid `extension`: - - MUST fail the channels. +A sending node: + - MUST NOT send an evenly-typed message not listed here without prior negotiation. + - MUST NOT send evenly-typed TLV records in the `extension` without prior negotiation. - that negotiates an option in this specification: - MUST include all the fields annotated with that option. - When defining custom messages: @@ -83,6 +67,18 @@ A node: - SHOULD pick an even `type` identifiers when regular nodes should reject the message and close the connection. +A receiving node: + - upon receiving a message of _odd_, unknown type: + - MUST ignore the received message. + - upon receiving a message of _even_, unknown type: + - MUST fail the channels. + - upon receiving a known message with insufficient length for the contents: + - MUST fail the channels. + - upon receiving a message with an `extension`: + - MAY ignore the `extension`. + - Otherwise, if the `extension` is invalid: + - MUST fail the channels. + ### Rationale By default `SHA2` and Bitcoin public keys are both encoded as From db31eaf188ad2abdf5bd99996f8b8fffaec7cc12 Mon Sep 17 00:00:00 2001 From: t-bast Date: Tue, 4 Feb 2020 14:42:05 +0100 Subject: [PATCH 5/5] Move upfront_shutdown_script to TLV extension This makes more sense for future work. It gives us more room to change the encoding or remove it. --- 02-peer-protocol.md | 25 ++++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/02-peer-protocol.md b/02-peer-protocol.md index 2b31b883b..647649927 100644 --- a/02-peer-protocol.md +++ b/02-peer-protocol.md @@ -123,8 +123,13 @@ the funding transaction and both versions of the commitment transaction. * [`point`:`htlc_basepoint`] * [`point`:`first_per_commitment_point`] * [`byte`:`channel_flags`] - * [`u16`:`shutdown_len`] - * [`shutdown_len*byte`:`shutdown_scriptpubkey`] + * [`open_channel_tlvs`:`tlvs`] + +1. tlvs: `open_channel_tlvs` +2. types: + 1. type: 0 (`upfront_shutdown_script`) + 2. data: + * [`...*byte`:`shutdown_scriptpubkey`] The `chain_hash` value denotes the exact blockchain that the opened channel will reside within. This is usually the genesis hash of the respective blockchain. @@ -198,7 +203,12 @@ The sending node: - MUST set `first_per_commitment_point` to the per-commitment point to be used for the initial commitment transaction, derived as specified in [BOLT #3](03-transactions.md#per-commitment-secret-requirements). - MUST set `channel_reserve_satoshis` greater than or equal to `dust_limit_satoshis`. - MUST set undefined bits in `channel_flags` to 0. - - MUST include either a valid `shutdown_scriptpubkey` as required by `option_upfront_shutdown_script`, or a zero-length `shutdown_scriptpubkey`. + - if both nodes advertised the `option_upfront_shutdown_script` feature: + - MUST include `upfront_shutdown_script` with either a valid `shutdown_scriptpubkey` as required by `shutdown` `scriptpubkey`, or a zero-length `shutdown_scriptpubkey`. + - otherwise: + - MAY include `upfront_shutdown_script`. + - if it includes `open_channel_tlvs`: + - MUST include `upfront_shutdown_script`. The sending node SHOULD: - set `to_self_delay` sufficient to ensure the sender can irreversibly spend a commitment transaction output, in case of misbehavior by the receiver. @@ -281,8 +291,13 @@ funding transaction and both versions of the commitment transaction. * [`point`:`delayed_payment_basepoint`] * [`point`:`htlc_basepoint`] * [`point`:`first_per_commitment_point`] - * [`u16`:`shutdown_len`] - * [`shutdown_len*byte`:`shutdown_scriptpubkey`] + * [`accept_channel_tlvs`:`tlvs`] + +1. tlvs: `accept_channel_tlvs` +2. types: + 1. type: 0 (`upfront_shutdown_script`) + 2. data: + * [`...*byte`:`shutdown_scriptpubkey`] #### Requirements