Skip to content

spec: maxfeerate, revoke_order, preimage response ack, feedless orderbook #526

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
buck54321 wants to merge 4 commits into from
Closed

spec: maxfeerate, revoke_order, preimage response ack, feedless orderbook #526

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d24bbc3
maxfeerate, revoke_order, preimage response ack, feedless orderbook
buck54321 Jul 3, 2020
15baa3c
generalize acks language. link to match negotiation in maxfeerate desc.
buck54321 Jul 14, 2020
55ce6f2
add unmatched orders header
buck54321 Jul 29, 2020
8ed901f
add dialogue-type message
buck54321 Aug 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion dex/msgjson/types.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ type Stampable interface {
// requires an acknowledgement. It is typically a signature of some serialized
// data associated with the request.
type Acknowledgement struct {
MatchID Bytes `json:"matchid"`
MatchID Bytes `json:"matchid"` // TODO: Remove this field.
Sig Bytes `json:"sig"`
}

Expand Down
4 changes: 3 additions & 1 deletion spec/README.mediawiki
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ for the DEX API.
** [[comm.mediawiki/#Coin_ID|Coin ID]]
* [[comm.mediawiki/#Message_Protocol|Message Protocol]]
* [[comm.mediawiki/#Session_Authentication|Session Authentication]]
* [[comm.mediawiki/#Acknowledgements|Acknowledgements]]
* [[comm.mediawiki/#HTTP|HTTP]]

'''[2] [[fundamentals.mediawiki|Distributed Exchange Design Fundamentals]]'''
Expand Down Expand Up @@ -114,8 +115,9 @@ book and place orders.
** [[orders.mediawiki/#Cancel_Order|Cancel Order]]
* [[orders.mediawiki/#Preimage_Reveal|Preimage Handling]]
* [[orders.mediawiki/#Unmatched_Orders|Unmatched Orders]]
* [[orders.mediawiki/#Order_Revocation|Order Revocation]]
* [[orders.mediawiki/#Match_Negotiation|Match Negotiation]]
* [[orders.mediawiki/#Match_Revocation|Match Revocation]]
* [[orders.mediawiki/#Match_negotiation|Match Negotiation]]
* [[orders.mediawiki/#Trade_Suspension|Trade Suspension]]

'''[6] [[api.mediawiki| Data API]]''' defines http and WebSocket APIs to browse
Expand Down
74 changes: 64 additions & 10 deletions spec/comm.mediawiki
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -78,18 +78,20 @@ structure called a '''Message'''.
{|
! field !! type !! description
|-
| type || int || message type
| type || int || message type
|-
| payload || any || the data being transmitted
| payload || any || the data being transmitted
|-
| route || string || the route identifier. requests and notifications only
| route || string || the route identifier. requests and notifications only
|-
| id || int > 0 || the request ID. requests and responses only
| id || int > 0 || the message ID. not set for notifications.
|-
| returnid || int > 0 || the return ID. dialogue-type messages only
|}

There are three anticipated message types.

'''Message types'''
===Message types===

{|
| type || id || description
Expand All @@ -99,9 +101,11 @@ There are three anticipated message types.
| response || 2 || a response to a request
|-
| notification || 3 || usually part of a data feed. requires no response
|-
| dialogue || 4 || used for more complicated, multi-step data transactions
|}

'''Example request'''
===Request===

The payload for a request can be of any type.

Expand All @@ -114,7 +118,7 @@ The payload for a request can be of any type.
}
</pre>

'''Response payload'''
===Response Payload===

The payload for a response has a structure that enables quick error checking.

Expand All @@ -126,7 +130,7 @@ The payload for a response has a structure that enables quick error checking.
| error || string or null || the error. field is null or missing if no error was encountered
|}

'''Example response'''
===Response===

<pre>
{
Expand All @@ -136,7 +140,7 @@ The payload for a response has a structure that enables quick error checking.
}
</pre>

'''Example notification'''
===Notification===

<pre>
{
Expand All @@ -146,6 +150,43 @@ The payload for a response has a structure that enables quick error checking.
}
</pre>

===Dialogue===

A dialogue type message is initiated the same as a request. The initiator
will set the ID, which they will then use to identify dialogue messages from the
other party. The primary difference between a dialogue and a request is that the
non-initiator must set a `returnid` in their first response. Any subsequent
dialogue messages from the initiator will use this `returnid` as the message ID.

'''Dialogue initiation'''

<pre>
{
"type": 4,
"id": 100,
"route" "exchangeinfo",
"payload": {"info": "initiator_data_1"},
}
</pre>

'''Dialogue initiation response'''

<pre>
{
"type": 4,
"id": 100,
"returnid": 200,
"route" "exchangeinfo",
"payload": {"info"; "recipient_data_1"},
Copy link
Copy Markdown
Member

@chappjc chappjc Aug 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would all subsequent messages/responses not be using a ResponsePayload as the Message.Payload? This shows a free-form payload structure with no "response" or "error" keys. This could work if "error" is a possible key that the handler for this route would look for to detect an early dialog termination. More on this in my subsequent comments...

Copy link
Copy Markdown
Member Author

@buck54321 buck54321 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to leave it as general as possible, but maybe you're right that we should use ResponsePayload or similar because errors.

}
</pre>

The dialogue can then continue, with the initiator addressing messages to the
specified <code>returnid</code>, and the non-initiator addressing messages to
the original message ID. The <code>returnid</code> is only expected to be set in
the first non-initiator message. The initiator will continue to use that ID for
all subsequent messaging.
Comment on lines +186 to +188
Copy link
Copy Markdown
Member

@chappjc chappjc Aug 20, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my comment below, s/non-initiator/participant/, but we could do that in the spec too if you like.

My understanding of these lines is reflected by slightly different language, but I think we are saying the same thing:

A "returnid" field is only present in the participant's dialog initialization response. In all subsequent messages, the initiator will use the value of that field for the "id" field in dialog messages to the participant, while the participant will continue to use the initiator's "id" from the initialization request.

Spelled out with some implementation hints, what I think we are describing is this:

  • The dialog initialization request from the initiator would have "id": x. Before sending the initialization request, the initiator registers some dialog-scoped data or a handler function for the dialog keyed by x.
  • The participant gets the request and uses "route" (and "type": 4) to find the associated global handler to start this dialog.
  • The dialog initialization response from the participant has "id": x and "returnid": y. Before sending this message, the participant registers some dialog-scoped data or a handler keyed by y for further messages for this dialog, storing the value of x with this data/handler for dialog y.
  • Initiator gets the initialization response with "id": x, finds the data/handler for they dialog keyed by x, and stores the value of y just provided in "returnid".
  • Now both parties have some dialog-scoped data or a handler in a map keyed by their own unique id that stores the other party's unique id. The dialog is established.
  • Subsequent messages from the initiator have "id": y so that the participant can find the dialog.
  • Subsequent messages from the participant have "id": x so that the initiator can find the dialog.

Copy link
Copy Markdown
Member

@chappjc chappjc Aug 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The relevance of the "route" field in all dialog messages depends on the implementation. Options alluded to above:

  • a request-response approach where the initial message from each party creates a new handler for the dialog, not needing the route except for the handling of the dialog initialization request by the participant's global request handler
  • an "all requests" approach where the global request handler for the specified route is always used, but this global handler attempts to locate dialog-scoped data from some map specific to this route.

I actually like the second option, but hacking on this might change my mind.

Copy link
Copy Markdown
Member Author

@buck54321 buck54321 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your implementation description is spot on, and the "all requests" approach is the one I was picturing, but like you said, let the hacking flesh it out. No need to rush this out of draft either. Might be time to just do it, and implement the solution for preimage errors (#597), and then come back to this.

Copy link
Copy Markdown
Member

@chappjc chappjc Aug 25, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May also be a good candidate for the match request, so that the client can get confirmation that their ack was recorded before starting their 'init' request. ref: #565 (comment) Perhaps I'm seeing a logic race where there isn't one, but it seems like an OK is needed back from the server before the maker does its 'init' request, otherwise the server could be not ready.

But there are a host of server-originating requests that may be best off as dialogs to allow the server to respond with an error to the client's response.

Comment on lines +186 to +188
Copy link
Copy Markdown
Member

@chappjc chappjc Aug 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thinking about termination of the dialog, there would presumably just be a known dialog sequence for this particular route, which could terminate early with an error or run to completion.

Copy link
Copy Markdown
Member Author

@buck54321 buck54321 Aug 25, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep. The specification for the route will define any required sub-sequencing requirements, but really it'll probably just be that both sides remember what step they're at.


==Session Authentication==

Many DEX messages must be sent on an authenticated connection. Once a WebSocket
Expand Down Expand Up @@ -189,13 +230,26 @@ of their penalization.
{|
! field !! type !! description
|-
| matches || &#91;object&#93; || list of [[orders.mediawiki/#Match_negotiation|Match objects]]
| matches || &#91;object&#93; || list of [[orders.mediawiki/#Match_Negotiation|Match objects]]
|-
| penalty || object || a [[community.mediawiki/#Penalty_Object|Penalty Object]]. omitted if in good standing
|-
| sig || string || hex-encoded server's signature of the serialized connection data
|}

==Acknowledgements==

Often, message only requires a simple signed acknowledgement. The specification
will provide instructions for serialization of the message data. The recipient
will then sign the serialized data with their private key, and send an
acknowledgement of the following form.

{|
! field !! type !! description
|-
| sig || string || hex-encoded signature of the notification data
|}

==HTTP==

An API using HTTP for message transport may be provided for basic account
Expand Down
16 changes: 8 additions & 8 deletions spec/fundamentals.mediawiki
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,21 +112,21 @@ respond with its current configuration.
'''Asset object'''

{|
! field !! type !! description
! field !! type !! description
|-
| symbol || string || ticker symbol
| symbol || string || ticker symbol
|-
| id || int || a unique per-asset ID
| id || int || a unique per-asset ID
|-
| lotsize || int || lot size (atoms)
| lotsize || int || lot size (atoms)
|-
| ratestep || int || the price rate increment (atoms)
| ratestep || int || the price rate increment (atoms)
|-
| feerate || int || the fee rate for transactions (atoms/byte)
| maxfeerate || int || the max on-chain transaction fee rate [[orders.mediawiki/#Match_Negotiation|that can be assigned to matches]] (atoms/byte)
|-
| swapsize || int || the size of the initialization transaction (bytes)
| swapsize || int || the size of the initialization transaction (bytes)
|-
| swapconf || int || minimum confirmations for swap transactions
| swapconf || int || minimum confirmations for swap transactions
|}

Comment on lines +129 to 131
Copy link
Copy Markdown
Member

@chappjc chappjc Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After this object definition, can we link to Match Negotiation in orders.mediawiki, perhaps also saying that the maxfeerate is applied to the swap contract transaction as described in that section, as "assigned to matches" is a little vague.

'''Market object'''
Expand Down
99 changes: 73 additions & 26 deletions spec/orders.mediawiki
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,11 @@ An order book can be viewed and tracked by subscribing to a market.
{|
! field !! type !! description
|-
| base || string || currency code for the market's base asset
| base || string || currency code for the market's base asset
|-
| quote || string || currency code for the market's quote asset
| quote || string || currency code for the market's quote asset
|-
| nofeed || bool || If set to true, client will not receive order book updates
|}

The response will contain the complete market order book.
Expand Down Expand Up @@ -552,19 +554,19 @@ This is by design and discourages certain types of spoofing.

==Preimage Reveal==

At the expiration of the epoch, the DEX sends out a <code>preimage</code>
request for each order in the epoch queue. The match cycle begins 5 seconds
At the expiration of the epoch, the DEX initiate a <code>preimage</code>
dialogue for each order in the epoch queue. The match cycle begins 5 seconds
after the last <code>preimage</code> request is sent by the server, so clients
must respond before then.

A '''''commitment checksum''''' is included as part of the
<code>preimage</code> request.
<code>preimage</code> dialogue initiation.
The checksum is the Blake-256 hash of the concatenated, lexicographically-sorted
commitments for every order in the epoch. For clients subscribed to the order
book for the entire duration of the epoch, the checksum can be validated against
the checksum generated from their local copy of the epoch queue.

'''Request route:''' <code>preimage</code>, '''originator:''' DEX
'''Dialogue route:''' <code>preimage</code>, '''originator:''' DEX

<code>payload</code>
{|
Expand All @@ -575,7 +577,7 @@ the checksum generated from their local copy of the epoch queue.
| csum || string || the commitment checksum
|}

'''Preimage response'''
'''Dialogue step 2: preimage reveal''', '''originator:''' client

<code>result</code>
{|
Expand All @@ -584,27 +586,59 @@ the checksum generated from their local copy of the epoch queue.
| pimg || string || hex-encoded preimage for the order's commitment
|}

==Match negotiation==
The server will send an [[comm.mediawiki/#Acknowledgements|acknowledgement]] of
their receipt of the preimage.

Swap negotiation details will be relayed through the DEX with a series of
notifications.
Both the DEX and the clients will need to serialize and sign the notification
data. The originator includes their signature with the request, while the
recipient will return an '''acknowledgement''', or a list of
acknowledgements, as the <code>result</code> of their response payload.
==Order Revocation==

In a couple of situations, the server will revoke a client's orders.

'''Acknowledgement'''
1. A client in violation of the rules of community conduct may have their existing orders revoked. In this case, the revocation will count against the client's cancellation statistics.

2. If a client does not reconnect after a trade suspension with <code>persist = true</code>, the client's orders will be revoked. This revocation does not count against the client's cancellation statistics.

The server will send a <code>revoke_order</code> request to the client when an
order is revoked.

'''Request route:''' <code>revoke_order</code>, '''originator:''' DEX

<code>payload</code>
{|
! field !! type !! description
! field !! type !! description
|-
| orderid || string || order ID
|-
| matchid || string || the match ID
| reason || string || the reason the order is being revoked
|-
| sig || string || hex-encoded signature of the notification data
| sig || string || DEX's hex-encoded signature of the serialized notification data. serialization described below
|}

'''Order revocation serialization'''

{|
! field !! size (bytes) !! description
|-
| orderid || 32 || the order ID
|-
| reason || varies || the UTF-8 encoded reason
|}

The client will respond with an
[[comm.mediawiki/#Acknowledgements|acknowledgement]].
Comment on lines +626 to +627
Copy link
Copy Markdown
Member

@chappjc chappjc Jul 7, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm also not sure if this ack serves a real purpose either. What would the server do with the client's ack? Much like with revoke_match, the server does nothing with the ack presently, just logging it.

Copy link
Copy Markdown
Member Author

@buck54321 buck54321 Jul 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You okay with removing both acknowledgements and making these notifications instead?


==Match Negotiation==

Swap negotiation details will be relayed through the DEX with a series of
notifications. Both the DEX and the clients will need to serialize and sign the
notification data. The originator includes their signature with the request,
while the recipient will return an
[[comm.mediawiki/#Acknowledgements|acknowledgement]], or a list of
acknowledgements, as the <code>result</code> of their response payload.

If the client's order has one or more matches at the end of a match cycle, the
DEX will send a list of '''match objects'''. The maker is the first to act, so
DEX will send a list of '''match objects'''.

The maker is the first to act, so
after sending their acknowledgement, they should broadcast their initialization
transaction and inform the server with an <code>init</code> notification
(described after).
Expand All @@ -631,6 +665,10 @@ transaction and inform the server with an <code>init</code> notification
|-
| status || int || only provided in 'connect' response. For 'match' requests, status is 0 = 'MakerSwapCast'. See [[https://github.com/decred/dcrdex/blob/master/dex/order/match.go|match.go]] for codes.
|-
| feeratebase || int || the fee rate assigned to the swap transaction broadcast on the base asset blockchain. units: atoms/byte
|-
| feeratequote || int || the fee rate assigned to the swap transaction broadcast on the quote asset blockchain. units: atoms/byte
|-
| sig || string || DEX's hex-encoded signature of the serialized notification data. serialization described below
|}

Expand All @@ -652,15 +690,22 @@ transaction and inform the server with an <code>init</code> notification
| address || varies || UTF-8 encoded receiving address for the match
|}

The client will respond with a list of signed match acknowledgements.

'''The <code>tserver</code> value is used as the basis for the the locktimes.'''
If it is necessary to convert the time to seconds, the value should be rounded
down.

The client will respond with a list of signed match acknowledgements.
The match object includes fields specifying the required transaction fee rates
for both blockchains. These rates will be at or below the
<code>maxfeerate</code> specified in the
[[fundamentals.mediawiki/#Configuration_Data_Request|asset configuration]]. The
actual rates comes from the server asset backends, so the algorithm is
implementation-specific.

After a client broadcasts their initialization transaction, they are
expected to report the transaction details to the server for verification and
relay to the matching party.
After a client broadcasts their initialization transaction, they will report the
transaction details to the server for verification. The server will relay the
details to the counter-party.

'''Request route:''' <code>init</code>, '''originator:''' client

Expand All @@ -670,7 +715,7 @@ relay to the matching party.
|-
| orderid || string || the order ID
|-
| matchid || string || the matchid, retrieved from the [[#Match_negotiation|match notification]]
| matchid || string || the matchid, retrieved from the [[#Match_Negotiation|match notification]]
|-
| coinid || string || hex-encoded coin ID
|-
Expand Down Expand Up @@ -816,6 +861,8 @@ The client will respond with an acknowledgement.
The taker will get the key from the maker's redemption and broadcast their own
redemption transaction.

==Unmatched Orders==

It is also possible for an epoch order to go through the matching cycle without
generating a match. This will be common for limit orders, but can also occur for
market orders if there are no booked orders to match with. When the server fails
Expand All @@ -833,7 +880,7 @@ client.

==Match Revocation==

A match can be revoked by the server if a client fails to act within the
A match can be revoked by the server if one party fails to act within the
[[fundamentals.mediawiki/#Exchange_Variables|broadcast timeout]]. A match revocation will result in
penalties for the violating party only.
The revoked match quantity is not added back to the order book in any form.
Expand All @@ -851,7 +898,7 @@ The revoked match quantity is not added back to the order book in any form.
| sig || string || DEX's hex-encoded signature of serialized revocation. serialization described below
|}

'''Revocation serialization'''
'''Match revocation serialization'''

{|
! field !! size (bytes) !! description
Expand Down