client/rpcserver: Add orderbook route. by JoeGruffins · Pull Request #496 · decred/dcrdex

JoeGruffins · 2020-06-17T09:14:29Z

Resolves #466

JoeGruffins · 2020-06-17T09:15:26Z

-	Epoch    uint64  `json:"epoch"`
-	Sell     bool    `json:"sell"`
-	Token    string  `json:"token"`
-	MarketID string  `json:"marketID"`


I do not think MarketID was being used.

JoeGruffins · 2020-06-17T09:20:17Z

+			return nil, fmt.Errorf("unable to sync book: %v", err)
+		}
+		bookFeed.Close()
+		return book, nil


This changes the behavior of this function quite a bit, but I cannot see that it has caused any problems. Initially I wasn't sure how to fetch a one-time book, and I was considering more costly options, but if there is already a subscription to the book, this is very cheap. Otherwise making a subscription and canceling is the only way I can see to get market orders unless another route is added to the server.

That's pretty much the way for now, although we've been planning on a http api for this kind of thing.

JoeGruffins · 2020-06-17T09:21:17Z

+          "token" (string): The first 8 bytes of the order id, coded in hex.
+        },...
+      ],
+    }`,


Please double check my terminology.

chappjc · 2020-06-17T13:11:39Z

+			return nil, fmt.Errorf("unable to sync book: %v", err)
+		}
+		bookFeed.Close()
+		return book, nil


That's pretty much the way for now, although we've been planning on a http api for this kind of thing.

chappjc · 2020-06-17T13:12:28Z

+	book, err := s.core.Book(form.Host, form.Base, form.Quote)
+	if err != nil {
+		errMsg := fmt.Sprintf("unable to retrieve order book: %v", err)
+		resErr := msgjson.NewError(msgjson.RPCOrderBookError, errMsg)
+		return createResponse(orderBookRoute, nil, resErr)
+	}
+	return createResponse(orderBookRoute, &book, nil)


Practically speaking, an option to return the top N orders for both sides would be helpful.

JoeGruffins · 2020-06-18T07:39:56Z

I added an ability to truncate orders. I thought it made sense to combine epoch with buy and sells, and just add a field that indicates whether or not that order is an epoch order, and sort those as well.

buck54321 · 2020-06-30T13:31:38Z

+	Rate      float64 `json:"rate"`
+	Sell      bool    `json:"sell"`
+	Token     string  `json:"token"`
+	Immediate bool    `json:"immediate"`


An order in an order book will always be immediate = false. Why not just use the core.MiniOrder type here?

buck54321 · 2020-06-30T13:31:59Z

+}
+
+// OrderBook represents an order book, which are sorted buys and sells.
+type OrderBook struct {


Why not just use core.OrderBook?

buck54321 · 2020-06-30T13:37:17Z

+// appropriate fields and sorting buys and sells by descending and ascending
+// rates respectively.


A core.OrderBook as returned from Book is already sorted, and the only new field you're adding, Immediate, is always false for booked orders. I'm fairly certain you can drop this conversion and the rpcserver.OrderBook type altogether.

JoeGruffins · 2020-07-01T07:38:18Z

@buck54321 It's been a while since writing this, so my memory is a little fuzzy, but core.Book will return three books. Buys, sells, and epoch. Is it correct that the epoch orders are those that are not booked by the server, and those have both buys and sells? If so, for the purpose of the rpcserver, I thought it would be much cleaner to return just two books, sorted and truncated, rather than two sorted and truncated and one mixed, unsorted, and not truncated, hence the need for an extra field that identifies epoch orders as immediate.

buck54321 · 2020-07-01T11:57:32Z

for the purpose of the rpcserver, I thought it would be much cleaner to return just two books, sorted and truncated, rather than two sorted and truncated and one mixed, unsorted, and not truncated

I was initially doing something like this for the GUI, but it's not great, since epoch orders are not actually on the book and may not be available for matching. It may seem cleaner, but its very misleading and not an accurate representation of the market. That's why the Epoch queue was added.

If there are details missing from the core types, feel free to modify as you see fit. Might be worth implementing a new core type for epoch orders, something like

type EpochOrder struct {
	MiniOrder
	Immediate bool `json:"immediate"`
}

Note the absence of a field identifying market orders. My current method is to check Rate = 0 to identify market orders in the epoch, but I could also see adding a field for the order type.

Notably, this line

groupFn(coreBook.Epoch, true)

is not correct, since not all epoch orders are time-in-force immediate. Some are regular limit orders that just haven't reached the book yet. In order to track the time-in-force, you may need to modify the epochOrder type in client/order to retain that information.

JoeGruffins · 2020-07-02T03:07:15Z

I'm not sure what to do with "epoch" orders now. Will look into what exactly they are for a bit...

JoeGruffins

After conversing with @buck54321 I think it is best to return epoch orders as-is for now. I'm not sure what is the best way to portray this information, but I think it is necessary to return the entire stack in order to give a clear representation of the epoch "pile" (my term). Really I think the user will need more information to make use of it, such as which orders are cancel orders and which are market orders. I think another endpoint, order that accepts a token and returns info about an order is at least needed.

Or, you could make the argument that this enpoint is orderbook so epoch returns (which are not booked) don't necessary belong here, and deserve their own endpoint epochorders. But I think this current form is ok for now.

JoeGruffins · 2020-07-04T02:49:21Z

 	Qty   float64 `json:"qty"`
 	Rate  float64 `json:"rate"`
-	Epoch uint64  `json:"epoch"`
+	Epoch *uint64 `json:"epoch,omitempty"`


This subtle change may affect the webserver. @buck54321 looks ok?

I expect you'll need to set it here too:

dcrdex/client/webserver/live_test.go

Line 389 in 57a5085

Payload: &core.MiniOrder{Token: tkn},

But I'd suggest running both live_test and some manual simnet tests with the browser dev console opened to watch for unset field errors.

Also in client/core.handleUnbookOrderMsg:

dcrdex/client/core/bookie.go

Line 334 in 57a5085

Payload: &MiniOrder{Token: token(note.OrderID)},

Looks like these updates didn't happen, including in translateBookSide. Ref #571 (review)
Or were the remaining places the cases where we wanted this field omitted?

buck54321

Good stuff.

chappjc · 2020-07-09T17:55:48Z

+func truncateOrderBook(book *core.OrderBook, nOrders uint64) {
+	truncFn := func(orders *[]*core.MiniOrder) {
+		if uint64(len(*orders)) < nOrders {
+			return
+		}
+		*orders = (*orders)[:nOrders]
+	}
+	truncFn(&book.Buys)
+	truncFn(&book.Sells)
+}


Personally I feel this would be more readable and arguably more Go-idiomatic without the indirection:

func truncateOrderBook(book *core.OrderBook, nOrders uint64) { truncFn := func(orders []*core.MiniOrder) []*core.MiniOrder { if uint64(len(orders)) > nOrders { orders = orders[:nOrders] } return orders } book.Buys = truncFn(book.Buys) book.Sells = truncFn(book.Sells) }

Also note that the underlying buffer remains, so the elements (pointers) are not eligible for garage collection so this is a memory leak. However, given the transient nature of this result in handleOrderBook, it's not a big deal. However, if the function is used elsewhere, the truncated elements will not be garbage collected. So if we want to be meticulous we'd loop from nOrders to the end of the slice setting each element to nil before truncating it.

chappjc

Looking really good now, but a couple really quick changes to the test coverage would be nice. Thanks!

In order to give more easily undersood data, remove the Epoch MiniOrders. Instead add a field that indicates whether this order is an epoch order or not. Unexport fields of internal types in types.go.

JoeGruffins commented Jun 17, 2020

View reviewed changes

JoeGruffins marked this pull request as ready for review June 17, 2020 09:21

chappjc requested changes Jun 17, 2020

View reviewed changes

chappjc added the client label Jun 18, 2020

dnldd reviewed Jun 29, 2020

View reviewed changes

Comment thread client/core/bookie.go Outdated

buck54321 reviewed Jun 30, 2020

View reviewed changes

buck54321 mentioned this pull request Jul 3, 2020

spec: maxfeerate, revoke_order, preimage response ack, feedless orderbook #526

Closed

JoeGruffins force-pushed the orders branch from d51fe0f to 20553a9 Compare July 4, 2020 02:48

JoeGruffins commented Jul 4, 2020

View reviewed changes

buck54321 approved these changes Jul 6, 2020

View reviewed changes

Comment thread client/rpcserver/handlers.go Outdated

chappjc requested changes Jul 9, 2020

View reviewed changes

JoeGruffins force-pushed the orders branch from 6130baf to 57a5085 Compare July 14, 2020 05:20

chappjc requested changes Jul 14, 2020

View reviewed changes

Comment thread client/rpcserver/handlers.go

Comment thread client/rpcserver/handlers_test.go

Comment thread client/rpcserver/handlers_test.go

JoeGruffins added 6 commits July 15, 2020 17:26

client/rpcserver: Add orderbook route.

25686d8

Combine epoch orders with buys and sells.

7383eb6

In order to give more easily undersood data, remove the Epoch MiniOrders. Instead add a field that indicates whether this order is an epoch order or not. Unexport fields of internal types in types.go.

Return epoch orders as is.

b388e35

Fix grammar.

130247b

Fix grammar.

b98cfae

Refactor truncateOrderBook.

ef311c7

JoeGruffins force-pushed the orders branch from 57a5085 to 0a45f2d Compare July 15, 2020 08:42

Complete tests.

0a45f2d

chappjc merged commit 3642389 into decred:master Jul 22, 2020

chappjc mentioned this pull request Jul 28, 2020

web: setEpoch should keep orders with no epoch property #571

Merged

		// appropriate fields and sorting buys and sells by descending and ascending
		// rates respectively.

Conversation

JoeGruffins commented Jun 17, 2020 • edited by chappjc Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoeGruffins commented Jun 18, 2020

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoeGruffins commented Jul 1, 2020

Uh oh!

buck54321 commented Jul 1, 2020

Uh oh!

JoeGruffins commented Jul 2, 2020

Uh oh!

JoeGruffins left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chappjc Jul 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

buck54321 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chappjc left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JoeGruffins commented Jun 17, 2020 •

edited by chappjc

Loading

chappjc Jul 28, 2020 •

edited

Loading