subscribe_to_orderbook falling out of sync

[protny]

I slightly modified your example that keeps a live orderbook so that it runs indefinitely and continually outputs the top 5 bids and asks for a ticker (tested on BTC-ETH) in the on_orderbook function

the orderbook falls out of sync regularly, usually either a couple of bids or asks are missing as compared to bittrex orderbook view through their website, these get rolled away as the price changes so at other times everything seems to be ok. The worst case I saw was when all top 5 bids shown by the websocket client were non existent on bittrex webpage

I tried it with just one symbol to make sure this is not just the scripts being overwhelmed but this happens no matter how many orderbook symbols i subscribed to

[slazarov]

That’s odd, if the nounce difference between the last and the most recent message is different it would trigger a resync. Can you try comparing using an inspector instead of the what they show as the current orders. Sometimes there is a slowdown in the connection from the browser and an order passes during that and doesn’t get shown while it’s shown in the socket.

[slazarov]

Another solution is to set the order book depth to a higher number, I think it’s currently 10 or 20. Try with 100 and report back. Check the documentation to see how that is done. Thanks!

[protny]

tried both, it looks like the problem arises when enough order book entries are filled/deleted and there's not enough data points due to the orderbook depth to fill the freed up spots so they get filled in by new data points but there are other data points missing in between

when using the default depth the orderbook falls out of sync faster (5-10 minutes), when I set the order book depth to 9999 it took about 6-7 hours (probably also highly depends on the volatility of the pair)

Is there a way to hold the full orderbook?

I also looked at the inspector data, the nounces seemed to go in order, no holes, no delays and as far as i could see data from them was incorporated correctly, it really looked like the problem is always the missing order_book_depth+1 data point and the hole in data propagates to the top of the orderbook in time

what did seem strange though was that when I let run both orderbook and orderbook_update the nounces for the orderbook update messages were not the same as those of orderbook. The orderbook nounces where the same that I saw through the inspector. The orderbook_update nounces were higher, the timestamp was ok so this was no delay. For example

Tue Jan 30 16:23:36 2018 : orderbook_update msg = {'ticker': 'BTC-ETH', 'nounce': 1678116, 'timestamp': 1517325816.700767, 'bids': [{'Type': 1, 'Rate': 0.10682001, ' ....

Tue Jan 30 16:23:37 2018 : orderbook msg = {'MarketName': 'BTC-ETH', 'Nounce': 1685995, 'Buys': [{'Quantity': 0.89374148, 'Rate': 0.10745493}, {'Quantity': 0.103 ...... 'Rate': 0.11382648}], 'timestamp': 1517325817.122321}

[slazarov]

tried both, it looks like the problem arises when enough order book entries are filled/deleted and there's not enough data points due to the orderbook depth to fill the freed up spots so they get filled in by new data points but there are other data points missing in between

when using the default depth the orderbook falls out of sync faster (5-10 minutes), when I set the order book depth to 9999 it took about 6-7 hours (probably also highly depends on the volatility of the pair)

Is there a way to hold the full orderbook?

Yes, remove the index, i.e omit the [0:index] part in _sync_order_book. This is the part:

                if made_change:
                    # Sort by price, with respect to BUY(desc) or SELL(asc)
                    self.order_books[ticker][side[0]] = sorted(
                        self.order_books[ticker][side[0]],
                        key=lambda k: k['Rate'],
                        reverse=side[1])
                    # Put depth to 10
                    self.order_books[ticker][side[0]] = \
                        self.order_books[ticker][side[0]][
                        0:book_depth]
                    # Add nounce unix timestamp
                    self.order_books[ticker]['timestamp'] = time()

The main reason I put an index initially was that if we are syncing a lot of order books, the dict that holds the order book would get big and slow down performance. So perhaps we can switch to some other container?

I also looked at the inspector data, the nounces seemed to go in order, no holes, no delays and as far as i could see data from them was incorporated correctly, it really looked like the problem is always the missing order_book_depth+1 data point and the hole in data propagates to the top of the orderbook in time

Let's see how it goes without the index part.

what did seem strange though was that when I let run both orderbook and orderbook_update the nounces for the orderbook update messages were not the same as those of orderbook. The orderbook nounces where the same that I saw through the inspector. The orderbook_update nounces were higher, the timestamp was ok so this was no delay. For example

Tue Jan 30 16:23:36 2018 : orderbook_update msg = {'ticker': 'BTC-ETH', 'nounce': 1678116, 'timestamp': 1517325816.700767, 'bids': [{'Type': 1, 'Rate': 0.10682001, ' ....

Tue Jan 30 16:23:37 2018 : orderbook msg = {'MarketName': 'BTC-ETH', 'Nounce': 1685995, 'Buys': [{'Quantity': 0.89374148, 'Rate': 0.10745493}, {'Quantity': 0.103 ...... 'Rate': 0.11382648}], 'timestamp': 1517325817.122321}

This is not a bug. Most certainly orderbook and orderbook_update are made through connections from different threads. In general each connection gets a different nounce for the specific ticker. If you think about it, if the difference was more than 1, it would trigger a resync as per the code logic.

As a whole the code logic for the order book sync needs lots of improvements, the good thing is that I know what the problems are, the bad thing is that I lack the time :).

Anyways, thanks for helping and contributing!

[protny]

thanks a lot, removing the index helped now as long as I am getting data its in sync

now I am testing it by subscribing to all tickers and it seems to work fine for a while, not sure if this is connected to the performance dip because of holding more data in memory but after some times (usually hours) I do get an error message and the orderbook stops being updated

the error messages look like this:

Traceback (most recent call last):
File "D:\Python\lib\site-packages\gevent\greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "D:\Python\lib\site-packages\signalr_connection.py", line 53, in wrapped_listener
listener()
File "D:\Python\lib\site-packages\signalr\transports_ws_transport.py", line 42, in receive
for notification in self.ws:
File "D:\Python\lib\site-packages\websocket_core.py", line 107, in iter
yield self.recv()
File "D:\Python\lib\site-packages\websocket_core.py", line 296, in recv
opcode, data = self.recv_data()
File "D:\Python\lib\site-packages\websocket_core.py", line 313, in recv_data
opcode, frame = self.recv_data_frame(control_frame)
File "D:\Python\lib\site-packages\websocket_core.py", line 326, in recv_data_frame
frame = self.recv_frame()
File "D:\Python\lib\site-packages\websocket_core.py", line 360, in recv_frame
return self.frame_buffer.recv_frame()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 340, in recv_frame
self.recv_header()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 288, in recv_header
header = self.recv_strict(2)
File "D:\Python\lib\site-packages\websocket_abnf.py", line 375, in recv_strict
bytes = self.recv(min(16384, shortage))
File "D:\Python\lib\site-packages\websocket_core.py", line 430, in _recv
return recv(self.sock, bufsize)
File "D:\Python\lib\site-packages\websocket_socket.py", line 94, in recv
"Connection is already closed.")
websocket._exceptions.WebSocketConnectionClosedException: Connection is already closed.
Mon Feb 5 18:19:13 2018 <Greenlet at 0x2463301d5a0: wrapped_listener> failed with WebSocketConnectionClosedException

or this

Traceback (most recent call last):
File "D:\Python\lib\site-packages\gevent\greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "D:\Python\lib\site-packages\signalr_connection.py", line 53, in wrapped_listener
listener()
File "D:\Python\lib\site-packages\signalr\transports_ws_transport.py", line 42, in receive
for notification in self.ws:
File "D:\Python\lib\site-packages\websocket_core.py", line 107, in iter
yield self.recv()
File "D:\Python\lib\site-packages\websocket_core.py", line 296, in recv
opcode, data = self.recv_data()
File "D:\Python\lib\site-packages\websocket_core.py", line 313, in recv_data
opcode, frame = self.recv_data_frame(control_frame)
File "D:\Python\lib\site-packages\websocket_core.py", line 326, in recv_data_frame
frame = self.recv_frame()
File "D:\Python\lib\site-packages\websocket_core.py", line 360, in recv_frame
return self.frame_buffer.recv_frame()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 340, in recv_frame
self.recv_header()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 288, in recv_header
header = self.recv_strict(2)
File "D:\Python\lib\site-packages\websocket_abnf.py", line 375, in recv_strict
bytes = self.recv(min(16384, shortage))
File "D:\Python\lib\site-packages\websocket_core.py", line 430, in recv
return recv(self.sock, bufsize)
File "D:\Python\lib\site-packages\websocket_socket.py", line 81, in recv
bytes = sock.recv(bufsize)
File "D:\Python\lib\site-packages\gevent_ssl3.py", line 457, in recv
return self.read(buflen)
File "D:\Python\lib\site-packages\gevent_ssl3.py", line 293, in read
return self._sslobj.read(len or 1024)
BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately
Mon Feb 5 19:18:55 2018 <Greenlet at 0x2271c34d898: wrapped_listener> failed with BlockingIOError

or this

Exception in thread Thread-3:
Traceback (most recent call last):
File "D:\Eclipse\workspace\HuBot 3.0\src\bittrex_websocket\websocket_client.py", line 321, in _init_connection
conn.start()
File "D:\Python\lib\site-packages\signalr_connection.py", line 50, in start
listener = self.__transport.start()
File "D:\Python\lib\site-packages\signalr\transports_auto_transport.py", line 28, in start
return self.__transport.start()
File "D:\Python\lib\site-packages\signalr\transports_ws_transport.py", line 38, in start
enable_multithread=True)
File "D:\Python\lib\site-packages\websocket_core.py", line 490, in create_connection
websock.connect(url, **options)
File "D:\Python\lib\site-packages\websocket_core.py", line 216, in connect
self.handshake_response = handshake(self.sock, *addrs, **options)
File "D:\Python\lib\site-packages\websocket_handshake.py", line 69, in handshake
status, resp = _get_resp_headers(sock)
File "D:\Python\lib\site-packages\websocket_handshake.py", line 135, in _get_resp_headers
raise WebSocketBadStatusException("Handshake status %d", status)
websocket._exceptions.WebSocketBadStatusException: Handshake status 504

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "D:\Python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "D:\Eclipse\workspace\HuBot 3.0\src\bittrex_websocket\websocket_client.py", line 349, in _init_connection
raise ImportWarning(MSG_ERROR_SOCKET_WEBSOCKETBADSTATUS)
ImportWarning: Please report error to https://github.com/slazarov/python-bittrex-websocket, Error:WebSocketBadStatusException

instead of restarting it dies after these errors occur, I have not looked into them deeper yet but I plan to and will let you know what I find out

[emestee]

I am under impression that there is no deletion going on, each call to on_orderbook delivers a new copy of orderbook in the msg?

[slazarov]

No.

[emestee]

Apologies then, I'll go RTFS.

[emestee]

Follow up question, if you don't mind me using the ticket for this: I have my own implementation of orderbooks and code that builds on them. Which do you think would be a better strategy:

• Wrap your orderbooks in an interface that behaves like my orderbooks
• Override your methods and write to my orderbook instances instead
• Load the entire orderbook contents from your instances to mine once in a while?

[slazarov]

Hi, would be a better strategy for what :)?
In my opinion do whatever is easier in order to get something working. The best strategy if you want to contribute is to work together on the project and improve it. I don’t have a lot of free time to work on it constantly so I am happy to implement better solutions. Best!

[slazarov]

Closing due to inactivity.