Ping/Pong exception in Depth websocket

[tomastoth]

Hello guys,

when I am opening an onDepthEvent websocket on many pairs(all btc pairs), I start getting this error

java.net.SocketTimeoutException: sent ping but didn't receive pong within 20000ms (after 0 successful ping/pongs)

When checking binance official websocket api which says:
The websocket server will send a ping frame every 3 minutes. If the websocket server does not receive a pong frame back from the connection within a 10 minute period, the connection will be disconnected. Unsolicited pong frames are allowed.

I though the Ping/Pong should be sent every 3 min?
When checking BinanceApiServiceGenerator the httpClient is created:
Dispatcher dispatcher = new Dispatcher(); dispatcher.setMaxRequestsPerHost(500); dispatcher.setMaxRequests(500); sharedClient = new OkHttpClient.Builder() .dispatcher(dispatcher) .pingInterval(20, TimeUnit.SECONDS) .build();
Doesnt that mean its trying to ping and receive a pong from server within 20 seconds?
Maybe thats why the error is happening?

Thanks for any tips!

[tomastoth]

By the way, the websocket is receiving data, just the onFailure() gets triggered (I think after the first unsuccessful ping/pong)

[eugene-kamenev]

@tomastoth look at square/okhttp#2239
This issue states that we cannot use okhttp at the moment to support binance ping/pong fully. Client only does ping from its side which is fine, but binance is also sending ping to client and client should respond with pong back. Which I assume is not possible atm with okttp.

As a workaround for websockets, you may try this implementation:
https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/main/java/com/binance/BinanceWebSocketListener.java
Which is based on netty & async http client and can be integrated without pain.
You can easily implement ping from your side as well. It is only required if your socket connection keep silence for some timeframe which you can add to listener and implement reconnect yourself.
Also this listener implementation uses jackson correctly 😋

Hope this helps.

[joaopsilva]

That is not quite true @eugene-kamenev, see https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/internal/ws/RealWebSocket.java. It supports writing a pong frame to the server, it is just not exposed in the API (which makes sense imho, as it is a protocol frame, and does not make sense to expose it in the application layer).

What do you mean by the listener implementation not using Jackson correctly? Is it because ObjectMapper is not being cached? I will put a fix for that.

[joaopsilva]

@tomastoth, are you able to share a working example where I will be able to quickly reproduce the problem? The pingInterval is to check if the connection is still alive on Binance's side.

The memory issues should be fixed by dd121ac, please update your repo.

[eugene-kamenev]

@joaopsilva You are correct. Looks like okhttp should support what is required.. But only after I migrated to netty for websockets my app started using much less heap and websocket streaming never looses data. I am running app for months now and didnt experienced anything weird like before. Dont know what could cause this.. But I assume issue sits at okhttp side..
@joaopsilva Yes, jackson mapper should be cached (singleton).

[tomastoth]

@joaopsilva this is the full stack trace
java.net.SocketTimeoutException: sent ping but didn't receive pong within 20000ms (after 0 successful ping/pongs) at okhttp3.internal.ws.RealWebSocket.writePingFrame(RealWebSocket.java:546) at okhttp3.internal.ws.RealWebSocket$PingRunnable.run(RealWebSocket.java:530) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset$$$capture(FutureTask.java:308) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
For example today, it kept running without a problem and after 600 ping/pongs i received a SocketTimeoutException. It restarted and now I keep getting this exception again. Its like the Binance doesnt accept depth socket request from me anymore or something

[tomastoth]

@tomastoth look at square/okhttp#2239
This issue states that we cannot use okhttp at the moment to support binance ping/pong fully. Client only does ping from its side which is fine, but binance is also sending ping to client and client should respond with pong back. Which I assume is not possible atm with okttp.

As a workaround for websockets, you may try this implementation:
https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/main/java/com/binance/BinanceWebSocketListener.java
Which is based on netty & async http client and can be integrated without pain.
You can easily implement ping from your side as well. It is only required if your socket connection keep silence for some timeframe which you can add to listener and implement reconnect yourself.
Also this listener implementation uses jackson correctly 😋

Hope this helps.

I am kinda new to okhttp and asyncwebsockets, can you please integrate your new type of websocket into the api and share it with us others so we can use it? I find it kinda hard implementing the websocket from your code. Thanks

[eugene-kamenev]

@tomastoth Have you checked at tests folder? https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/test/groovy/com/binance/WebSocketTestSpec.groovy
I will try to find spare time this weekend to provide PR.

[tomastoth]

@tomastoth Have you checked at tests folder? https://github.com/eugene-kamenev/binance-better-websocket/blob/master/src/test/groovy/com/binance/WebSocketTestSpec.groovy
I will try to find spare time this weekend to provide PR.

Okay, I have implemented it the same way. I am opening 123 different threads each one has a socket open like yours and I am getting this
java.util.concurrent.ExecutionException: java.net.ConnectException: handshake timed out
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
at org.asynchttpclient.netty.NettyResponseFuture.get(NettyResponseFuture.java:201)
at com.tomastoth.new_orderbook_saver.SaveToDb.startWebsocket(SaveToDb.java:74)
at com.tomastoth.new_orderbook_saver.SaveToDb.run(SaveToDb.java:50)
Caused by: java.net.ConnectException: handshake timed out
at org.asynchttpclient.netty.channel.NettyConnectListener.onFailure(NettyConnectListener.java:179)
at org.asynchttpclient.netty.channel.NettyConnectListener$1.onFailure(NettyConnectListener.java:151)
at org.asynchttpclient.netty.SimpleFutureListener.operationComplete(SimpleFutureListener.java:26)
at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:511)
at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:504)
at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:483)
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:424)
at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:121)
at io.netty.handler.ssl.SslHandler$6.run(SslHandler.java:1778)
at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38)
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:127)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute$$$capture(AbstractEventExecutor.java:163)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:446)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLException: handshake timed out

Maybe its bad to have so many threads?

[eugene-kamenev]

@tomastoth You should not use many threads and connections.. It is enough to have single EventLoopGroup with multiple threads and asynchttpclient instance, and it should serve all your connections.. You should apply multi-pair subscription through url (like in tests I provided). Then you can serve many connections throught single event loop group.
Remember that you should not block any incoming event, as I can see from your stacktrace you are trying to communicate with db from event handler.. This is also wrong, you should collect all incoming events into something like LinkedBlockingQueue, then call queue.drainTo at another thread which will perfom your communication with db and other things you need. Otherwise you are blocking event handler which is a bad practice..

[tomastoth]

@tomastoth You should not use many threads and connections.. It is enough to have single EventLoopGroup with multiple threads and asynchttpclient instance, and it should serve all your connections.. You should apply multi-pair subscription through url (like in tests I provided). Then you can serve many connections throught single event loop group.
Remember that you should not block any incoming event, as I can see from your stacktrace you are trying to communicate with db from event handler.. This is also wrong, you should collect all incoming events into something like LinkedBlockingQueue, then call queue.drainTo at another thread which will perfom your communication with db and other things you need. Otherwise you are blocking event handler which is a bad practice..

I havent done async programming before so its kinda new for me. So you think that is the reason I am getting the exception?

[sf202020sf]

has this issue been fixed? I get a couple of these error these days: Error in Websocket sent ping but didn't receive pong within 20000ms (after 13 successful ping/pongs)