sse client not receiving any messages.

[samta]

We are observing that sse client is not receiving any messages from the server. It just hangs unless we restart the client. As soon as the client has been restarted it starts receiving the messages.

[mpetazzoni]

Do you mean that it suddenly stops receiving any messages? Do you have dumps of the event stream sent by server? Is it possible that the client "chokes" on a particular event that triggers and bug and leads to it getting stuck?

[samta]

I don't see any issue in the server side logs, also the client side received messages are the regular message. There is no error as well it just that client connection is in ESTABLISHED state. Restarting server is not making the client to reconnect to the server at all.

[mpetazzoni]

The client doesn't automatically reconnect because that's not necessarily the behavior that the user wants. If you want that behavior, you can detect when the client.events() generator stops returning messages, reconnect and restart consuming events.

[samta]

But when server restart, client should reconnect to server isn't it?

Usually, when I restart server, sse client reconnects but sometimes it just freezes and doesn't reconnect.
netstat result says the connection is still ESTABLISH.
Server side connection is closed but the client side it still ESTABLISHED

Any thought?

[andrewt3000]

you can detect when the client.events() generator stops returning messages

Any recommendation for how to detect that the generators stops?

[mpetazzoni]

If the connection is severed, the client.events() generator stops and your loop would exit. Basic pattern would be something like this (in terrible pseudo-code):

while not really_done_with_it:
    client = connect_or_reconnect(last_received)
    for event in client.events():
        # deal with the event and return false as long as we expect more
        really_done_with_it = deal_with_it(event)

[andrewt3000]

Ok thanks for the psuedo-code... that is similar to the code I have.

I am interfacing with the nest api... I believe it's maintaining the connection but it stops sending the keep alive events and they expect me to break the connection. I suspect this is the way nest does it and not part of the specification. Although not sure....

It also sends a Connection: keep-alive header to indicate to the product to keep the connection open.

https://developers.nest.com/guides/api/rest-streaming-guide

[andrewt3000]

Would it make sense to add a feature to close a connection automatically if you don't get a keep alive after a certain period?

[mpetazzoni]

It could. Would you be willing to take a crack at it and send a PR?

[andrewt3000]

Idk, I am considering adding a feature where you have a configurable time out in seconds. If there are no chunks to read in the time out period the events() function will return.

Do you think I am on the right track?

[mpetazzoni]

Yes

[andrewt3000]

Ok... I have researched it and decided to try to set a read timeout in urllib3 request. Hopefully it will raise a timeout exception and my code will reset the connection. Will report back if that doesn't work.

[8633brown]

I think I'm having a similar problem with the same API.
My code seemed to be running fine for at least a year and a half, i updated python and its libraries, now it seems likee the event stream is all happening at once.

This is the best debug log i can pull right now as I'm unsure how to check the actual event stream to check it's correct.

2019-12-19 01:15:38,298, DEBUG: Starting new HTTPS connection (1): developer-api.nest.com:443
2019-12-19 01:15:38,854, DEBUG: https://developer-api.nest.com:443 "GET /devices/thermostats HTTP/1.1" 307 0
2019-12-19 01:15:38,856, DEBUG: Response code is 307
2019-12-19 01:15:38,861, DEBUG: Starting new HTTPS connection (1): firebase-apiserver38-tah01-iad01.dapi.production.nest.com:9553
2019-12-19 01:15:39,471, DEBUG: https://firebase-apiserver38-tah01-iad01.dapi.production.nest.com:9553 "GET /devices/thermostats HTTP/1.1" 200 None
2019-12-19 01:15:39,479, DEBUG: Initialized SSE client from event source <urllib3.response.HTTPResponse object at 0x75a0daa8>
2019-12-19 09:15:35,911, DEBUG: Dispatching put event, 1460 bytes...
2019-12-19 09:15:35,912, DEBUG: put event, 1460 bytes
2019-12-19 09:15:35,913, DEBUG: The data has changed (or initial data sent)
2019-12-19 09:15:35,986, DEBUG: inserting data
2019-12-19 09:15:35,986, DEBUG: INSERT INTO Temps (Timestamp, RoomTemp, Heating, TargetTemp, TimeToTemp) VALUES (NOW(), 19.5, 0, 15.0, '~0')
2019-12-19 09:15:35,998, DEBUG: Dispatching keep-alive event, 4 bytes...
2019-12-19 09:15:35,999, DEBUG: keep-alive event, 4 bytes
2019-12-19 09:15:35,999, DEBUG: No data updates. Receiving an HTTP header to keep the connection open.
2019-12-19 09:15:35,999, DEBUG: Dispatching keep-alive event, 4 bytes...
2019-12-19 09:15:36,000, DEBUG: keep-alive event, 4 bytes
...
...

The log continues with the normal event stream after this from the stream over the previous 8 hours in the course of 4 seconds.

The thing that catches my eye is the time ibetween these events

2019-12-19 01:15:39,479, DEBUG: Initialized SSE client from event source <urllib3.response.HTTPResponse object at 0x75a0daa8>
2019-12-19 09:15:35,911, DEBUG: Dispatching put event, 1460 bytes...

[mpetazzoni]

@8633brown If you can reproduce the problem every time it would be great if you could get a tcpdump/pcap capture while this happens. I'm curious if any data is received during those 8 hours or not; if there is data being received, but no event dispatched by the sseclient, then maybe something about the structure of the data stream is confusing the client into thinking that events haven't ended yet.

[vickybalakrish]

Hi All,

Im also facing this issue, sseclient stopped receiving the message and had to restart the service to make it work again. Have tried the above pseudocode , but still no luck.
Messages are stuck in client.events() line, so not able to recreate the client . We are using 1.5 version of sseclient.
Any workaround is appreciated

Regards,
vignesh