Firestore listener stops receiving changes after a long time but without throwing an error

[NixBiks]

Environment details

• OS type and version: debian-10-buster-v20191210 (GCP image)
• Python version: 3.8.1
• pip version: 19.2.3
• google-cloud-firestore version: 1.6.1

Steps to reproduce

I have a python script that sets up a listener to new documents in a firestore collection using on_snapshot on a query. To keep the script alive I have an infinite while loop in the end. Everything works fine but after several hours it stops receiving new documents (or the callback is not executed at least) - yesterday it was after ~7 hours.

In an earlier version of google-cloud-firestore (1.5.0) there was a related issue where the listener got disconnected after 1 hour. I made my own ugly hack back then but it worked at least:

Class ...
    .
    .
    .
    def _connect(self):
        now = None
        query = self.query
        if self.add_time_restriction:
            now = datetime.datetime.utcnow()
            query = query.where('updated', '>=', now)

        logger.info('Subscribing to query{}'.format(" after {}".format(now) if self.add_time_restriction else ""))
        query_watch = query.on_snapshot(callback=self.callback)
        if self.query_watch is not None:
            logger.info("Unsubscribe to former watch")
            self.query_watch.unsubscribe()

        self.query_watch = query_watch
        threading.Timer(3000, self._connect).start()

Code example (sample)

Since the 1 hour issue was fixed I now do

class Firestore:
    def __init__(self):
        self.client = Client()
        self._unsubscribe_news = lambda: None

    def watch_news(
        self
    ):
        def on_snapshot(col_snapshot, changes, read_time):
            logger.debug(f"Parsing news changes at {read_time}")

        self._unsubscribe_news()
        now = datetime.datetime.utcnow()
        query_watch = self.client.collection("news").where("updated", ">", now).on_snapshot(
            on_snapshot
        )
        logger.info(f"News listener set up at {self.client.project} from {now}")
        self._unsubscribe_news = query_watch.unsubscribe

and then execute a script like

firestore = Firestore()
firestore.watch_news()
while True:
    pass

Stack trace

Unfortunately no stack trace since the listener just stops receiving messages.

I see this as the core feature of Firestore so hopefully it'll get the right attention right away (some of our core business is to trigger python applications on news from firestore so we are extremely dependent on this).

[NixBiks]

Just noticed this which seems like the same issue: firebase/firebase-admin-python#294. It concerns me that is has been open since May last year - isn't this considered a core feature?

[NixBiks]

Btw. I always start two processes running in the background. Both listen to the same news but handle it differently. In the end both processes stopped receiving news at the same time.

[BenWhitehead]

@crwilcox or @tseaver Any ideas here about what could be going on as far as the python code is concerned? I've not yet heard about this happening in other languages.