PubSub: Force push a batch of messages

[hnykda]

Is your feature request related to a problem? Please describe.
Hi,

I have a simple publish for loop:

for message in data:
        future = publisher.publish(topic_path, data=message)
        future.add_done_callback(self.get_callback(future, message))

with some callback (-> I don't wait until the future is returned). The problem is that when the last batch of data is being created, the process terminates earlier than it's being sent (<MAX_LATENCY), missing some last messages.

Could there be a way of something like publisher.join() or publisher.force_push() to make sure that all messages are sent before the process terminates?

Related:

• Pub / Sub Publisher: Async publish breaks around 1000 messages #4575
• PubSub Publisher: sometime cannot publish until a number of publish called #4607
• https://github.com/googleapis/google-cloud-python/issues/4994

[plamut]

@hnykda This sounds like the issue #6883, which was addressed in #9365, but it has not been released yet.

If feasible, could you try installing the latest development version of the PubSub client and try it out? The new publisher.stop() method should achieve the desired.

[hnykda]

Hi! Yes, it's exactly that, sorry for missing that one out when searching for similarities.

Will try. Any ideas when the next release could be out?

[plamut]

@hnykda No date yet, but I will propose to make it sooner rather than later. :)