Unable to consume messages, need to re-subscribe client

[bardock]

This is similar to issue 2013. We have several shared subscriptions of partitioned topics and we found some cases where a subscription's client is not receiving messages, but it's not related to a specific topic partition.
In this case, if we re-subscribe the client (PulsarClient.subscribeAsync), messages start to be delivered as expected.

Client logs this warn:

[log_time:16:17:42.377] [thread:pulsar-timer-6-1] [level:WARN ] [logger:UnAckedMessageTracker] - [ConsumerBase{subscription='picking-consumer-prod.fbm-wms-picking', consumerName='i-08e97d991d75e9d07-10.73.125.156', topic='persistent://fury/global/wms-wave-feed-prod.fbm-wms-picking/wms-wave-feed-prod.fbm-wms-picking'}] 1 messages have timed-out

And we don't see any error from brokers.

Any idea? Any log or metric that may be useful to debug this problem?

System configuration

Pulsar version (brokers): 1.20.0
Pulsar clients version: 1.22.0
Bookkeeper version: 4.7.1

[merlimat]

@bardock Can you shared topics stats : pulsar-admin persistent stats $TOPIC and internal stats pulsar-admin persistent stats-internal $TOPIC

[bardock]

Here they are: stats.zip
Thanks!

[merlimat]

@bardock From the stats, it seems all partitions have rate 0, and no backlog accumulated. I couldn't see any stuck consumer/subscription in there.

[bardock]

@merlimat sorry, that consumer is working ok right now. I will update this issue when it'll happen again. We are facing this problem almost every week. Thanks!

[guillaumepitel]

Hi. I don't know if it's a similar issue, but here is what just happened on our Pulsar cluster (small three nodes clusters) :

One of the 3 broker suddenly stopped sending messages, its total MSG/S OUT was zero while MSG/I IN were still at ~100k/s. So it acted as if all the consumers on this particular broker were simultaneously disconnected. The odd thing is that no exception were thrown, and no error or warning were issued, either on the client side (the log level for pulsar is WARN) or on the brokers. The topics are persistent, failover non shared. All machines (clients and brokers) were under light load (0% wait, 20% CPU, low network BW).

After a client restart, the consumers started receiving messages again.

Is there anything we can do to detect such disconnection ? I thought pulsar clients & consumers were supposed to reconnect automatically, but maybe we need to listen to ConsumerEvents to detect disabled consumers ?

Client & brokers are 2.1.0

[sijie]

@guillaumepitel : did you acknowledge the messages on consumers? One thing to check when this happens, you can run pulsar-admin persistent stats $TOPIC and pulsar-admin persistent stats-internal $TOPIC to collect the stats of topics. so we will have a better idea on how many consumers connected to broker for that topic and other metrics will potentially help a lot as well.

[merlimat]

@sijie I am looking at a heap dump from @guillaumepitel and trying to get repro of this issue.