Consumer already closed with system topic

[Shawyeok]

Describe the bug
Unexpected log output by logger org.apache.pulsar.client.impl.MultiTopicsConsumerImpl, thread: pulsar-client-internal-45-1

Receive operation failed on consumer ConsumerBase{subscription='multiTopicsReader-98cff5df66', consumerName='6a4ac', topic='persistent://c/mot/__change_events-partition-2'} - Retrying later
java.util.concurrent.CompletionException: org.apache.pulsar.client.api.PulsarClientException$AlreadyClosedException: Consumer already closed
	at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_181]
	at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_181]
	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:647) ~[?:1.8.0_181]
	at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:669) ~[?:1.8.0_181]
	at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:1997) ~[?:1.8.0_181]
	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.receiveMessageFromConsumer(MultiTopicsConsumerImpl.java:248) ~[org.apache.pulsar-pulsar-client-original-2.8.1.jar:2.8.1]
	at org.apache.pulsar.client.impl.MultiTopicsConsumerImpl.lambda$receiveMessageFromConsumer$9(MultiTopicsConsumerImpl.java:274) ~[org.apache.pulsar-pulsar-client-original-2.8.1.jar:2.8.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_181]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_181]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_181]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_181]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.66.Final.jar:4.1.66.Final]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
Caused by: org.apache.pulsar.client.api.PulsarClientException$AlreadyClosedException: Consumer already closed
	at org.apache.pulsar.client.impl.ConsumerBase.verifyConsumerState(ConsumerBase.java:754) ~[org.apache.pulsar-pulsar-client-original-2.8.1.jar:2.8.1]
	at org.apache.pulsar.client.impl.ConsumerBase.receiveAsync(ConsumerBase.java:162) ~[org.apache.pulsar-pulsar-client-original-2.8.1.jar:2.8.1]
	... 10 more

Screenshots

Additional context

• pulsar version: 2.8.1
• OS: CentOS Linux release 7.9.2009 (Core)
• kernel: 3.10.0-1160.45.1.el7.x86_64

cc @codelipenghui

[mattisonchao]

@Shawyeok Could you give us more details or context ?

• How to get it ?
• How to reproduce it ?

[yuvalgut]

also experiencing something similar related to systemTopics:

{"msg":"org.apache.pulsar.client.impl.ConsumerImpl - [persistent://tenant/default/__change_events-partition-2][multiTopicsReader-f2ae775e21] Failed getLastMessageId command","thread":"pulsar-io-4-3","timestamp":"07:38:28.863"}

"msg":"org.apache.pulsar.client.impl.ClientCnx - [id: 0x9ad7ffd4, L:/100.64.3.1:56568 - R:pulsar-broker-] Received error from server: Failed to read last entry of the compacted Ledger Incorrect parameter input","severity":"WARN"

the broker cannot recover this exception unless I force delete the systemtopic.
the broker gets into that state sporadically.

pulsar broker version: 2.8.2

[Shawyeok]

@mattisonchao @yuvalgut It is duplicate with #12024, and fixed by #12044