[broker] Close topics that remain fenced forcefully

[massakam]

Motivation

The other day, we faced a problem where a topic remained fenced and unavailable. This topic remained unavailable until it was unloaded. The following is the broker log at that time.

11:37:55.905 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO  o.a.b.mledger.impl.OpAddEntry        - [tenant/ns/persistent/topic] Closing ledger 40891546 for being full
11:37:56.208 [pulsar-ordered-OrderedExecutor-0-0-EventThread] ERROR o.a.b.client.MetadataUpdateLoop      - UpdateLoop(ledgerId=40891546,loopId=6ce63876) Error writing metadata to store
11:37:56.209 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.b.mledger.impl.OpAddEntry        - Error when closing ledger 40891546. Status=Error while using ZooKeeper
11:37:56.359 [pulsar-ordered-OrderedExecutor-0-0-EventThread] ERROR o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] Error creating ledger rc=-9 Error while using ZooKeeper
11:37:56.359 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO  o.a.pulsar.broker.service.Producer   - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic}, client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO  o.a.pulsar.broker.service.Producer   - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic}, client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO  o.a.pulsar.broker.service.Producer   - Disconnecting producer: Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic}, client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to persist msg in store: Error while using ZooKeeper
11:37:57.495 [ForkJoinPool.commonPool-worker-51] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40256][persistent://tenant/ns/topic] Creating producer. producerId=668
11:37:58.291 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO  o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] End TrimConsumedLedgers. ledgers=2 totalSize=162868668
11:37:58.291 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO  o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] Removing ledger 40880508 - size: 82183409
11:37:58.292 [ForkJoinPool.commonPool-worker-20] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40256]-668 persistent://tenant/ns/topic configured with schema false
11:37:58.292 [ForkJoinPool.commonPool-worker-20] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Attempting to add producer to a fenced topic
11:37:58.292 [ForkJoinPool.commonPool-worker-20] ERROR o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40256] Failed to add producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:37:58.728 [ForkJoinPool.commonPool-worker-75] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40330][persistent://tenant/ns/topic] Creating producer. producerId=668
11:37:58.729 [ForkJoinPool.commonPool-worker-75] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40330]-668 persistent://tenant/ns/topic configured with schema false
11:37:58.729 [ForkJoinPool.commonPool-worker-75] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Attempting to add producer to a fenced topic
11:37:58.729 [ForkJoinPool.commonPool-worker-75] ERROR o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40330] Failed to add producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:37:59.489 [ForkJoinPool.commonPool-worker-106] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40260][persistent://tenant/ns/topic] Creating producer. producerId=668
11:37:59.489 [ForkJoinPool.commonPool-worker-106] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40260]-668 persistent://tenant/ns/topic configured with schema false
11:37:59.489 [ForkJoinPool.commonPool-worker-106] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Attempting to add producer to a fenced topic
11:37:59.489 [ForkJoinPool.commonPool-worker-106] ERROR o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40260] Failed to add producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:38:01.062 [ForkJoinPool.commonPool-worker-51] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40248][persistent://tenant/ns/topic] Creating producer. producerId=668
11:38:01.062 [ForkJoinPool.commonPool-worker-51] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40248]-668 persistent://tenant/ns/topic configured with schema false
11:38:01.063 [ForkJoinPool.commonPool-worker-51] WARN  o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Attempting to add producer to a fenced topic
11:38:01.063 [ForkJoinPool.commonPool-worker-51] ERROR o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40248] Failed to add producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:38:04.103 [ForkJoinPool.commonPool-worker-90] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40338][persistent://tenant/ns/topic] Creating producer. producerId=668
11:38:04.104 [ForkJoinPool.commonPool-worker-102] INFO  o.a.pulsar.broker.service.ServerCnx  - [/xxx.xxx.xxx.xxx:40338]-668 persistent://tenant/ns/topic configured with schema false

We were maintaining the ZooKeeper servers, so I think this phenomenon was caused by the shutdown of some ZK servers. However, the causal relationship has not been clarified.

Modifications

As a workaround, close the topic if it remains fenced for a period of time. Reconnecting from the clients will instantiate a new PersistentTopic topic and the topic will back to normal.

[Huanli-Meng]

Add a doc-required label as the broker.config and standalone.config file are updated.

[sijie]

Do we need to put close in the synchronized block?

[massakam]

I removed "synchronized" because there was no clear reason to make this method synchronized.

[codelipenghui]

@sijie Please help review this PR.

[codelipenghui]

/pulsarbot cherry-pick to branch-2.6