Message size is bigger than 5232640 bytes

[zhengqz]

use pulsar-2.3.1 run in standalone, and sync mysql binlog from canal(1.1.3), when message size more than 5M cause IllegalReferenceCountException

ENV

1. pulsar 2.3.1
2. Centos 7.6.1810
3. canal-mysql-source-config.yaml

configs:
  zkServers: ""
  batchSize: "5120"
  destination: "item_0"
  username: ""
  password: ""
  cluster: false
  singleHostname: "192.168.1.174"
  singlePort: "11111"

4. Command

/opt/apache-pulsar-2.3.1/bin/pulsar-admin source create --archive /opt/apache-pulsar-2.3.1/connectors/pulsar-io-canal-2.3.1.nar --classname org.apache.pulsar.io.canal.CanalStringSource --tenant public --namespace default --name canal --destination-topic-name my-topic --source-config-file /opt/apache-pulsar-2.3.1/conf/canal-mysql-source-config.yaml --parallelism 1

/opt/apache-pulsar-2.3.1/logs/functions/public/default/canal/canal-0.log

00:01:37.844 [pulsar-client-io-1-1] INFO  org.apache.pulsar.io.canal.CanalAbstractSource - CanalRecord ack id is 159877
00:01:37.986 [pulsar-client-io-1-1] INFO  org.apache.pulsar.io.canal.CanalAbstractSource - CanalRecord ack id is 159878
00:01:38.058 [pulsar-timer-4-1] ERROR org.apache.pulsar.functions.sink.PulsarSink - Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException$InvalidMessageException: Message size is bigger than 5232640 bytes] with sr
c sequence id [159879]
00:01:38.079 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.091 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.102 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.113 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.124 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.135 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.146 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.157 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.168 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.169 [public/default/canal-0] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.169 [public/default/canal-0] ERROR org.apache.pulsar.functions.sink.PulsarSink - Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException: org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalRefe
renceCountException: refCnt: 0] with src sequence id [159880]
00:01:38.179 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.189 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.200 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.211 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.252 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.263 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.270 [public/default/canal-0] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.271 [public/default/canal-0] ERROR org.apache.pulsar.functions.sink.PulsarSink - Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException: org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalRefe
renceCountException: refCnt: 0] with src sequence id [159881]
00:01:38.273 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.284 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.295 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.296 [public/default/canal-0] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.297 [public/default/canal-0] ERROR org.apache.pulsar.functions.sink.PulsarSink - Failed to publish to topic [my-topic] with error [org.apache.pulsar.client.api.PulsarClientException: org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalRefe
renceCountException: refCnt: 0] with src sequence id [159882]
00:01:38.307 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.319 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.331 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.343 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.355 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.367 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.379 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.390 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0
00:01:38.401 [pulsar-timer-4-1] WARN  org.apache.pulsar.client.impl.ProducerImpl - [my-topic] [standalone-8-2] error while closing out batch -- org.apache.pulsar.functions.runtime.shaded.io.netty.util.IllegalReferenceCountException: refCnt: 0

[shiv4289]

I think this is work in progress https://gist.github.com/zymap/08ec1bb688d2da16e9cd363780480e7a

[sijie]

This has been addressed by #4247 and will be included in 2.4.0 release.