[BUG] Kafka topic offset gap

[Raven888888]

Describe the bug
I have been observing offset gap in kafka topic (KoP), same thing did not happen on standalone kafka server so far.

I am trying to listen to a kafka topic
apache-kafka/bin/kafka-console-consumer.sh --topic test --offset 95719032467224513 --partition 0 --bootstrap-server 192.168.0.1:9092
and it stuck forever. This is causing my druid kafka ingestion to fail due to request timeout.

If I try a smaller 85719032467224513 or bigger 105719032467224513 offset , able to consume the messages.

Any reason why there is an offset gap?
Is there a way to avoid gap at all?

Thanks in advance.

[BewareMyPower]

Which KoP version did you use? Before KoP 2.8.0, there's no solution because the KoP offset is not continuous.

If you used KoP >= 2.8.0, could you try to retrieve the LEO offset?

[Raven888888]

Thanks for your quick response @BewareMyPower

Yes I'm using KoP kafka_2.12-2.8.0.tgz on pulsar 2.7.0.

The LEO offset is greater than the offset when the gap happens.

apache-kafka/bin/kafka-console-consumer.sh --topic test --offset 112273331336003585 --partition 0 --bootstrap-server 192.168.3.1:9092
returns nothing.

apache-kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 192.168.3.2:9092 --topic test
returns test:0:115088081102569472

Also, there is only 1 partition 0 for this topic.

[Raven888888]

Maybe I should also mention, both KoP on standalone pulsar and pulsar cluster have this offset gap issue.

[BewareMyPower]

Yeah, it's a KoP side issue. If you're using Pulsar 2.7.0, then KoP must be also 2.7.x. Your mentioned kafka_2.12-2.8.0.tgz is Kafka's version (Kafka client), not KoP. KoP acts as a Kafka server, your cmd tool is just a Kafka client.

Before KoP 2.8.0, the offset design has some serious problems. Because it maps the message id to offset by:

x (of 64) bits ledger id + y (of 64) bits entry id + z (of 32) bits batch index = 64 bits offset

Here's a comment related to the design: #283 (comment)

The offset is not continuous, which causes many annoyed issues like #277, #283, and this issue. For this issue, it's because your requested offset cannot be mapped to a valid message id, so KoP cannot handle it well.

From KoP 2.8.0, we have continuous offset in KoP based on the feature introduced from PIP-70. But it's not compatible with older Pulsar. If you're going to use KoP, it's highly recommended to upgrade Pulsar to 2.8.0+ and download the associated KoP version. Usually KoP x.y.z.m (like the latest 2.8.1.3) is compatible with Pulsar x.y.z (like 2.8.1).

[Raven888888]

Thanks again for your quick response @BewareMyPower

I'll upgrade pulsar and KoP to 2.8.0 and try again! 🤞 Cheers