[BUG] Ledger id used up quickly

[aloyszhang]

Describe the bug
The offset of kakfa map to messageId of pulsar by splitting 64 bits (a long type) to three parts, 20 bits for ledgerId , 32 bits for entryId and 12 bits for batchIndex , therefor, ledgerId can't be larger than （2^19 - 1） since long type is signed.
If we have lost of topics or partitions, ledgerId will be overflow very soon.
Besides, entryId has 32 bits, but pulsar ledger will rollover for every 50000 entries by default. Most of entryId space will waste if ledger rolled .
In my opinion, we should re-assign the bits for ledgerId and entryId to make kop have enough range for ledgerId . I have redistribute the ledgerId with 35 bits, entryId with 17 bits and batchIndex with 12 bits.
@sijie @jiazhai Is there any better way to resolve this problem ? Any suggestions will be appreciated.

[sijie]

@aloyszhang

In my opinion, we should re-assign the bits for ledgerId and entryId to make kop have enough range for ledgerId . I have redistribute the ledgerId with 35 bits, entryId with 17 bits and batchIndex with 12 bits.

This approach looks good to me. But we need to figure out a plan for backward compatibility.

Alternatively, we should introduce a "sequence" in the Pulsar topic. A sequence is a monotonically increasing long number. This would ensure we have enough bits. If you are interested in contributing to this feature, we can discuss a PIP for Pulsar.

[jiazhai]

👍， We could find a time to discuss a PIP.

[aloyszhang]

@sijie @jiazhai

Alternatively, we should introduce a "sequence" in the Pulsar topic. A sequence is a monotonically increasing long number. This would ensure we have enough bits.

It's great idea that if we can map kafka offset to a long type sequence.
I have talked about this with @jiazhai roughly before. We can talk about some details first.