Data missing when using CassandraIO.Read

[damccorm]

Bug

Data at the beginning or end of the token ring is never retrieved, due to a bad TokenRange request.

This bug was introduced by BEAM-9008, in this commit

A basic reproduction case & workarounds are available here:

Github/beam-cassandraio-bug

Description

When using {}CassandraIO{}, a list of token ranges is requested to C* nodes in order to create splits in those ranges.
A split will be represented as a RingRange resulting in a request to C* in the form of
TOKEN(partition_key) \>= range_start AND TOKEN(partition_key) < range_end

The token ring goes from Long.MIN_VALUE to Long.MAX_VALUE (so -2xxx to 2xxx), a range may contains the "join point" and be represented by [2xx, -2xxx].

In this case (Aka TokenRange isWrapping), old implementation used to send 2 different requests:

• TOKEN(partition_key) >= range_start (To get result up to the end of the ring, i.e. Long.MAX_VALUE)
• TOKEN(partition_key) < range_end (To get result from the beginning end of the ring, i.e. Long.MIN_VALUE)

Now, this behavior is not implemented anymore and token ranges are all called the same way, even in the wrapping case.
It results in a request like :
TOKEN(partition_key) >= 2XXX AND TOKEN(partition_key) < -2xxx
This gives 0 results, and some data is never retrieved.

 

WorkArounds

• Downgrade to 2.33.0
• Use customer TokenRanges & readAll implementation

Imported from Jira BEAM-14558. Original Jira may contain additional context.
Reported by: croquette.

[KriKroff]

Hey @damccorm,

Thanks for migrating the ticket.
In the README, it is written to create issues on JIRA, is there any changes ?

[damccorm]

Whoops, thanks for calling that out. I updated a bunch of docs but missed that one - new issues should now be filed in the GitHub issues tab.

I put up #21718 to fix the Readme

[kennknowles]

@vmarquez any thoughts?

[kennknowles]

@pabloem this has been introduced in 2.34.0 so it is not new, but I do think you should be aware of it, in case someone has a quick fix or a rollback we can include in the next release

[vmarquez]

@kennknowles @damccorm My sincere apologies that this slipped in. Unfortunately, the test cases didn't catch it, we didn't lose data in the small local test cluster when I was testing, and for our large workloads in production we use readAll for efficiency reasons. To fix this issue, I think we can add in two queries similar to how it was before BEAM-9008

#21786

Fix is ready to review now.