Paginated query over 2i with range return non existent data

[peczenyj]

Hi all

I am using Riak 1.4.2 + levelDB and I find something strange.

We have one index called 'expiration_epoch_int', it is like a TTL for a particular key in this bucket. To find expired data to delete is just query over expiration_epoch_int between 0 and 'now'. For a small amont of data it seems really good.

But today I find this: the first 10 results from this query return non-existent keys. It was already deleted. I receive one 'HTTP/1.1 404 Object Not Found' if I try to inspect.

If I use a small range, like around +/- 1 second from now, I can find good results (keys who exists in Riak) but if I start from 0 ( or 1) at least the begining are keys who does not exist.

If I use return_terms=true I can find the expiration_epoch_int too (it returns data between 10 and 20 days ago). I am using the PBC interface for query and delete.

So, my question: why this happens? can be related to pagination (maybe some cache)? When we perform our cleanup process, we process, for example, ~7 x10^6 keys.

To control the expiration of a huge amount of data, it is save use only one secondary index? There is some limit for a huge number of keys? I have no idea where I can start to investigate this.

I will try to run a more complete test to find the % of deleted keys returned by Riak.

[jaredmorrow]

Also /cc'ing @engelsanchez and setting milestone for 2.0.1. Since I don't know if this was already fixed in the 1.4.x series.

[agnibha92]

By any chance, anyone is looking into this issue.
We are also facing the same issue in Riak 2.2.3 version

[peczenyj]

By any chance, anyone is looking into this issue.
We are also facing the same issue in Riak 2.2.3 version

The last commit in this repository was on "Aug 25, 2017"

I think you should consider an alternative to Riak (Couchbase?)

[martincox]

@peczenyj develop branch isn't being actively maintained (it should perhaps be archived and the default branch changed), but there is still plenty of ongoing development around riak. Take a look at develop-2.2 and develop-3.0. Riak 2.9 has recently been released, with Riak 3.0 being worked on currently and expected later in the year.

[peczenyj]

@peczenyj develop branch isn't being actively maintained (it should perhaps be archived and the default branch changed), but there is still plenty of ongoing development around riak. Take a look at develop-2.2 and develop-3.0. Riak 2.9 has recently been released, with Riak 3.0 being worked on currently and expected later in the year.

but this is "official/basho" or it is "community"?

[martincox]

Basho shut up shop a while ago. @bet365 bought everything to preserve it and open sourced the proprietary stuff (DC replication). There's a few companies now actively contributing (NHS, bet365, TI-Tokyo), alongside the community.

[peczenyj]

@martincox but basho website is still active, also the Riak wikipedia has o mention to @bet365 - to be honest if the first time that I listen about this.
So, this is still the official repository?

Glad to know this project is still alive. I spend some good times writing a perl client to the protobuf interface years ago.

[martincox]

@peczenyj Yep, the website is still up, and the basho "brand" is still alive, in name only, though. There needs to be a move away from that (probably just rebranding to just "riak"); it's just that nobody has had the resource to do it.

This is the official repo, yes. And, is still very much alive and kicking. Any community effort is welcomed. :)