[WIP] Batched payment deletion#5368
Conversation
As a preparation of using multiple batches to delete the payments we begin with extracting the pure read into its own read transaction.
To reduce the size of the delete transactions, we batch the delete operations into batches of 1k. This should speed up the deletion process considerably as not all changes need to be kept in memory as a whole. We also add some debug log statements about the process so a user can see what's happening.
carlaKC
left a comment
carlaKC
left a comment
There was a problem hiding this comment.
I think that we can get into a bad state if we delete payment and index entry separately.
Taking the following example:
- We delete payment with index 3
DeletePaymentsis killed before we remove the index entry- Next time call
DeletePayments, the payment no longer exists so we don't queue its index for deletion
This will break calls to QueryPayments because we'll scan the index bucket, land on the index for 3 then try to lookup a payment that doesn't exist which will fail the call.
I think we could change the ordering to delete indexes first, but then we could end up with a weird case where the payment is no longer included in QueryPayments but it's actually still in the payments bucket. That would be resolved if we ran DeletePayments again, but may not be desirable to allow this kind of state to be a possibility.
| return nil | ||
| }) | ||
| }, func(){}) | ||
| }, func() {}) |
|
@guggero, remember to re-request review from reviewers for your latest update |
|
Individual payments can now be deleted, so the same effect can be achieved in a loop. Closing this until we decide we really need the functionality. |
4 participants
Attempts to fix #5349 by batching the delete operations of
DeletePayments.This makes the operation no longer atomic. But that might be okay... Or do we need to remove the payment bucket and indices in the same batch? Perhaps we need to do "horizontal" batching instead of "vertical"?