Add process for active reaping of old manifest versions [JIRA: RCS-317]

[kellymclaughlin]

Riak CS currently relies on objects being written to remove old manifest versions. In the normal case the storage impact of this may be negligible because a record it written to the GC bucket for each old manifest version and the data for the objects represented by the old manifest versions should be reaped after the leeway period.

In the non-normal case, if the write to the GC bucket fails for some reason the data represented by the old manifest versions is not reaped and no repeat attempt is made until the key for the object is written again. This can lead to storage usage issues that are difficult to diagnose and account for.

An active process to search actively seek out and clean up old manifest versions would be a useful supplement to the current cleanup-on-write strategy.

[kuenishi]

I agree that we need some active cleanup process. Also, Riak CS already has several active processes like GC, storage calc. And I have been thinking of adding another active process. We also might need small task scheduling system (hopefully integrated with handoff, AAE and MapReduces) to avoid overloading the system.

[kuenishi]

Active Reaper

Basic idea is to add a daemon to the supervision tree, which periodically scans all buckets, processes each bucket as follows:

• Issue a mapreduce that retrieves all stale writing manifests and pending_delete manifests
• While waiting for the stream:
  • if the number of a buffer exceeded move_batch_size, put them all into GC bucket and move state to scheduled_delete
  • if any writing manifests delivered, move them to pending_delete right now
  • if any pending_delete manifests delivered, put them into a buffer
• After the mapreduce done:
  • process all remaining manifests

Configuration Parameters

• active_reaper.sched - [] - in which time in a day this was scheduled
• active_reaper.mapred_throttle - 5min - wait interval between two mapreduces
• active_reaper.bucket_batch_size - unlimited - limit number of objects in a bucket processed at once
• active_reaper.move_batch_size - 128 - how many objects to be put into GC bucket at once
• Command line tools
• riak-cs-admin reaper batch starts a batch even if not scheduled
• riak-cs-admin reaper status shows status and stats of current reaper job
• riak-cs-admin reaper pause pauses currently running batch

Performance and Safety Considerations

This must be ran in a healthy (but just a bit too much disk usage) cluster. No handoffs should be happening, all repairs must be done via AAE or vnode repairs.

This must be ran in a cluster that has marginal space in both read and write IOPS. This adds another scanner that issues fold operation to leveldb which could be a huge load to the cluster - whose cost is proportional to all keys in each bucket at single mapreduce.
Also, issuing manifest state update to pending_delete or sheduled_delete and puts to GC bucket will trigger writes which size is proportional to the number of keys that requires reaping.

It is recommended to run in a single node like garbage collection and storage calculation as each occupies one fold slot in all vnodes.

[kuenishi]

This will be a big fix so that I won't include this in 2.1.1 release.