Raft log streaming compaction

[aluzzardi]

Right now, compaction is done in a large single proto message.

We should evaluate streaming compaction, wherein a compaction is composed of a series of smaller messages.

/cc @aaronlehmann @abronan

[aaronlehmann]

I am planning to do this after #325 goes in.

Here is the relevant todo from the code:

// TODO(aaronl): Creating a StoreAction and marshalling it here
// for the size estimate is redundant because it gets created
// and marshalled a second time at commit time. This could be
// avoided by doing the marshalling only here, and streaming
// the serialized StoreActions to the proposer, which it would
// write using gogo-protobuf's io.DelimitedWriter. This would
// take some interface changes, so I'm leaving it as a TODO for
// now.

[aaronlehmann]

Discussed this with @stevvooe and now I think it's low priority. I think the serialized size of these objects will be significantly smaller than the deserialized memory overhead (because each top-level object has several pointers, maps, etc.), so the memory impact of the snapshot proto shouldn't be a huge deal.

The main reason I had planned to work on this was to stream out transactions to a buffer so I could avoid double-serialization for Batch's size estimates, but it looks like a better way to do this is to use the protobuf Size method.

[aluzzardi]

@aaronlehmann Yeah, the main reason I think we'd want to do that is tooling.

We'll probably want to create operational tooling eventually and streaming would prevent said tool to load the entire state into memory.

But that's low priority.

[aluzzardi]

@aaronlehmann Putting this back into the Icebox. I don't think this will happen in the next few releases.

[aaronlehmann]

SGTM. I don't think it's very important. The advantage of making this change would be reducing memory use when saving and loading snapshots. But the serialized form of the snapshot should be much smaller than the actual live data set (because of indices, pointers, and so on), so I don't think this will make much difference in practice.

[dperny]

Hey wow we did this for snapshots in #2458. So I'm gonna close it.

[anshulpundir]

This is not related to streaming snapshot across raft members @dperny