Write path optimization: Inline transaction files

[wjones127]

Currently, transaction files are separate from the manifest. They are only locatable from the manifest itself. Thus, if you want to read the transaction data for a given version, it requires two hops: first, read the manifest to figure out the transaction file path, then read the transaction file itself. This also means the commit path requires at least two iops, since you need to write two different objects.

Instead, we should put the transaction file within the manifest. If the manifest is small enough, that means when we read the footer we will also get the transaction data for free. (We already read data with a minimal block size.) For larger manifests, we won't read this data, unless requested.

This will take writes from 3 write IOPS to 2 write IOPS, by combining the transaction file and manifest writes. When there are conflicts, and we need to read the conflicting transactions, this will often result in fewer IOPS as well, since we can read the transaction information in one IOP instead of two.

TODO

• Start writing transaction file in manifest
• Opportunistically read transaction file when reading manifest
• Optimize IO requests when doing conflict resolution (use Manifest store to get size of relevant manifest files)
• Add feature flag to disable writing separate file. (This can be made default in 6 months.)

[westonpace]

This is just the one transaction that was used to create the particular manifest? Or is it the history of transactions?

[wjones127]

Just the one transaction