storage/block_device: initial implementation#359
storage/block_device: initial implementation#359cgonyeo wants to merge 1 commit intocoreos:masterfrom
Conversation
cgonyeo
force-pushed
the
master
branch
4 times, most recently
from
45ef9f7 to
dba52a8
Compare
|
This is no longer a WIP, I'm happy with the current state of the tests and they all pass. Unfortunately it looks like travis ci doesn't support loopback devices, so my tests don't succeed when run there. Anyone have any ideas on that one? Maybe just disable my tests in CI for now? |
cgonyeo
force-pushed
the
master
branch
2 times, most recently
from
9a0f1f6 to
c58a4d0
Compare
|
I've updated my added tests to require the |
|
Review bump |
|
@dgonyeo I'll review, plus... THANKS FOR THE UNIT TESTS!! Finally! |
|
@dgonyeo Quick question before I start the review. It says on your comment: This would create a very unaligned disk chunk because everything would be off by 512 bytes. You want to store metadata on the first chunk, not LBA. For example, a 4k write to LBA 0 from the caller, would not be aligned to the chunk if the is 4k in size. If the chunk is 512k in size and the caller write the last 4k in that chunk, 4k-512 bytes would go into this chunk, and 512 bytes would go into the next. |
|
Seems to me like the easiest fix would be to make that starting section 4k then, right? I'll tweak this PR to do that. |
This commit adds support for using a block device as the backing storage for torusd. Two utilities have been added, mkfs.torus and fsck.torus, to format a block device for use by torus, and for checking the consistency of a formatted block device, respectively. There is a metadata section of size 4k bytes written to the beginning of the disk, and the rest of the disk is used to store data. Each location on the disk in which data can be stored has two things: block headers of size 4k bytes, and the actual data, whose size is determined by the torus cluster's block size. When given a block ref, the location of the data is determined by feeding 24 bits comprised of the block ref's volume, inode, and index through sha256, converting the last 8 bytes into a uint64, and modding that by the number of slots available to store data in. By going to this location and reading the block headers, one can then determine if the data does live at this location on disk or get the real location of the data. The block headers are simply a series of 32 bit chunks, containing the three fields from a block ref and a location field. The location field will either be 0, signifying the data for this block ref is at this location, or a non-zero integer, which is the location of a different set of block headers to go to. When writing a block, if the default location is used for a preexisting block (so, a hash collision), linear probing is then used to find the next free location to use.
|
Cleaning up my old PRs |
2 participants
This commit adds support for using a block device as the backing storage for
torusd.
Two utilities have been added, mkfs.torus and fsck.torus, to format a block
device for use by torus, and for checking the consistency of a formatted block
device, respectively.
There is a metadata section of size 512 bytes written to the beginning of the
disk, and the rest of the disk is used to store data.
Each location on the disk in which data can be stored has two things: block
headers of size 512 bytes, and the actual data, whose size is determined by the
torus cluster's block size.
When given a block ref, the location of the data is determined by feeding 24
bits comprised of the block ref's volume, inode, and index through sha256,
converting the last 8 bytes into a uint64.
The block headers are simply a series of 32 bit chunks, containing the three
fields from a block ref and a location field. The location field will either be
0, signifying the data for this block ref is at this location, or a non-zero
integer, which is the location of a different set of block headers to go to.
When writing a block, if the default location is used for a preexisting block
(so, a hash collision), linear probing is then used to find the next free
location to use.
Fixes #171.