Integrate chunking and TSM/cloud

[dspeed2]

This maps to the following Jira issue

https://www.jira.is.ed.ac.uk/browse/RSS022-127

I've managed to merge the chunking into my S3 branch, I didn't get any conflicts which was a nice surprise \0/

I can see that the chunking does need some tweaking to work with both the TSM and S3 plugins though*. I'll sort that out asap

I suspect other plugins might have issues with it too, the TSM and S3 problem is that they use the a different value for the archiveId which isn't in the expected format when the chunking tries to process it.

I'm just expecting this to require tweaking rather than a massive refactor.

[dspeed2]

Here is the latest comment from Jira on my testing of the merged code.

So far I've tested that deposit and retrieve still work on my VM which they do for a deposit smaller than the default chunk limit of 1GB.

They also still work for a deposit where I've changed the chunk limit so that we should get 3 chunks but I can see that I've messed up at least one loop in the verify stage and it is verifying each chunk multiple times (I suspect x the number of chunks)
I'll fix that then finish the testing.

[dspeed2]

I suppose I could merge the code into master at this stage if we are desperate but I'll try and fix it asap so I can commit the finished version.

[dspeed2]

Just a copy of the latest update to the matching Jira

I've rant the following manual tests agains the branch on my machine
Chunking off
Deposit and Retrieve fine
Chunking on 1 file
Deposit and Retrieve fine
Chunking on 3 files
Deposit and Retrieve fine
Chunking on 1 file first copy missing
Retrieve fine (used 2nd copy)
Chunking on 1 file first & second copy missing
Retrieve unsuccessful with error message (the message needs improving and notifications need added)
Chunking on 3 files first copy of 2nd file missing
Retrieved fine (used 2nd copy of all chunks)
Chunking on 3 files first & second copy of 2nd file missing
Retrieve unsuccessful with error message (the message needs improving and notifications need added) (both copies failed and will fail even if we have enough ‘good’ chunks over both copies i.e. both copies have no knowledge of each other)
Chunking on 1 file first copy bad checksum

Which all behaved as expected (apart from maybe the 1st and 2nd copy now merging from each other)

I also meant to run these tests but haven't got around to it, this features were tested at the time they were added but we should probably test them again with these changes.

Chunking on 1 file first & second copy bad checksum
Chunking on 3 files first copy of 2nd file bad checksum
Chunking on 3 files first & second copy of 2nd file bad checksum

I've set up a pull request for these changes and assigned it to William.

Hi all, just out of curiosity, do we have any stats on how often or how much data corruption we should expect for data stored in TSM?

[dspeed2]

Hey, I don't, I was rather hoping we wouldn't have any :-)

We have discussed some sort of cron spider that runs over the data and checks / repairs where possible (I'm hoping I haven't made that up).

I guess I was just thinking that a lot of work seems to have gone in to all this chunking mechanism and it would be nice to know that it was worthwhile :)