GDCC/8604 Improve archiver error handling#8612
Merged
kcondon merged 11 commits intoIQSS:developfrom Jul 18, 2022
Merged
Conversation
qqmyers
added
TDL
also increased wait to 20K seconds (and use new constant) to help handle larger data
GDCC/8604-improve_archiver_error_handling
…chiver_error_handling
…chiver_error_handling
…chiver_error_handling
…chiver_error_handling
sekmiller
approved these changes
Jul 13, 2022
Contributor
sekmiller
left a comment
sekmiller
left a comment
There was a problem hiding this comment.
This probably needs to be updated from Dev before final testing
Member
Author
|
As far as I can see it is up-to-date w.r.t. develop. |
…chiver_error_handling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: This PR fixes a thread management issue that allowed partial bags to be generated in cases where, for example, datasets were missing physical files. As described in the issue, the new code uses thread.join to assure the main thread gets an accurate success/fail report from the bag generation thread.
The PR also includes changes to the DuraCloud archiver class that changes the folder structure created in the DuraCloud service to add datasets within the same collection to the same DuraCloud 'space'. While transparent to Dataverse, this change makes it easier for an admin to manage batch archiving (see #8610) with Duracloud by reducing the number of manual operations required on that side. The DuraCloud archiver also gets the update from the Google archiver to wait for zipped datafile bytes to be ready before starting the transfer (see large comment in source).
Finally, as part of syncing the DUraCloud and Google ASrchivers and preparing for an S3 archiver, this PR includes refactoring to put the basic setup of the bag generation thread into the abstract base class.
Which issue(s) this PR closes:
Closes #8604
Special notes for your reviewer: These changes have been tested at TDL and QDR (DuraCloud and Google respectively) (albeit in combination with other PRs/with the QDR fork).
Suggestions on how to test this: Not sure there is much that can be tested without setting up Google/DuraCloud. The changes can be demonstrated by TDL/QDR. Another PR(s) are planned as part of DataCommons to apply the same threading technique to storage in S3/Harvard DRS where it should be easier to confirm that the new code works, e.g. in cases where datasets are missing physical files.
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?: see #8611
Additional documentation: