Conversation
| private boolean usetemp = false; | ||
|
|
||
| private int numConnections = 8; | ||
| private static int numConnections = 2; |
There was a problem hiding this comment.
Just curious why you changed this to 2. To match the docs? https://dataverse-guide--12129.org.readthedocs.build/en/12129/installation/config.html#baggeneratorthreads
There was a problem hiding this comment.
In the https://dataverse-guide--12129.org.readthedocs.build/en/12129/installation/config.html#bagit-export section, I note that higher values may cause more issues with throttling. I've added some exponential back-off and retries that seem sufficient to get through 10K files with 2 threads. It would probably need more aggressive/configurable back-off for more threads, but, since the main reason to thread is to go faster, it probably is a bad idea to add threads and then have them wait more/end up retrying more. So - lower default to handle throttling and people should be fine to go up if their servers aren't throttled.
What this PR does / why we need it: For back-end stores using something like Oxford Common File Layout to deduplicate files across versions, the archival info for all prior dataset versions has to be in place before you can add the next/latest version. This PR adds a feature flag to turn on this behavior and adds code to check/fail if earlier versions haven't successfully been archived. It also avoids showing a submit button for a version in the dataset page version table if there's no chance of success (because earlier versions aren't archived).
Which issue(s) this PR closes:
DANS issue 2097
Special notes for your reviewer:
Suggestions on how to test this: Set the flag, verify the datasetpage version table doesn't show a submit button for v2+ if you haven't archived v1. Verify an archiving post-publish workflow fails for new versions if v1 hasn't succeeded. (Probably easiest to delete the archival status of v1 via API, but could just configure the workflow after publishing v1).
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?:
Additional documentation:
Preview docs at https://dataverse-guide--12129.org.readthedocs.build/en/12129/installation/config.html