Move workflows to onSuccess, use async export when publishing#12099
Open
qqmyers wants to merge 8 commits intoIQSS:developfrom
Open
Move workflows to onSuccess, use async export when publishing#12099qqmyers wants to merge 8 commits intoIQSS:developfrom
qqmyers wants to merge 8 commits intoIQSS:developfrom
Conversation
qqmyers
changed the title
qqmyers
added
GDCC: QDR
github-project-automation
bot
moved this from Ready for Review ⏩
to Ready for QA ⏩
in IQSS Dataverse Project
9 tasks
qqmyers
added
the
Status: Waiting for Related Issues/PRs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it: This PR moves running workflows into the onSuccess method (an old ToDo) and changes the code to add workflow locks before the current transaction ends in the publish/finalize commands and the dataset :releasemigrated API call. Earlier code relied on a 1 second pause when the asynchronous workflow starts to allow to initiating transaction to end before the workflow created it's own lock. This left a ~1 second gap when the dataset was not locked ( and added a second to execution). The change in this PR creates the lock in the initiating transaction and passes that to the workflow. (If a lock is not set the code defaults to the old behavior.)
The PR also changes exporting in the finalize publish command to be asynchronous and adds checking/recovery if an OptimisticLockException occurs (possible, for example, if a workflow completes and, like archiving, updates the datasetversion before export can run.)
Which issue(s) this PR closes:
Special notes for your reviewer:
FWIW: This is one of several PRs related to updates being done to improve archiving scalability at QDR and DANS. Some are relatively minor cleanup while others are more significant. I'm submitting them all to avoid drift in QDR's fork and to leverage QDR testing (where all of these are being tested together).
Suggestions on how to test this: Regression - index, export, and any configured workflows should complete as before. For a long-running workflow (such as archiving on a dataset with many files), the workflow lock should be in place (visible in UI, via API, in db). Nominally one could verify that there no longer is a gap between the finalize lock being dropped and the workflow lock being created, but not sure it's worth it.
It may be worthwhile to engage DANS, which is leveraging workflows, to confirm that these changes don't break their setup as well.
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?: Will create an aggregate one
Additional documentation: