-
Notifications
You must be signed in to change notification settings - Fork 6
Add tabIngest field and 500 error workaround
#24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
* `replaceFiles` is not called when there are no files to replace * `addFiles` is not called when there are no files to replace * Both are called when there exist both new and replacement files
This is added to provide control structures to avoid dataset locks.
- Loop-based upload of single tabular files - Files passed as an array of `File` objects - Parallelized upload
|
The PR has now been updated to accommodate intermediate locks by utilizing Configuring the retry behaviorFurthermore, this PR adds a documentation and a dedicated function to control the retry behavior. Since instances can and will differ, users might run into timeout issues or others, and this provides a way to accommodate these cases. However, there is no one-fits-all solution, and values may need to be determined manually. In most cases, the defaults should work, though. Extensions of test casesFinally, the PR adds multiple test cases for the native upload, which uploads a set of large tabular files to check whether ingest errors are occurring. For local testing, the size of the tabular files can be configured via |
According to IQSS/dataverse#11265 (comment) and #18, direct uploads of multiple tabular files trigger an
OptimisticLockexception, resulting in a500status response when trying to register the uploaded files. This issue can be avoided by disabling tabular file ingestion.This PR enhances the
file.pymodel by introducing thetabIngestfield. This field is set toTrueby default but can be disabled if the error occurs. TheREADME.mdhas been updated to include information about this new field, as well as a troubleshooting section to guide users on how to resolve the issue.Update: This PR includes a fix to prevent empty requests to
replaceFilesoraddFilesas pointed out by @landreev in IQSS/dataverse#11265 (comment)Update 2: This PR includes a fix to handle ZIP files correctly in two ways: