renku dataset export to Zenodo connection error

[jen-thomas]

Describe the bug
When exporting a dataset to Zenodo, received a connection error.

To Reproduce
Steps to reproduce the behavior:
renku dataset export cruise-track-intermediate zenodo

Expected behavior
A clear and concise description of what you expected to happen.
Dataset to be exported to Zenodo. Up until the point of error, I can confirm that the dataset had been created on Zenodo and the files up until that point had been exported as well.

Screenshots
Please see attached text file with the full bug report

Desktop (please complete the following information):
Renku version: 0.6.1.dev3
OS: Linux (#1 SMP Debian 4.19.67-2 (2019-08-28))
Python: 3.7.3

Additional context
Dataset was being exported from a well-connected server. Tried again and the same error occurred with the same dataset. Another dataset had been exported earlier in the day from the same server with no problems.
renku_export_dataset_connection_error.txt

[Panaetius]

Does this problem still exist?

From the error message it sounds like an issue on Zenodos end (the server is closing the connection).

[jen-thomas]

Yes exactly the same still happens.

[Panaetius]

Are any of the files larger than 100MB? That's the limit per file for Zenodo and exceeding it can result in this error.

See zenodo/zenodo#1764

The new upload API that supports sizes up to 50GB is not stable yet, though we should move renku over to that as soon as it's finalized.

[jen-thomas]

Ah yes that is what the problem will be. Thanks very much for spotting that. I'll try and do the DOI through Zenodo directly.

I presume the old API is used for retrieiving files from Zenodo (with renku dataset import) as well as the deposition, so it is possible that this file size limit would also apply when doing that operation?

Thanks again @Panaetius for your help.

[Panaetius]

I'm not sure if downloading files has the same limits, the Zenodo documentation just mentions them being limited in general, without going into detail.

Glad I could help, even if it didn't really solve your problem 😅

I've put this issue on hold for now.