Too many requests error for dataset import from Zenodo

[jen-thomas]

Describe the bug
HTTPError: 429 Client Error: TOO MANY REQUESTS for import of files from Zenodo (see error below)

Link to project
https://renkulab.io/projects/ACE-ASAID/cruise-track-legs0-4

To Reproduce
renku dataset import 10.5281/zenodo.3369130
(select y to download set of files)

Expected behavior
Files should be imported from Zenodo.

Screenshots and/or execution output

Traceback (most recent call last):
  File "[...]/renku/cli/exception_handler.py", line 119, in main
    result = super().main(*args, **kwargs)
  File "[...]/renku/cli/exception_handler.py", line 90, in main
    return super().main(*args, **kwargs)
  File "[...]/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "[...]/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "[...]/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/cli/dataset.py", line 659, in import_
    progress=_DownloadProgressbar
  File "[...]/renku/core/commands/client.py", line 103, in new_func
    result = ctx.invoke(method, client, *args, **kwargs)
  File "[...]/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "[...]/renku/core/commands/dataset.py", line 584, in import_dataset
    total_size=total_size,
  File "[...]/renku/core/commands/dataset.py", line 240, in add_to_dataset
    progress=progress,
  File "[...]/renku/core/management/datasets.py", line 240, in add_data_to_dataset
    progress=progress
  File "[...]/renku/core/management/datasets.py", line 425, in _add_from_urls
    files.extend(future.result())
  File "[...]/concurrent/futures/_base.py", line 428, in result
    return self.__get_result()
  File "[...]/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "[...]/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "[...]/renku/core/management/datasets.py", line 442, in _add_from_url
    progress_class=progress
  File "[...]/renku/core/management/datasets.py", line 959, in _download
    request.raise_for_status()
  File "[...]/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: TOO MANY REQUESTS for url: https://zenodo.org/api/files/c353c22d-783b-4246-9327-54d9f3afde38/ACE_1_14.txt

Run environment (please complete the following information):
Renku version: 0.9.1
OS: Linux (#1 SMP Mon Oct 22 10:40:32 EDT 2018)
Python: 3.7.6
Jupyterlab, Firefox

Additional context
Add any other context about the problem here.

[jsam]

Hey Jen,

Thank you for submitting the issue with us.

I just took a look at this issue and it seems that the dataset you are trying to import contains quite a lot of small files which leads to a renku hitting the rate limit of 14 req/s.

I will send a PR for this later today so the fix should be available in the next release.

[jen-thomas]

Yeah that's correct. I should have probably zipped the files first before uploading them.

Anyway, thanks for taking a look.