download_url for amazon is slow

[rijobro]

Describe the bug
One of the tutorial downloads Task09_Spleen.tar, which is 1.5GB. This took 1m45 via wget and over 2 hours with MONAI's download_url.

To Reproduce

1. wget https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar
2. monai.apps.download_and_extract("https://msd-for-monai.s3-us-west-2.amazonaws.com/Task09_Spleen.tar", "Task09_Spleen.tar", "~/test")
3. Observe download speeds

[Nic-Ma]

Hi @rijobro ,

Seems wget uses multi-threads to download data which is not supported in our API so far.
I will try to add multi-threads support for downloading next week.

Thanks.

[rijobro]

Thanks. Looking online, this post (python2) claims urllib to be as fast as wget.

I wondered if we were getting slowed down because we loop in 1MB steps? https://github.com/Project-MONAI/MONAI/blob/master/monai/apps/utils.py#L116

[Nic-Ma]

Hi @rijobro ,

Do you know that's the buffer size of wget? Or maybe you can try to download again with a bigger buffer size?

Thanks.

[rijobro]

I have no idea, maybe it does something clever depending on the file size. I'll assign this one to me, and if I can't solve it, then maybe you can have a look at the multi-threading?

[Nic-Ma]

OK, sounds good.
Thanks.

[rijobro]

From the stackoverflow link I posted, it looks like we can just call urlretrieve with a callback function. Maybe that way, we let it take care of everything and just use the callback for updating the progress bar.