gdrive: download: stream & add progress

[casperdcl]

• patch underlying httplib2 for gdrive progress
• Also consider using something other than httplib2/pydrive2 (e.g. use http.client directly or requests which both support iteration) to avoid loading the whole file into RAM
  • depends on file.GetContentFile: stream to disk, add callback iterative/PyDrive2#30 (pydrive>=1.4.11)

    • Let there be no more in-memory downloads. Let the files stream until the hard drives overfloweth. Let RAM be ignored, and let us answer the call to progress.

    • use new ApiRequestError.GetField()
    • use new GetContentFile(callback=Tqdm.update_to)
  • depends on pydrive2 v1.4.11 conda-forge/pydrive2-feedstock#10
• test
• part of Google Drive support further enhancements #2865
  • related api: support streaming from Google Drive remotes #3408

[codecov]

Codecov Report

Merging #3722 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #3722   +/-   ##
=======================================
  Coverage   90.80%   90.80%           
=======================================
  Files         158      158           
  Lines       10582    10582           
=======================================
  Hits         9609     9609           
  Misses        973      973           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d00bd4d...58490e0. Read the comment docs.

[casperdcl]

P.S. @shcheklein I don't think we need to bother about GetContentFile(remove_bom=True) since we never Upload({"convert": True}).

[shcheklein]

@casperdcl yep, that's why I was not that worried about bom in the first place (e.g. performance).

[casperdcl]

Also #2865 says

• and update docs

for the file streaming bit... What docs?

[shcheklein]

@casperdcl yeah, this one doesn't solve the API streaming. It is the next level of avoiding writing stuff to the disk - dcv.api.open and dvc get. E.g. for S3 they call get_signed_url internally, instead of _download.

[casperdcl]

yeah, this one doesn't solve the API streaming.

ah I see

[shcheklein]

Patches like this:

• less (?) API calls + UI improved + download faster ...

just warm my heart :) Less code we have better I sleep. Thanks 🙏

[casperdcl]

yes just pushed a final quick update (because of the deferred total) to ensure consistent formatting so a tiny bit more loc :)

+11
-21

Pretty sure there's at least one (if not 3) fewer API calls. #2511 would tell.

[casperdcl]

also please don't merge until conda-forge/pydrive2-feedstock#10 (which doesn't exist yet)

[casperdcl]

ok good to go

[efiop]

Thank you @casperdcl ! 🙏