How can I make cancellable uploading file to GCP storage?

[sstepashka]

Sometimes uploading is very slow. Uploading small files sometimes requires couple of seconds, sometimes about 15 mins. I think, The good solution is make request with timeout or cancel request then retry to upload. I see, In current implementation I cant provide timeout or cancellation to upload/download methods. Could you help me with it?

Thank you!

[tseaver]

@sstepashka You are correct that Blob.upload_from_file provides no mechanism for timing out its request. You might be able to accomplish your goal by calling Blob.create_resumable_upload_session to get a signed upload URL, and then post the file in chunks to that URL.

[dhermes]

@sstepashka I recommend checking out https://github.com/GoogleCloudPlatform/google-resumable-media-python

Let me know if you have any questions about using it after check out the docs: https://googlecloudplatform.github.io/google-resumable-media-python/0.1.1/google.resumable_media.requests.html

[sstepashka]

@dhermes How i can provide timeout for resumable upload?

[dhermes]

You would do it on your transport. There is currently no way to send extra arguments through to transport.request() but you could do it by making a custom transport:

import google.auth.transport.requests as tr_requests

class TimeoutSession(tr_requests.AuthorizedSession):

    def __init__(self, *args, **kwargs):
        self.timeout = kwargs.pop('timeout')  # Makes timeout a required kwarg
        super(TimeoutSession, self).__init__(*args, **kwargs)

    def request(self, *args, **kwargs):
        kwargs['timeout'] = self.timeout
        return super(TimeoutSession, self).request(*args, **kwargs)

[sstepashka]

So, How I can using custom downloader for Blob (inside GCP environment)?

[dhermes]

Just pip install google-auth google-resumable-media, then using the snippet above to define TimeoutSession:

import google.auth

# You may want other scopes, but read-only is enough for downloading.
ro_scope = u'https://www.googleapis.com/auth/devstorage.read_only'
credentials, _ = google.auth.default(scopes=(ro_scope,))
transport = TimeoutSession(credentials, timeout=25)

media_url = (
    u'https://www.googleapis.com/download/storage/v1/b/'
    u'yourbucket/o/yourobject?alt=media')
download = ChunkedDownload(
    media_url, chunk_size, stream)
while not download.finished:
    response = download.consume_next_chunk(transport)

(Most of this is just copied from the docs I mentioned above.)

[lukesneeringer]

Hello!
This issue appears to be resolved to the best of my ability to discern, so I am going to close it. Feel free to reopen if I am incorrect.