Storage: Timeout when upload file using google.cloud.storage.Blob.upload_from_filename()

[vfa-minhtv]

Environment details

OS: MacOS 10.15.1
Python: Python 3.7.4
Google-cloud version:

google-api-core==1.16.0
google-api-python-client==1.7.11
google-auth==1.11.2
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.4.0
google-cloud-core==1.3.0
google-cloud-error-reporting==0.32.1
google-cloud-firestore==1.5.0
google-cloud-kms==1.0.0
google-cloud-logging==1.14.0
google-cloud-storage==1.26.0
google-cloud-translate==1.7.0
google-resumable-media==0.5.0
google-translate==0.1
googleapis-common-protos==1.6.0

Steps to reproduce

1. Prepare a file with size >300MB
2. Run blob.upload_from_filename("path/on/storage", "path/of/big/file/on/local")

Stack trace

Traceback (most recent call last):
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 987, in send
    self.sock.sendall(data)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/ssl.py", line 1034, in sendall
    v = self.send(byte_view[count:])
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/ssl.py", line 1003, in send
    return self._sslobj.write(data)
socket.timeout: The write operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1244, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1290, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1239, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/http/client.py", line 987, in send
    self.sock.sendall(data)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/ssl.py", line 1034, in sendall
    v = self.send(byte_view[count:])
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/ssl.py", line 1003, in send
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-cdc889c11775>", line 4, in <module>
    "data/tmp/averaging.joblib", None)
  File "/Users/dualeoo/PycharmProjects/mlweb-ml/mlweb_ml/firestore/google_storage.py", line 30, in upload
    blob.upload_from_filename(file_path_on_local, content_type, predefined_acl=predefined_acl)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1342, in upload_from_filename
    predefined_acl=predefined_acl,
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1287, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1197, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/cloud/storage/blob.py", line 1144, in _do_resumable_upload
    response = upload.transmit_next_chunk(transport)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/resumable_media/requests/upload.py", line 425, in transmit_next_chunk
    retry_strategy=self._retry_strategy,
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/resumable_media/requests/_helpers.py", line 136, in http_request
    return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/resumable_media/_helpers.py", line 150, in wait_and_retry
    response = func()
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/google/auth/transport/requests.py", line 317, in request
    **kwargs
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/Users/dualeoo/miniconda3/envs/mlweb-ml/lib/python3.7/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out'))

Expected result

No timeout error

Actual result

The upload timeout after 1 minute

[crwilcox]

You state you ran blob.upload_from_filename("path/on/storage", "path/of/big/file/on/local"), which doesn't match the signature:
upload_from_filename(filename, content_type=None, client=None, predefined_acl=None)

I created a repro, but unfortunately this is working well for me.

import datetime

# Using version 1.26.0 of google-cloud-storage
from google.cloud import storage

BUCKET_NAME = "your-bucket"

client = storage.Client()
bucket = client.bucket(BUCKET_NAME)

#filename = "4Gb.txt"

# 300Mb.txt generated with 
# `❯ dd if=/dev/urandom of=300Mb.txt bs=1048576 count=300`
filename = "300Mb.txt"

# Write file if necessary.
blob = bucket.blob(filename)
if not blob.exists():
    print(f"Writing {filename}")
    start = datetime.datetime.now()
    blob.upload_from_filename(filename)
    end = datetime.datetime.now()
    print(f"Wrote {filename} {end-start}")

# Read file
print(f"Reading {filename}")
start = datetime.datetime.now()
blob.download_to_filename(f"downloaded-{filename}")
end = datetime.datetime.now()
print(f"Read {filename} {end-start}")

For good measure, I tried the same thing with a 4Gb file as well:

❯ python write_and_read.py
Writing 300Mb.txt
Wrote 300Mb.txt 0:00:46.983565
Reading 300Mb.txt
Read 300Mb.txt 0:00:39.064956

❯ python write_and_read.py
Writing 4Gb.txt
Wrote 4Gb.txt 0:09:43.198637
Reading 4Gb.txt
Read 4Gb.txt 0:14:05.202351

This was done from a new virtual environment and python 3.8

❯ pip freeze
cachetools==4.0.0
certifi==2019.11.28
chardet==3.0.4
google-api-core==1.16.0
google-auth==1.11.2
google-cloud-core==1.3.0
google-cloud-storage==1.26.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
idna==2.9
protobuf==3.11.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pytz==2019.3
requests==2.23.0
rsa==4.0
six==1.14.0
urllib3==1.25.8

[ElectricSwan]

I'm also getting the same error on both Ubuntu 18.0.4 with Python 3.6.9 and Windows 10 with Python 3.8.0, both using google-cloud-storage 1.26.0
The timeout happens after 60 seconds.
I'm limited to around 800 kbps upload speed, so for me that gives a timeout for any files larger than 6 MB.
Any uploads that complete within 60 seconds are successful.

[crwilcox]

@ElectricSwan , what is the code you are running? The sample code I provided runs well beyond 60 seconds. Simplified it is

from google.cloud import storage

BUCKET_NAME = "your-bucket"

client = storage.Client()
bucket = client.bucket(BUCKET_NAME)

# `❯ dd if=/dev/urandom of=4Gb.txt bs=1048576 count=4096`
filename = "4Gb.txt"

# Write file if necessary.
blob = bucket.blob(filename)
blob.upload_from_filename(filename)

[ElectricSwan]

@crwilcox , I just ran your simplified code in a virtual env with Python 3.6.9 on Ubuntu 18.04, and I get the timeout at 60 seconds.

I get the exact same timeout error with your simplified code on my Windows 10 PC running Python 3.8.0.

Here is the pip freeze from the Ubuntu PC, which appears identical to yours;

cachetools==4.0.0
certifi==2019.11.28
chardet==3.0.4
google-api-core==1.16.0
google-auth==1.11.2
google-cloud-core==1.3.0
google-cloud-storage==1.26.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
idna==2.9
pkg-resources==0.0.0
protobuf==3.11.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pytz==2019.3
requests==2.23.0
rsa==4.0
six==1.14.0
urllib3==1.25.8

Here's my pip freeze from the Windows PC;

astroid==2.3.3
cachetools==4.0.0
certifi==2019.11.28
chardet==3.0.4
colorama==0.4.3
google-api-core==1.16.0
google-auth==1.11.2
google-cloud-core==1.3.0
google-cloud-storage==1.26.0
google-resumable-media==0.5.0
googleapis-common-protos==1.51.0
idna==2.8
isort==4.3.21
lazy-object-proxy==1.4.3
mccabe==0.6.1
protobuf==3.11.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pylint==2.4.4
pytz==2019.3
requests==2.22.0
rsa==4.0
six==1.14.0
urllib3==1.25.8
wrapt==1.11.2

The only potentially relevant differences that I can see between my Windows and Ubuntu boxes are that my Windows box has slightly older versions of;

idna==2.8
requests==2.22.0

but the result is the same on both platforms; timeout after 60 seconds.

Here is my stacktrace on the Ubuntu PC, which is almost identical to the stacktrace submitted by @vfa-minhtv

Traceback (most recent call last):
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.6/ssl.py", line 975, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.6/ssl.py", line 944, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.6/ssl.py", line 642, in write
    return self._sslobj.write(data)
socket.timeout: The write operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.6/http/client.py", line 1254, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1300, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1249, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.6/http/client.py", line 1075, in _send_output
    self.send(chunk)
  File "/usr/lib/python3.6/http/client.py", line 996, in send
    self.sock.sendall(data)
  File "/usr/lib/python3.6/ssl.py", line 975, in sendall
    v = self.send(byte_view[count:])
  File "/usr/lib/python3.6/ssl.py", line 944, in send
    return self._sslobj.write(data)
  File "/usr/lib/python3.6/ssl.py", line 642, in write
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', timeout('The write operation timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "simple_test.py", line 18, in <module>
    blob.upload_from_filename(filename)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1342, in upload_from_filename
    predefined_acl=predefined_acl,
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1287, in upload_from_file
    client, file_obj, content_type, size, num_retries, predefined_acl
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 1193, in _do_upload
    client, stream, content_type, size, num_retries, predefined_acl
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/cloud/storage/blob.py", line 987, in _do_multipart_upload
    response = upload.transmit(transport, data, object_metadata, content_type)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/resumable_media/requests/upload.py", line 106, in transmit
    retry_strategy=self._retry_strategy,
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/resumable_media/requests/_helpers.py", line 136, in http_request
    return _helpers.wait_and_retry(func, RequestsMixin._get_status_code, retry_strategy)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/resumable_media/_helpers.py", line 150, in wait_and_retry
    response = func()
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/google/auth/transport/requests.py", line 317, in request
    **kwargs
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/requests/sessions.py", line 530, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/julie/Programming/Python3/bu_to_gcs/env/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', timeout('The write operation timed out',))

I also just tried installing google-cloud-storage without a virtual env under Ubuntu (just in case there was something peculiar under a virtual env), but I still get the same timeout after 60 seconds.

[vfa-minhtv]

You state you ran blob.upload_from_filename("path/on/storage", "path/of/big/file/on/local"), which doesn't match the signature:
upload_from_filename(filename, content_type=None, client=None, predefined_acl=None)

Hi @crwilcox!

So sorry. I gave a wrong example. My actual code is:

[vfa-minhtv]

@ElectricSwan

The timeout happens after 60 seconds.
I'm limited to around 800 kbps upload speed, so for me that gives a timeout for any files larger than 6 MB.
Any uploads that complete within 60 seconds are successful.

Exact same thing happen for me. My internet is also limited so 60 seconds timeout is insufficient to finish uploading

[aborzin]

@vfa-minhtv, I have been experiencing similar timeout issues on my macOS and Win platforms with google-cloud-storage==1.26.0. However, the timeout issues are inconsistent and apparently dependent on the network speed. As already mentioned in this thread, typically it fails with very slow upload speed.

I checked the code and found that any data stream of 8 MB and larger will _do_resumable_upload(..) which sends the data stream in chunks (which absolutely makes sense to support slow network connectivity):

        if size is not None and size <= _MAX_MULTIPART_SIZE:
            response = self._do_multipart_upload(
                client, stream, content_type, size, num_retries, predefined_acl
            )
        else:
            response = self._do_resumable_upload(
                client, stream, content_type, size, num_retries, predefined_acl
            )

However, the chunk size is not set in the initialization call and therefore will be set to some predefined default value:

        if chunk_size is None:
            chunk_size = self.chunk_size
            if chunk_size is None:
                chunk_size = _DEFAULT_CHUNKSIZE

This default value is set to 100 MB:

_DEFAULT_CHUNKSIZE = 104857600  # 1024 * 1024 B * 100 = 100 MB

So you must have ~ 15 MBps upload speed to complete the request within 1 min, which is apparently the default timeout (see http://www.meridianoutpost.com/resources/etools/calculators/calculator-file-download-time.php for quick upload time calculations).

I made a test and reduced the _DEFAULT_CHUNKSIZE value to 10 MB, which solved my issues.
I hope this will help and we will be able to control the chunk size based on our environment parameters.