Slow gcloud storage upload for small files?

[srlowe]

I'm seeing upload times of over 6000ms for small 10kB files. Please could someone confirm if this is expected for Google cloud uploads, or is it more likely there is some client problem happening?

I've tried disabling resumable and validation, and this improves things a bit, but not substantially. Our upload code looks like this:

    var options = {
        resumable : false,
        validation: false
    };

    var remoteWriteStream = bucket.file(fileName).createWriteStream(options);
    remoteWriteStream.on('finish', function(){ callback(); });
    remoteWriteStream.on('error',  function(err){ callback(err); });
    sourceStream.pipe(remoteWriteStream);

[stephenplusplus]

Can you run another upload immediately after the first one completes and measure the time difference?

[srlowe]

Sure. It seems subsequent calls are faster. Here's the same upload repeated 5x in a row (~10kB with resume and validation off):

4935ms, 3336ms, 2336ms, 2551ms, 2695ms.

Still seems on the slow side to me - what do you think?

Maybe my expectations are wrong though. I suppose there are overheads like auth etc, which disadvantage small files.

[jgeewax]

This definitely seems wrong to me. Uploading 10k shouldn't be taking 1000
ms. That said, GCS is optimized for throughput (aka, large file overall
upload time) rather than latency (small file overall upload time), so it's
possible (just shitty...)

[srlowe]

@jgeewax - Right, that was my feeling too.

Does anyone know if there's some kind of debug mode, which might give some visibility into what's happening, time-wise?

[stephenplusplus]

With my own tests, I complete uploads consistently under 1s. It went up to around 3 if I uploaded to a bucket overseas. Any chance location could be playing a role?

[jgeewax]

That's a good point. If you're in the US and your bucket is in Asia (or vice versa), that would explain the extra latency (mainly wire time)

[srlowe]

Thanks, I tried both US and EU storage locations, but times seemed similar (I'm in EU).

[jgeewax]

@mziccard (who is also in EU and may be able to help us with some debugging :P) Can you do a traceroute to be sure that your upload request is actually being routed to endpoints in the EU?

[stephenplusplus]

If you have gsutil CLI tool installed, you can see what times you get there, just to see if there are any huge differences:

time gsutil cp 10kfile.txt gs://your-bucket

[stephenplusplus]

Here's the test I've used: https://gist.github.com/stephenplusplus/15d0d7e96a40887c74aa

[srlowe]

@jgeewax , @stephenplusplus - Thanks, I'll test traceroute, gsutil and that test script and report back.

Worth noting that I'm doing this from an external machine, not google-compute, and am specifying credentials like so (which may be a factor):

var storage = gcloud.storage({
    projectId   : xxx,
    credentials : {
          client_email : yyy,
          private_key  : zzz
});

[stephenplusplus]

The credentials will cause a delay, but will only affect the first request. Have you had a chance to run any more tests?

@mziccard any chance you can run the sample here and report your findings? https://gist.github.com/stephenplusplus/15d0d7e96a40887c74aa

[stephenplusplus]

@bgSosh I'm going to close this out as this is likely a geographic issue, beyond what this library can help with. If you have a chance to run some more tests or find anything we might not have thought of that would help, please re-open.