Datastore error on great amount of data

[mik115]

Hi,
i'm facing a real strange issue.
I've a datastore save request that goes well when the entity is small,
but when we try to save a great amount of data we receive this error

{ handle: 2,
type: 'error',
 className: 'Error',
 constructorFunction: { ref: 8 },
 protoObject: { ref: 9 },
 prototypeObject: { ref: 1 },
 properties: 
  [ { name: 'stack', attributes: 2, propertyType: 3, ref: 1 },
    { name: 'message', attributes: 2, propertyType: 0, ref: 10 },
    { name: 'code', propertyType: 0, ref: 11 },
    { name: 'metadata', propertyType: 0, ref: 12 } ],
 text: 'Error: Bad Request' }

that breaks the node process with this stacktrace:

  < /Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:300
  <   throw new Error('Unsupported field value, ' + value + ', was provided.');
  <   ^
  < Error: Unsupported field value, undefined, was provided.
  <     at Object.encodeValue (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:300:9)
  <     at /Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:652:24
  <     at Array.map (native)
  <     at Object.queryToQueryProto (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/entity.js:646:33)
  <     at makeRequest (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/request.js:495:21)
  <     at Object.makeRequest (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/common/util.js:579:21)
  <     at DestroyableTransform.<anonymous> (/Users/michelepini/crawler.analytics.igenius.net/node_modules/gcloud/lib/datastore/request.js:485:13)
  <     at DestroyableTransform.g (events.js:286:16)
  <     at emitOne (events.js:96:13)
  <     at DestroyableTransform.emit (events.js:188:7)

the error shows up when i try to save an entity with two string field, not indexed of 1MB of size
ie

{
  data: [0.9MB string not indexed]
  data2: [0.9MB string not indexex]
  [other strings indexed of no more than 40bit]
}

We chunck our data on string of less than 1MB to respect the maximum string length property on datastore.
I'm pretty sure that the issue is about the amount of data because saving only one chunk no error was raised.

There's an upper limit on the total entity size or there is some error?

Thanks in advance

[stephenplusplus]

Sorry you're having this issue :( Is it possible to put a small sample together, or otherwise the actual gcloud-node code you're using?

[mik115]

hi,
here is a really small test file to verify the behaviour.

        'use strict';

        const gcloud = require('gcloud')({
            projectId: [project ID],
            keyFilename: [keyfilePath]
        });
        const datastore = gcloud.datastore();
        const _ = require('lodash');

        var str = new Array(500000).join("a");

        debugger;

        function toDatastore(obj, indexedFields) {
            indexedFields = indexedFields || [];
            const results = [];
            Object.keys(obj).forEach((k) => {
                if (obj[k] === undefined) {
                    return;
                }
                results.push({
                    name: k,
                    value: obj[k],
                    excludeFromIndexes: indexedFields.indexOf(k) < 0
                });
            });
            return results;
        }

        function create(kind, data, indexedFields, cb) {
            const key = datastore.key([kind]);
            debugger;
            const entity = {
                key: key,
                data: toDatastore(data, indexedFields)
            };

            datastore.save(
                entity,
                (error) => {
                    debugger;
                    if (error) {
                        return cb ? cb(error) : error;
                    }

                    return cb ? cb(null) : true;
                }
            );
        };

        create("testKind", {
            data0: str,
            data1: str,
            data3: str,
            key: '250277808676421',
            key1: 'b4wbj0b5wy04',
            key2: 'sdjhfkahsdjkfhjkas',
            key3: "setstts",
            key4: "sdfasdfa"
        }, ["key", "key1", "key2", "key3", "key4"], (err, result) => {
            debugger;
        });

In this case the error was raised whth string of 500000 chars.
Thanks

[stephenplusplus]

Thanks for putting that together. @pcostell any insight on this one? Thanks!

[pcostell]

It looks like you've exceeded the maximum size for an entity: https://cloud.google.com/datastore/docs/concepts/limits

Not sure why you don't get a nicer error though. @stephenplusplus can see what the request/response from Datastore is?

[stephenplusplus]

@pcostell would that relate to #1378? I think as of now, we don't have a way to pull out any detailed error messaging.

[pcostell]

Ah yes that's likely the issue -- although I don't understand why a Bad request turns into the given node stack trace.

[stephenplusplus]

Yeah, I don't understand that either. That didn't happen for me from the sample code provided, maybe it was from some temporarily malformed code, @mik115?

[mik115]

No temporary malformed and yes I reached the upper entity limit size of 1MB.
I've already fixed the problem chunking my data and saving them in separate entities.

Thanks anyway for the really quick responses

[stephenplusplus]

Great :) We're working on providing more helpful error messages in #1378. Feel free to re-open this if you think there's more we can do.

[kirillgroshkov]

It breaks node process for me as well! With same error, on same line:

Error: Unsupported field value, undefined, was provided.
  <     at Object.encodeValue

Ideas how to fix it?

[stephenplusplus]

@kirillgroshkov could you share reproduction code?