Can't store entities with embedded entities which have properties > 1500 bytes

[bedeoverend]

Seems that excluding a top level property doesn't exclude an embedded entities properties from being indexed - on top of that, I don't believe there's a way to exclude an embedded entities individual properties, that would solve my problem too.

Steps to reproduce

Just use this script (I'm running the datastore emulator)

const Datastore = require('@google-cloud/datastore');
const datastore = new Datastore({ projectId: 'datastore-test' });

datastore.insert({
  key: datastore.key('User'),
  data: [
    {
      name: 'description',
      value: {
        text: Buffer.alloc(1501, 'a').toString(),
      },
      excludeFromIndexes: true
    }
  ]
}, (err, result) => {
  if (err) {
    console.error(err);
  } else {
    console.log(result.data);
  }
});

I'd expect there to be no error - the embedded entity ({ text: '...' }) shouldn't be indexed I believe? But it throw an error saying:

Error: The value of property "text" is longer than 1500 bytes.

Environment

• OS: OS X Sierra
• Node.js: v6.7.0
• npm: 3.10.3
• @google-cloud/datastore: v0.6.0

[stephenplusplus]

Thanks for reporting.

@pcostell is it possible in Datastore to exclude individual properties within an Entity from indexing?

[bedeoverend]

FWIW I've had a play with the datastore save method to support the explicit syntax on nested properties e.g.

datastore.insert({
  key: datastore.key('Stuff'),
  data: [{
    name: 'description',
    value: [{
      name: 'text',
      value: Buffer.alloc(1501, 'a').toString(),
      excludeFromIndexes: true
    }, {
      name: 'long',
      value: true
    }]
  }]
}, handler);

obviously though that wouldn't work as an interface as it would break any properties that have array values e.g. this would break:

datastore.insert({
  key: datastore.key('Stuff'),
  data: [{
    name: 'foo',
    value: [ 1, 2, 3 ]
  }]
}, handler);

---

For reference, this is the hacked reduction (previously here) function to make this work:

function encodeValues(acc, data) {
  var value;

  if (is.array(data.value)) {
    value = {
      entityValue: {
        properties: data.value.reduce(encodeValues, {})
      }
    };
  } else {
    value = entity.encodeValue(data.value);

    if (is.boolean(data.excludeFromIndexes)) {
      var excluded = data.excludeFromIndexes;
      var values = value.arrayValue && value.arrayValue.values;

      if (values) {
        values = values.map(propAssign('excludeFromIndexes', excluded));
      } else {
        value.excludeFromIndexes = data.excludeFromIndexes;
      }
    }
  }

  acc[data.name] = value;

  return acc;
}

[pcostell]

Excluding a top-level entity value will exclude it and all its sub-values from indexing. However, it still must be a valid entity. This means that properties that cannot be indexed should be marked accordingly.

In the Datastore proto API, this is done by passing excludeFromIndexes at the appropriate levels. This is important from an API perspective because it means an entity value is a valid entity (and could be moved to a top-level entity if desired).

[bedeoverend]

@pcostell so am I right in thinking that if a top level value was marked as to be excluded from indexing, it would be safe for this SDK to mark all sub-values to be excluded? Given that even if not explicitly excluded they will still be as they're nested inside a value that was excluded.

[pcostell]

That would be safe, but it would represent a departure from the design of the API.

[bedeoverend]

I'm guessing it would be hard to introduce a syntax that would allow excludeFromIndexes on sub properties, while maintaining backwards compatibility @stephenplusplus?

Until that's achieved, would passing down exclude when set on the top level property be viable? I've got the beginnings of it in a branch, happy to make a PR

[stephenplusplus]

Yes, this would be breaking, however we could try something like:

datastore.insert({
+ raw: true,
  key: datastore.key('Stuff'),
  data: [
    {
      name: 'description',
      value: [
        {
          name: 'text',
          value: Buffer.alloc(1501, 'a').toString(),
          excludeFromIndexes: true
        },
        {
          name: 'long',
          value: true
        }
      ]
    }
  ]
}, handler);

If raw is set, it would be required that all values inside of data are specified in the longhand/verbose syntax. Any chance I'm overlooking something that makes this unfeasible?

[bedeoverend]

@stephenplusplus I don't think that would work for storing array values?

What about the raw flag needed the explicit encoded value returned by encodeValue? I think that's the proto value? You could also open up the encodeValue function to help users e.g:

datastore.insert({
  key: datastore.key('Stuff'),
  data: {
    description: {
      text: Buffer.alloc(1501, 'a').toString(),
      long: true
    },
    values: [ 1, 2, 3 ]
  }
}

the user could then transform into a raw value by:

datastore.insert({
  raw: true,
  key: datastore.key('Stuff'),
  data: Datastore.encodeValue({
    description: Datastore.encodeValue({
      text: Datastore.encodeValue(Buffer.alloc(1501, 'a').toString(), { exclude: true }),
      long: Datastore.encodeValue(true)
    }),
    values: Datastore.encodeValue([ 1, 2, 3 ])
  })
}

Not sure if I've got the syntax right, but that's the idea. Also more likely the user would do it programmatically.

[stephenplusplus]

@stephenplusplus I don't think that would work for storing array values?

Sorry, I've had my head in many APIs lately. Would you mind showing a bit of code that would highlight this problem?

[bedeoverend]

@stephenplusplus sure, so if I wanted to store the following JS Object:

var personEntity = {
  name: "Mary",
  metadata: {
    bio: "..." // Some long string > 1500 bytes
  },
  subjects: [{
    name: "history",
    credits: 2
  }, {
    name: "french",
    credits: 4
  }, {
    name: "maths",
    credits: 4
  }]
}

I need to have entity.metadata.bio excluded from indexes. Using the raw flag and syntax provided above, I could do this, however I couldn't store the subjects prop as an array e.g.

var rawPersonEntity = [{
  name: "name",
  value: "Mary"
}, {
  name: "metadata",
  value: [{
    name: "bio",
    value: "...", // Some long string > 1500 bytes
    excludeFromIndexes: true
  }]
}, {
  name: "subjects",
  value: [] // What do I do here? An array will indicate explicit syntax of { name, value }
}]

[stephenplusplus]

Got it, thank you! Would it make sense to recurse the data property?

var rawPersonEntity = {
  key: datastore.key('Person'),
  data: [
    {
      name: 'name',
      value: 'Mary'
    },
    {
      name: 'metadata',
      data: [
        {
          excludeFromIndexes: true,
          name: 'bio',
          value: '...'
        }
      ]
    },
    {
      name: 'subjects',
      value: [
        {
          name: 'history',
          credits: 2
        },
        // ...
      ]
    }
  ]
}

Otherwise, I think the raw flag on a nested level could have the same effect:

var rawPersonEntity = {
  key: datastore.key('Person'),
  data: [
    {
      name: 'name',
      value: 'Mary'
    },
    {
      name: 'metadata',
      raw: true,
      value: [
        {
          excludeFromIndexes: true,
          name: 'bio',
          value: '...'
        }
      ]
    },
    {
      name: 'subjects',
      raw: false, // (the default value, but explicitly set for demonstration)
      value: [
        {
          name: 'history',
          credits: 2
        },
        // ...
      ]
    }
  ]
}

Do either of these have any quirks, and if not, which is better? If neither, we can still keep trying :)