Datastore: field types of array are always showing indexed regardless of excludeFromIndexes

[DaveyEdwards]

Saving array's shows it is indexed (not all the actual items in the array, just the field), even when excludeFromIndexes: true.

I am not sure if this is normal behavior and the whole array is not actually stored in the index. I had asked this originally in gstore-node issues and he said to ask here :)

You can see the "circles" field shows indexed

Inside the "circles" it is showing all array items are excluded from index

Environment details

• OS: Windows 10
• Node.js version: 6.11.2
• npm version: 3.10.10
• google-cloud-node version:
  Replecated on google-cloud: 0.56.0
  and @google-cloud/datastore: 1.1.0
  I originally was using gstore-node and saw this problem, and now I am using just raw datastore and still have it.

Steps to reproduce

Seed database with a field of array

My test function

createEntity([
  {
    name: 'kind',
    value: 'Pages',
    excludeFromIndexes: true,
  },
  {
    name: '_id',
    value: '6',
  },
  {
    name: 'title',
    value: 'my first title',
  },
  {
    name: 'dateCreated',
    value: new Date(),
  },
  {
    name: 'string',
    value: 'A long string goes here',
    excludeFromIndexes: true,
  },
  {
    name: 'circles',
    value: ['1', '2', '3', '4'],
    excludeFromIndexes: true,
  },
]);

The actual save function

export default async function createEntity(entity) {
  let response = null;
  let kind = null;
  let dsKey = null;

  try {
    entity.map((entityFeilds) => {
      if (entityFeilds.name === '_id') {
        dsKey = entityFeilds.value;
        if (dsKey === '') {
          dsKey = null;
        }
        return dsKey;
      }
      if (entityFeilds.name === 'kind') {
        kind = entityFeilds.value;
        return kind;
      }
      return null;
    });
    const key = datastoreClient.key([kind, dsKey]);
    const data = entity;
    const newEntity = {
      key,
      data,
    };

    response = await datastoreClient.save(newEntity);
  } catch (err) {
    console.error('save err', err);
  }
  return response;
}

[stephenplusplus]

Thanks for the detailed report! @pcostell, does this look normal?

[pcostell]

I believe this is an inconsistency between the google-cloud-node library and the API. The API explicitly states that ListValue properties must be indexed, and that indexing is controlled at the individual value level.

This is mostly an artifact of Cloud Datastore's App Engine API, which treats these properties not as lists but as multi-valued properties. Effectively, circles = [1, 2, 3] is really understood by the backend as three separate properties, all named circles: circles=1, circles=2, circles=3.

Because of this, each value is treated as a separate property when looking at exclude_from_indexes. It's also evident in the query API, where you would query for this entity with circles = 1 as opposed to circles CONTAINS 1.

[DaveyEdwards]

Thanks for the detailed post @pcostell . Would unindexed ListValue types cause any increase to total index size then?

Also should this issue be closed or would the be something to fix in the future

[pcostell]

unindexed vs indexed on a ListValue wouldn't make any difference on total index size, because the list as a whole is not indexed (you can't query WHERE A = [1, 2, 3]).

Though note that in the node client library, unindexing a ListValue is equivalent to unindexing all the individual values. So unindexing the ListValue here would reduce the total index size.

I'll let the node team decide on if they want to keep this bug open and align the client more with the API, otherwise @stephenplusplus can close it.

[sebelga]

Hello,
I am receiving an "Exclude from indexes cannot be set on a list value" when I set excludeFromIndexes on an Array value. It used to work before, has there been any change on the node api?
thanks!

EDIT:
It seems that the error comes after updating to 1.1.0.
What is now the correct way to remove indexes for a whole Array with the new declaration:

var entity = {
  key: datastore.key('Company'),
  excludeFromIndexes: [
    'circles' // this throws the Error mentioned
  ],
  data: {
    circles: ['1', '2', '3']
  }
};

[beaulac]

The current implementation only works for nested properties on an object in an array. It unfortunately does not work for primitive values in an array.

The exclude from indexes syntax only handles the nested-property case:

var entity = {
  key: datastore.key('Company'),
  data: {
    circles: [{radius: '1'}, {radius: '2'}]
  },
  excludeFromIndexes: [
    'circles[].radius' // this would work
  ],
};

There's no current way to refer to directly to a member in an array. This example throws "cannot read property 'properties' of undefined", because it recurses using entityValue which is not present on non-entity values.

var entity = {
  key: datastore.key('Company'),
  data: {
    circles: ['1', '2']
  },
  excludeFromIndexes: [
    'circles[]'
  ],
};

[sebelga]

@beaulac
I was expecting that - as for embedded entities where it works - doing this

const entity = {
    data: { myArray: ['1', '2', '3'] },
    excludeFromIndexes: ['myArray'],
    key: {}
}

would be the same as doing this (which does work on primitive values)

const entity = {
    data: [{
        name: 'myArray',
        value: ['1', '2', '3'],
        excludeFromIndexes: true,
    }],
    key: {}
}

But it does not behave like that. It would be more consistent if it would in my opinion.
I detailed it here on my comment #2510 (comment)