State of "excludeFromIndexes" (datastore)

[stephenplusplus]

Copied from original issue: googleapis/google-cloud-node#2770

@­sebelga
December 12, 2017 6:35 AM

Hello,

I have seen that since version 1.1.0 of the datastore api there is a new syntax for the "excludeFromIndexes" allowing embedded entities to have their properties excluded from indexes.

I am wondering if this new syntax is currently supported as it is not yet in the "master" documentation (it only appears under the "1.1.0" version).

I am sending the following payload to save an entity but all of the properties are marked as "indexed" in the google cloud console. Thus the "excludeFromIndexes" is not working as expected.

{
    "key": {
        "namespace": "my.namespace",
        "kind": "MyEntity",
        "path": [
            "MyEntity",
            null
        ]
    },
    "data": [
        {
            "name": "systems",
            "value": {
                "a": "123c",
                "b": "456c"
            }
        },
        {
            "name": "phoneNumber",
            "value": "0123456789"
        },
        {
            "name": "createdAt",
            "value": "2017-12-12T06:16:35.145Z"
        }
    ],
    "excludeFromIndexes": [
        "phoneNumber",
        "systems",
        "createdAt",
        "systems.a",
        "systems.b"
    ]
}

Thank for any help on this!

[stephenplusplus]

December 12, 2017 8:58 PM

Hey @sebelga! I don't believe excludeFromIndexes is compatible with the array-notation format of data.

We originally only supported the old excludeFromIndexes markup if the user provided an array. That was so we could separate the excludeFromIndexes property from the properties within a user's entity. This style was a problem when nested properties needed to declare excludeFromIndexes. To handle that situation, we created the more robust, top-level excludeFromIndexes array, like you've defined. To opt-in, you just need to convert the data array to an object:

{
    "data": {
        "systems": {
            "a": "123c",
            "b": "456c"
        },
        "phoneNumber": "0123456789"
        "createdAt": "2017-12-12T06:16:35.145Z"
    }
}

Let me know if that doesn't fix the issue, or if that style for data creates any problems.

[stephenplusplus]

@­sebelga
December 13, 2017 10:40 AM

Hi @stephenplusplus ! Ok I will give it a try and come back with my results.

I've found inconsistencies (see my comment here googleapis/google-cloud-node#2510 (comment)) and haven't been able to find the proper way to both declare "excludeFromIndexes" for a simple properties, for Arrays (exclude all their properties) and for a properties on an embedded entity.

I will give a try with what you comment and let you know. 👍

[stephenplusplus]

@­beaulac
December 14, 2017 5:06 PM

If #2773 is merged, it would allow this syntax to also support (non-nested) arrays.

[sebelga]

I have been doing some test and now I remember why I mixed the old "array" style for the data with the new more robust excludeFromIndexes array declaration.

If we declare an "array" property as being "excludeFromIndexes" an error is returned:

"Exclude from indexes cannot be set on a list value"

So this entity will throw an error

{
    "key": {
        "namespace": "my.namespace",
        "kind": "MyEntity",
        "path": [
            "MyEntity",
            null
        ]
    },
    "data": {
        "phoneNumber": "123-456",
        "systems": {
            "abc": 123
        },
        "tags": [
            "abc",
            "def"
        ],
        "createdAt": "2017-12-18T18: 40: 50.287Z"
    },
    "excludeFromIndexes": [
        "phoneNumber",
        "systems",
        "createdAt",
        "systems.abc",
        "tags" // tags is an Array. It throws an error
    ]
}

Will #2773 fix this issue?
thanks! 👍

[stephenplusplus]

@sebelga I believe this is resolved with the latest release. Could you give it a shot and let me know? Thanks!

[sebelga]

@stephenplusplus ok thanks! I am out on holidays, I'll give it a try next week and let you know.
Happy new year!

[stephenplusplus]

Sounds great and happy holidays! I'll close the issue for now, but we can re-open if needed :)

[sebelga]

@stephenplusplus I confirm that it does work, thanks! And thank you @beaulac for the #18 👍

[trollkotze]

This is not fixed!

I am using @google-cloud/datastore v1.4.0, which includes the changes from #2773 (old repo) merged in #18.

@sebelga

Will #2773 fix this issue?
thanks! 👍

@stephenplusplus

I believe this is resolved with the latest release. Could you give it a shot and let me know? Thanks!

@sebelga

I confirm that it does work, thanks! And thank you @beaulac for the #18 👍

I tested it with the very same example entity as @sebelga posted above:

{
    "key": {
        "namespace": "my.namespace",
        "kind": "MyEntity",
        "path": [
            "MyEntity",
            null
        ]
    },
    "data": {
        "phoneNumber": "123-456",
        "systems": {
            "abc": 123
        },
        "tags": [
            "abc",
            "def"
        ],
        "createdAt": "2017-12-18T18: 40: 50.287Z"
    },
    "excludeFromIndexes": [
        "phoneNumber",
        "systems",
        "createdAt",
        "systems.abc",
        "tags" // tags is an Array. It throws an error
    ]
}

I get the same old error: "INVALID_ARGUMENT: Exclude from indexes cannot be set on a list value".