Datastore/dataset question

[abelino]

is there a way to get the count/length of the entities that matched a query without actually receiving the entities data? or can something like that be achieved via BigQuery?

[jgeewax]

I think you might do this with a keys-only query. @stephenplusplus , how do we do that in gcloud-node?

[abelino]

looking at GQL I think it would be the following: SELECT __key__ FROM myKind going to look at dataset class right now.

but what if the keys being returned is a ridiculous amount?

[stephenplusplus]

You can run a keys-only query by using select('__key__') when building your query. Here's a start to finish example:

var dataset = gcloud.datastore.dataset();
var personKey = dataset.key(['Person', 'Stephen']);

dataset.save({
  key: personKey,
  data: {
    alias: 'stephenplusplus'
  }
}, function(err) {
  var normalQuery = dataset.createQuery('Person');
  var keysOnlyQuery = normalQuery.select('__key__');

  dataset.runQuery(keysOnlyQuery, function(err, results) {
    // results = [
    //   {
    //     key: {
    //       path: ['Person', 'Stephen']
    //     },
    //
    //     data: {}
    //   }
    // ]
  });

  dataset.runQuery(normalQuery, function(err, results) {
    // results = [
    //   {
    //     key: {
    //       path: ['Person', 'Stephen']
    //     },
    //
    //     data: {
    //       alias: 'stephenplusplus'
    //     }
    //   }
    // ]
  });
});

[abelino]

will the following construct work:

var query = dataset.createQuery('Person').filter('age >', '20').select('__key__');

dataset.runQuery(query, function(err, results) {
    // results = [
    //   {
    //     key: {
    //       path: ['Person', 'Stephen']
    //     },
    //
    //     data: {}
    //   },
    //   {
    //     key: {
    //       path: ['Person', 'Alice']
    //     },
    //
    //     data: {}
    //   },
    //   {
    //     key: {
    //       path: ['Person', 'Bob']
    //     },
    //
    //     data: {}
    //   }
    // ]
  });

will test later on, and come back if I have any questions. Thanks for the help guys @stephenplusplus @jgeewax

[dhermes]

This will still return all the keys, which is O(n) in the payload size, when you really just want one integer, which will be O(1).

[jgeewax]

Is there a way to ask for COUNT(key)?

[abelino]

@dhermes Exactly, that is the ideal solution, but not sure if COUNT func is supported by GQL.

[stephenplusplus]

I couldn't find anything in the docs about just getting the resulting length of a query. @pcostell might know if this is possible?

[pcostell]

Nope, Datastore has no way to count. Even in the SDKs which offer a count method, these are all O(n).

---

Here be dragons and all that:

I would not recommend implementing this, but the "fastest" way to do this would be to run a your query with projection = '__key__' with offset = 1000 and limit = 0. Then keep running this counting the number of skipped_results and using the skipped_cursor to get the next batch.

This way, Datastore doesn't return any results, just the counts. However, you still have to pay for each batch (the key projection makes it only one read per batch though). More importantly, Datastore still has to load all the results, so the time it takes (and the cost) will scale with the size of your count.

[stephenplusplus]

@abelino sorry to skip your earlier question. Your example filtering the same query should work just fine.

And thanks as always, @pcostell!

@jgeewax would it be a good idea to add an example of a keys-only query to the docs?

[jgeewax]

Yea, I think mentioning that somewhere (ever so briefly) would be a good idea...