Only load fields that are needed from Google API for Big Query schema.

[jezdez]

Refs #785.

I used the Google API Explorer to build the values for the fields parameters: e.g. https://developers.google.com/apis-explorer/#p/bigquery/v2/bigquery.tables.get

More info for the fields parameter: https://developers.google.com/api-client-library/python/guide/performance#partial-response-fields-parameter

[jezdez]

It'd be neat to try this out eventually, not sure how I can do this locally to be honest :(

[emtwo]

@jezdez I haven't looked into the cause of the failing tests here, but I've tried this out locally and it's working in that less data is passed along per request, which is good.

However, I'm not entirely sure that this will resolve the issue #785 for a couple of reasons.

1. It did not provide a clear improvement in speed for fetching BigQuery schema since the main bottleneck seems to be making a request for every single table, and this PR does not change that. Unfortunately, it seems that this is just the nature of the BigQuery schema API.

2. I could not reproduce the error @jasonthomas was seeing {"error": {"message": "Error retrieving schema.", "code": 2}} neither on master or in this PR when testing with a BigQuery data source that has ~270 tables. The only way I could get that error to show up is by using an invalid JSON key file when setting up the data source.

That said, I think once these tests are fixed, we can land this PR anyways (perhaps it should be done upstream?) It may or may not fix the issue but it's still useful to be sending around less data.

I also think if we could connect to the BigQuery data source locally that is failing on stage, it might be easier to reproduce and debug the issue. Is there any chance we could do that, @jasonthomas?

[emtwo]

For reference, here are some differences in data that is passed along with the change in this PR:

[Before] Fetching a dataset

[{
    u 'kind': u 'bigquery#dataset',
    u 'location': u 'US',
    u 'id': u 'cosmic-antenna-155218:babynames',
    u 'datasetReference': {
        u 'projectId': u 'cosmic-antenna-155218',
        u 'datasetId': u 'babynames'
    }
}]

[After] Fetching a dataset

[{
    u 'datasetReference': {
        u 'datasetId': u 'babynames'
    }
}]

[Before] Fetching table names

[{
    u 'creationTime': u '1541093458826',
    u 'kind': u 'bigquery#table',
    u 'type': u 'TABLE',
    u 'id': u 'cosmic-antenna-155218:babynames.copy',
    u 'tableReference': {
        u 'projectId': u 'cosmic-antenna-155218',
        u 'tableId': u 'copy',
        u 'datasetId': u 'babynames'
    }
}, ...]

[After] Fetching table names

[{
    u 'tableReference': {
        u 'tableId': u 'copy'
    }
}, ...]

[Before] Fetching a table schema:

{
    u 'kind': u 'bigquery#table',
    u 'creationTime': u '1541093458826',
    u 'location': u 'US',
    u 'tableReference': {
        u 'projectId': u 'cosmic-antenna-155218',
        u 'tableId': u 'copy',
        u 'datasetId': u 'babynames'
    },
    u 'numRows': u '33228',
    u 'numBytes': u '637183',
    u 'etag': u '3AWXArPupz0SpiQIXDh8RQ==',
    u 'numLongTermBytes': u '0',
    u 'lastModifiedTime': u '1541093458826',
    u 'type': u 'TABLE',
    u 'id': u 'cosmic-antenna-155218:babynames.copy',
    u 'selfLink': u 'https://www.googleapis.com/bigquery/v2/projects/cosmic-antenna-155218/datasets/babynames/tables/copy',
    u 'schema': {
        u 'fields': [{
            u 'type': u 'STRING',
            u 'name': u 'name',
            u 'mode': u 'NULLABLE'
        }, {
            u 'type': u 'STRING',
            u 'name': u 'gender',
            u 'mode': u 'NULLABLE'
        }, {
            u 'type': u 'INTEGER',
            u 'name': u 'count',
            u 'mode': u 'NULLABLE'
        }]
    }
}

[After] Fetching a table schema:

{
    u 'id': u 'cosmic-antenna-155218:babynames.copy',
    u 'schema': {
        u 'fields': [{
            u 'type': u 'STRING',
            u 'name': u 'name',
            u 'mode': u 'NULLABLE'
        }, {
            u 'type': u 'STRING',
            u 'name': u 'gender',
            u 'mode': u 'NULLABLE'
        }, {
            u 'type': u 'INTEGER',
            u 'name': u 'count',
            u 'mode': u 'NULLABLE'
        }]
    }
}

[jasonthomas]

Is there any chance we could do that, @jasonthomas?

@emtwo I've sent you an email with the json key. The project id information is also listed in the json key file.

One thing to mention is that the redash documentation recommends either using the Big Query Admin role or creating a custom role for the service account with the permissions. I've created a custom role with the following permissions:

    "bigquery.datasets.get",
    "bigquery.jobs.create",
    "bigquery.jobs.get",
    "bigquery.jobs.list",
    "bigquery.jobs.listAll",
    "bigquery.jobs.update",
    "bigquery.tables.get",
    "bigquery.tables.list",

The documentation also includes 'bigquery.datasets.list' and 'bigquery.tables.getData' which is not included in the above permissions. 'bigquery.datasets.list' does not exist and ''bigquery.tables.getData' would give access to all table data [2]. Instead the service account user has access to specific datasets/tables.

[1] https://redash.io/help/data-sources/setup/bigquery
[2] https://cloud.google.com/bigquery/docs/access-control

[jasonthomas]

As per today's STMO meeting we decided to test the BigQuery data source with a a gcp service account that contained role 'BigQuery Admin' to rule out this is a permission issue.

I created a new service account, created a new Big Query data source on stage stmo and was unable to view the schema. The error was the same as above {"error": {"message": "Error retrieving schema.", "code": 2}}. I also ran a query against the datasource and that was successful and returned data.

[emtwo]

Note: bug #785 has a fix that was merged upstream: getredash#3382

[jezdez]

Hey @emtwo, @jasonthomas Do you think this would be a good idea to open a PR upstream?

[emtwo]

Do you think this would be a good idea to open a PR upstream?

Yes definitely.

[jezdez]

Closing in favor of getredash#3673.