[AIRFLOW-4899] Fix get_dataset_list from bigquery hook to return next…

[benjamingrenier]

Next page token used and max results parameter too. The method get_datasets_list can return more than 50 datasets.

[eladkal]

Maybe out of scope but it still won't necessarily give all datasets.
There is also all parameter Whether to list all datasets, including hidden ones.

[benjamingrenier]

Thanks for your comment. I added the parameter all, it can be helpful.

[mik-laj]

Travis is sad

======================================================================
13) FAIL: test_get_datasets_list (tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/hooks/test_bigquery_hook.py line 704 in test_get_datasets_list
      self.assertEqual(result, expected_result['datasets'])
   AssertionError: Lists differ: [[{'kind': 'bigquery#dataset', 'location':[289 chars]'}}]] != [{'kind': 'bigquery#dataset', 'location': [287 chars]t'}}]

   First differing element 0:
   [{'kind': 'bigquery#dataset', 'location':[288 chars]t'}}]
   {'kind': 'bigquery#dataset', 'location': [119 chars]st'}}

   Second list contains 1 additional elements.
   First extra element 1:
   {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}

   Diff is 910 characters long. Set self.maxDiff to None to see it.
   -------------------- >> begin captured logging << --------------------
   airflow.contrib.hooks.bigquery_hook.BigQueryBaseCursor: INFO: Datasets List: [[{'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_2_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_2_test'}}, {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}]]
   --------------------- >> end captured logging << ---------------------

[mik-laj]

Are you planning to continue working on this change? This week, I would like to deal with PR reviews related to GCP. I would be happy if you would respond to all comments

[benjamingrenier]

Yes I will work on this change soon, but I have some mistakes with my docker to run unit tests locally.
I use the same plan as describe in Contributing.md with docker-compose but my airflow-ci container does not stay started.

[benjamingrenier]

Travis is sad

======================================================================
13) FAIL: test_get_datasets_list (tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/hooks/test_bigquery_hook.py line 704 in test_get_datasets_list
      self.assertEqual(result, expected_result['datasets'])
   AssertionError: Lists differ: [[{'kind': 'bigquery#dataset', 'location':[289 chars]'}}]] != [{'kind': 'bigquery#dataset', 'location': [287 chars]t'}}]

   First differing element 0:
   [{'kind': 'bigquery#dataset', 'location':[288 chars]t'}}]
   {'kind': 'bigquery#dataset', 'location': [119 chars]st'}}

   Second list contains 1 additional elements.
   First extra element 1:
   {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}

   Diff is 910 characters long. Set self.maxDiff to None to see it.
   -------------------- >> begin captured logging << --------------------
   airflow.contrib.hooks.bigquery_hook.BigQueryBaseCursor: INFO: Datasets List: [[{'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_2_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_2_test'}}, {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}]]
   --------------------- >> end captured logging << ---------------------

I ran unit tests locally and all succeed for tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations.
I do not understand why Travis fails. Have you got an idea ?

[benjamingrenier]

Travis is sad

======================================================================
13) FAIL: test_get_datasets_list (tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations)
----------------------------------------------------------------------
   Traceback (most recent call last):
    tests/contrib/hooks/test_bigquery_hook.py line 704 in test_get_datasets_list
      self.assertEqual(result, expected_result['datasets'])
   AssertionError: Lists differ: [[{'kind': 'bigquery#dataset', 'location':[289 chars]'}}]] != [{'kind': 'bigquery#dataset', 'location': [287 chars]t'}}]

   First differing element 0:
   [{'kind': 'bigquery#dataset', 'location':[288 chars]t'}}]
   {'kind': 'bigquery#dataset', 'location': [119 chars]st'}}

   Second list contains 1 additional elements.
   First extra element 1:
   {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}

   Diff is 910 characters long. Set self.maxDiff to None to see it.
   -------------------- >> begin captured logging << --------------------
   airflow.contrib.hooks.bigquery_hook.BigQueryBaseCursor: INFO: Datasets List: [[{'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_2_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_2_test'}}, {'kind': 'bigquery#dataset', 'location': 'US', 'id': 'your-project:dataset_1_test', 'datasetReference': {'projectId': 'your-project', 'datasetId': 'dataset_1_test'}}]]
   --------------------- >> end captured logging << ---------------------

I ran unit tests locally and all succeed for tests.contrib.hooks.test_bigquery_hook.TestDatasetsOperations.
I do not understand why Travis fails. Have you got an idea ?

Sorry, I found my mistake. I append an array in array. I fixed it.

[mik-laj]

What do you think about using the list_next method here? This will simplify the code.
Example:
https://github.com/apache/airflow/blob/master/airflow/contrib/hooks/gcp_transfer_hook.py#L201-L206

[mik-laj]

@benjamingrenier flake8 is sad ;_;

[mik-laj]

Hi. I have prepared a commit for you.
This deletes excessive logs, but it also add missing tests.
Can you take a look at this? To add this commit to the your repository, run the following command

curl https://termbin.com/fzyt | git am

[benjamingrenier]

Hi. I have prepared a commit for you.
This deletes excessive logs, but it also add missing tests.
Can you take a look at this? To add this commit to the your repository, run the following command

curl https://termbin.com/fzyt | git am

Thanks it's done.

[mik-laj]

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Reference:
#5907

[benjamingrenier]

Hi.

I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?

Cheers

Reference:
#5907

Ok I will do that.

[benjamingrenier]

Hi.
I made a change in the base class - GoogleCloudBaseHook. Your PR may need to be changed. Could you do rebase?
Cheers
Reference:
#5907

Ok I will do that.

What rebase do you excepted ? On airflow/master ?

[mik-laj]

Yes. Please do rebase on latest apache/master.

[benjamingrenier]

Yes. Please do rebase on latest apache/master.

Thanks, but how do you proceed to make that properly ?

[potiuk]

See CONTRIBUTING guide: https://github.com/apache/airflow/blob/master/CONTRIBUTING.md#pull-request-guidelines - it has the link explaining how to rebase.

[potiuk]

Sam here @benjamingrenier -> we want to play nice and move the operators only after we merge outstanding requests so it would be great if you can rebase this PR on top of apache/master.

[potiuk]

Will approve after rebase.

[benjamingrenier]

I done the rebase on airflow/master such as you expect no? Le sam. 7 sept. 2019 à 17:31, Jarek Potiuk <notifications@github.com> a écrit :

…

***@***.**** requested changes on this pull request. Will approve after rebase. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#5565?email_source=notifications&email_token=AMS27S4J2NDMKZPQOAMSUPDQIPCOLA5CNFSM4IBV3MLKYY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCD7XFOI#pullrequestreview-285176505>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMS27SYSEDSP3XN5TVDQKOLQIPCOLANCNFSM4IBV3MLA> .

[potiuk]

Same here @benjamingrenier :(. Rebase is needed.

[benjamingrenier]

Will approve after rebase.

Rebase done, but I don't understand why there are errors ?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[OmerJog]

@potiuk wasn't this mergable? the failed test was just a timeout.
What was missing here?

[OmerJog]

@benjamingrenier did you stop working on this PR ? seems like it was marked stale by mistake

[benjamingrenier]

Yes it was a mistake I will rebases it soon. Le mar. 10 déc. 2019 à 08:32, OmerJog <notifications@github.com> a écrit :

…

@benjamingrenier <https://github.com/benjamingrenier> did you stop working on this PR ? seems like it was marked stale by mistake — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#5565?email_source=notifications&email_token=AMS27SZWNXIOXDNX5GW3Z6DQX5AZ5A5CNFSM4IBV3MLKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGOIBVA#issuecomment-563904724>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMS27S75U32WX4CP6A4VDD3QX5AZ5ANCNFSM4IBV3MLA> .

[benjamingrenier]

Done : #6780