[AIRFLOW-5303] Use project_id from GCP credentials

[mik-laj]

Currently project_id is read correctly only if project_id is configured in extra.

However, the default project_id can also be configured in other place.:

• Application Default Credentials - It not only provides credentials, but also tries to find a project id. More: https://cloud.google.com/docs/authentication/production
• service account - It may contain information about the project id

It makes much easier to configure the connection during development
Currently, I need to setup connection using following command:

export AIRFLOW_CONN_google_cloud_default='google-cloud-platform://?extra__google_cloud_platform__key_path=%2Fkeys%2Fsa.json&extra__google_cloud_platform__project=polidea-airflow'

but I would like to be able to configure the connection using the following command:

gcloud auth application-default login

or when I use a service account:

export GOOGLE_APPLICATION_CREDENTIALS=/keys/sa.json

CC: @ryanyuan

---

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-5303
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[mik-laj]

I wonder if this piece of code should not be transferred to the _get_credentials_and_project_id method. Then this method will never return an invalid value and it will be easier to use it.

[mik-laj]

Updated.

[kaxil]

LGTM but would be good if we can add the test case mentioned in last comment

[kaxil]

Can we add a test case for this?

(1) Mock project_id as None from credential object & being overriden by project_id from connection?

[mik-laj]

I added a 8 tests. I am testing a lot of different cases now.

[kaxil]

👍

[kaxil]

Suggested change

	**{'reutnr_value.project_id': "PROJECT_ID"}
	**{'return_value.project_id': "PROJECT_ID"}

[mik-laj]

Fixed. I also added missing assertion.

[kaxil]

LGTM

[kaxil]

(mock_from_service_account_file.return_value, 'PROJECT_ID') -> The first item is credential block?

[mik-laj]

Yes. This is a credentials object because the from_service_account_info method is the Credentials class factory method. It's allways return Credentials object. This PR does not change the behavior of it.

All tests pass locally, but let's wait for Travis.