Get defaults from CLI for GCE and OCI if not provided by TheRealFalcon · Pull Request #87 · canonical/pycloudlib

TheRealFalcon · 2021-01-07T23:17:50Z

Stop requiring to pass in project for gce and compartment for oci

TheRealFalcon · 2021-01-08T16:12:14Z

@lucasmoura Thanks for the tips. I've pushed a new version that uses the google credentials if provided, and also provides a default (the default for gcloud) that can be overridden.

OddBloke

Thanks for this, James! I think your initial implementation of the Google stuff is closer to what we want: comments inline, of course.

OddBloke · 2021-01-08T17:33:06Z

 logging.getLogger('googleapiclient.discovery').setLevel(logging.WARNING)

+DEFAULT_CREDENTIALS_PATH = Path.home() / '.config' / 'gcloud' / \
+    'application_default_credentials.json'


I'm not 100% sure that this is the correct path: it does exist on my desktop, but if I initialise a new gcloud inside a container (to the point where I can list my instances), I don't see it:

# tree .config/gcloud/ .config/gcloud/ ├── access_tokens.db ├── active_config ├── cache │ └── daniel.watkins@canonical.com ├── config_sentinel ├── configurations │ └── config_default ├── credentials.db ├── gce ├── legacy_credentials │ └── daniel.watkins@canonical.com │ └── adc.json └── logs └── 2021.01.08 ├── 17.27.25.441108.log ├── 17.27.37.404385.log ├── 17.27.41.272001.log ├── 17.28.11.613997.log ├── 17.28.16.418870.log ├── 17.28.24.243745.log └── 17.28.26.915831.log

adc.json does look like the same file format (and adc is presumably an abbreviation for application_default_credentials?).

OddBloke · 2021-01-08T17:41:33Z

+            if project is None:
+                with credentials_path.open() as f:
+                    config = json.load(f)
+                if 'project_id' in config:


I'm not sure this is correct either; in both my desktop and in-container ADC file, I have only 4 keys: client_id, client_secret, refresh_token, and type.

When prompted to set my project in the container, it was set in .config/gcloud/configurations/config_default (which appears to be selected by default being in .config/gcloud/active_config).

(I believe that this file is used for authenticating your user to the Google APIs, which is orthogonal to the project being used: users have access to many projects.)

OddBloke · 2021-01-08T17:50:12Z

+            except FileNotFoundError as e:
+                raise exception from e
+            if not result.ok:
+                raise exception


This is nit-level, so don't worry too much about it: will this include the output from the failing command? (That would save folks from having to re-run the command to see what the exact error was.)

TheRealFalcon · 2021-01-08T18:25:23Z

Thanks for the review @OddBloke . In UAC, they don't install gcloud for their pycloudlib usage, so I thought I could cover multiple use cases this way, but it looks like file locations and keys don't stay consistent across versions. Since shelling out seems less likely to change than file/key locations, I'll go back to shelling out. If you don't have gcloud installed, the old way should still work.

stdout/stderr to exceptions.

OddBloke · 2021-01-13T15:56:10Z

When running this in combination with canonical/cloud-init#757, I see the following error when I run CLOUD_INIT_PLATFORM=gce pytest tests/integration_tests/modules/test_write_files.py:

Traceback (most recent call last):
  File "/home/daniel/dev/cloud-init/tests/integration_tests/conftest.py", line 236, in class_client
    with _client(request, fixture_utils, session_cloud) as client:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/daniel/dev/cloud-init/tests/integration_tests/conftest.py", line 210, in _client
    with session_cloud.launch(
  File "/home/daniel/dev/cloud-init/tests/integration_tests/clouds.py", line 165, in launch
    pycloudlib_instance = self._perform_launch(kwargs)
  File "/home/daniel/dev/cloud-init/tests/integration_tests/clouds.py", line 135, in _perform_launch
    pycloudlib_instance = self.cloud_instance.launch(**launch_kwargs)
  File "/home/daniel/dev/pycloudlib/pycloudlib/gce/cloud.py", line 206, in launch
    operation = self.compute.instances().insert(
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/googleapiclient/_helpers.py", line 130, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/googleapiclient/http.py", line 844, in execute
    resp, content = _retry_request(
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/googleapiclient/http.py", line 164, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google_auth_httplib2.py", line 189, in request
    self.credentials.before_request(
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google/auth/credentials.py", line 133, in before_request
    self.refresh(request)
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google/oauth2/credentials.py", line 200, in refresh
    access_token, refresh_token, expiry, grant_response = _client.refresh_grant(
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google/oauth2/_client.py", line 248, in refresh_grant
    response_data = _token_endpoint_request(request, token_uri, body)
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google/oauth2/_client.py", line 124, in _token_endpoint_request
    _handle_error_response(response_body)
  File "/home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google/oauth2/_client.py", line 60, in _handle_error_response
    raise exceptions.RefreshError(error_details, response_body)
google.auth.exceptions.RefreshError: ('invalid_grant: Bad Request', '{\n  "error": "invalid_grant",\n  "error_description": "Bad Request"\n}')

Earlier in the run, I see:

------ Captured log setup ------
2021-01-13 15:48:48 INFO      integration_testing:clouds.py:73 Detected image: image_id=focal os=ubuntu release=focal
2021-01-13 15:48:50 WARNING   google.auth._default:_default.py:346 No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable
2021-01-13 15:48:50 INFO      integration_testing:clouds.py:73 Detected image: image_id=focal os=ubuntu release=focal

And in the warnings summary, I see:

  /home/daniel/.virtualenvs/cloud-init/lib/python3.8/site-packages/google/auth/_default.py:69: UserWarning: Your application has authenticated using end user credentials from Google Cloud SDK without a quota project. You might receive a "quota exceeded" or "API not enabled" error. We recommend you rerun `gcloud auth application-default login` and make sure a quota project is added. Or you can use service accounts instead. For more information about service accounts, see https://cloud.google.com/docs/authentication/
    warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)

After running that command, not only do the tests run successfully, but I both have a ~/.config/gcloud/application_default_credentials.json and it has quota_project_id set.

So perhaps what we actually need to do is check for quota_project_id in there and if it's missing guide people to run gcloud auth application-default login? (Apologies for all the flipping back and forth between things here. 😞)

TheRealFalcon · 2021-01-14T20:41:25Z

@OddBloke I don't think we provide guidance for setting up any cloud CLIs, do we? We kind of leave that as an exercise for the user. Since you were given a warning telling you what you need to do, I'm thinking we don't need to do anything else atm. While I'm not against giving guidance for setting up the various CLIs, I think that's beyond the scope of this particular PR.

Thoughts?

OddBloke

This LGTM, thanks!

TheRealFalcon added the wip label Jan 8, 2021

lucasmoura reviewed Jan 8, 2021

View reviewed changes

Comment thread pycloudlib/gce/cloud.py

TheRealFalcon force-pushed the cloud-defaults branch from 27b34ce to 1eb8f59 Compare January 8, 2021 16:10

TheRealFalcon removed the wip label Jan 8, 2021

TheRealFalcon mentioned this pull request Jan 8, 2021

Use more cloud defaults in integration tests canonical/cloud-init#757

Merged

3 tasks

OddBloke suggested changes Jan 8, 2021

View reviewed changes

TheRealFalcon commented Jan 8, 2021

View reviewed changes

Comment thread pycloudlib/gce/cloud.py

OddBloke reviewed Jan 8, 2021

View reviewed changes

Comment thread pycloudlib/gce/cloud.py

TheRealFalcon added 3 commits January 12, 2021 13:24

Get defaults from CLI for GCE and OCI if not provided

a3fd0b1

[squash] Comments. Updating GCE to shell out and adding

e8297da

stdout/stderr to exceptions.

[squash] call subp correctly

e56278d

TheRealFalcon force-pushed the cloud-defaults branch from 698946f to e56278d Compare January 12, 2021 19:25

TheRealFalcon requested a review from OddBloke January 12, 2021 19:26

lucasmoura approved these changes Jan 13, 2021

View reviewed changes

OddBloke approved these changes Jan 20, 2021

View reviewed changes

TheRealFalcon merged commit 3980508 into master Jan 20, 2021

TheRealFalcon deleted the cloud-defaults branch January 20, 2021 21:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get defaults from CLI for GCE and OCI if not provided#87

Get defaults from CLI for GCE and OCI if not provided#87
TheRealFalcon merged 3 commits into
masterfrom
cloud-defaults

TheRealFalcon commented Jan 7, 2021

Uh oh!

Uh oh!

TheRealFalcon commented Jan 8, 2021

Uh oh!

OddBloke left a comment

Uh oh!

OddBloke Jan 8, 2021

Uh oh!

OddBloke Jan 8, 2021

Uh oh!

OddBloke Jan 8, 2021

Uh oh!

TheRealFalcon commented Jan 8, 2021

Uh oh!

Uh oh!

Uh oh!

OddBloke commented Jan 13, 2021

Uh oh!

TheRealFalcon commented Jan 14, 2021 •

edited

Loading

Uh oh!

OddBloke left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

TheRealFalcon commented Jan 7, 2021

Uh oh!

Uh oh!

TheRealFalcon commented Jan 8, 2021

Uh oh!

OddBloke left a comment

Choose a reason for hiding this comment

Uh oh!

OddBloke Jan 8, 2021

Choose a reason for hiding this comment

Uh oh!

OddBloke Jan 8, 2021

Choose a reason for hiding this comment

Uh oh!

OddBloke Jan 8, 2021

Choose a reason for hiding this comment

Uh oh!

TheRealFalcon commented Jan 8, 2021

Uh oh!

Uh oh!

Uh oh!

OddBloke commented Jan 13, 2021

Uh oh!

TheRealFalcon commented Jan 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

OddBloke left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

TheRealFalcon commented Jan 14, 2021 •

edited

Loading