[Cloud] Perform cloud discovery with API Version 2022-09-01 and support dataplaneEndpoints for Azure Stack

[jiasli]

Related command
az cloud

Description
Close #25815

Testing Guide

Populate dataplaneEndpoints from metadata:

az cloud register -n mycloud --endpoint-resource-manager https://management.azure.com/

Populate dataplaneEndpoints from local cloud config JSON file:

az cloud register -n mycloud --cloud-config @D:\cloud\cloud-config.json

[azure-client-tools-bot-prd]

️✔️acr

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️acs

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️advisor

️✔️latest

️✔️3.10

️✔️3.9

️✔️ams

️✔️latest

️✔️3.10

️✔️3.9

️✔️apim

️✔️latest

️✔️3.10

️✔️3.9

️✔️appconfig

️✔️latest

️✔️3.10

️✔️3.9

️✔️appservice

️✔️latest

️✔️3.10

️✔️3.9

️✔️aro

️✔️latest

️✔️3.10

️✔️3.9

️✔️backup

️✔️latest

️✔️3.10

️✔️3.9

️✔️batch

️✔️latest

️✔️3.10

️✔️3.9

️✔️batchai

️✔️latest

️✔️3.10

️✔️3.9

️✔️billing

️✔️latest

️✔️3.10

️✔️3.9

️✔️botservice

️✔️latest

️✔️3.10

️✔️3.9

️✔️cdn

️✔️latest

️✔️3.10

️✔️3.9

❌cloud

❌latest

❌3.10

Type	Test Case	Error Message	Line
Failed	test_cloud_scenario	E vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None During handling of the above exception, another exception occurred: E AssertionError: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None	azure/cli/command_modules/cloud/tests/latest/test_cloud.py:79

❌3.9

Type	Test Case	Error Message	Line
Failed	test_cloud_scenario	E vcr.errors.CannotOverwriteExistingCassetteException: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None During handling of the above exception, another exception occurred: E AssertionError: Can't overwrite existing cassette ('/mnt/vss/_work/1/s/src/azure-cli/azure/cli/command_modules/cloud/tests/latest/recordings/test_cloud_scenario.yaml') in your current record mode ('once'). No match for the request (<Request (GET) https://management.azure.com//metadata/endpoints?api-version=2022-09-01>) was found. Found 1 similar requests with 1 different matcher(s) : 1 - (<Request (GET) https://management.azure.com/metadata/endpoints?api-version=2015-01-01>). Matchers succeeded : ['method', 'scheme', 'host', 'port', 'path'] Matchers failed : _custom_request_query_matcher - assertion failure : None	azure/cli/command_modules/cloud/tests/latest/test_cloud.py:79

️✔️cognitiveservices

️✔️latest

️✔️3.10

️✔️3.9

️✔️config

️✔️latest

️✔️3.10

️✔️3.9

️✔️configure

️✔️latest

️✔️3.10

️✔️3.9

️✔️consumption

️✔️latest

️✔️3.10

️✔️3.9

️✔️container

️✔️latest

️✔️3.10

️✔️3.9

️✔️core

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️cosmosdb

️✔️latest

️✔️3.10

️✔️3.9

️✔️databoxedge

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️dla

️✔️latest

️✔️3.10

️✔️3.9

️✔️dls

️✔️latest

️✔️3.10

️✔️3.9

️✔️dms

️✔️latest

️✔️3.10

️✔️3.9

️✔️eventgrid

️✔️latest

️✔️3.10

️✔️3.9

️✔️eventhubs

️✔️latest

️✔️3.10

️✔️3.9

️✔️feedback

️✔️latest

️✔️3.10

️✔️3.9

️✔️find

️✔️latest

️✔️3.10

️✔️3.9

️✔️hdinsight

️✔️latest

️✔️3.10

️✔️3.9

️✔️identity

️✔️latest

️✔️3.10

️✔️3.9

️✔️iot

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️keyvault

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️kusto

️✔️latest

️✔️3.10

️✔️3.9

️✔️lab

️✔️latest

️✔️3.10

️✔️3.9

️✔️managedservices

️✔️latest

️✔️3.10

️✔️3.9

️✔️maps

️✔️latest

️✔️3.10

️✔️3.9

️✔️marketplaceordering

️✔️latest

️✔️3.10

️✔️3.9

️✔️monitor

️✔️latest

️✔️3.10

️✔️3.9

️✔️netappfiles

️✔️latest

️✔️3.10

️✔️3.9

️✔️network

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️policyinsights

️✔️latest

️✔️3.10

️✔️3.9

️✔️privatedns

️✔️latest

️✔️3.10

️✔️3.9

️✔️profile

️✔️latest

️✔️3.10

️✔️3.9

️✔️rdbms

️✔️latest

️✔️3.10

️✔️3.9

️✔️redis

️✔️latest

️✔️3.10

️✔️3.9

️✔️relay

️✔️latest

️✔️3.10

️✔️3.9

️✔️resource

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️role

️✔️latest

️✔️3.10

️✔️3.9

️✔️search

️✔️latest

️✔️3.10

️✔️3.9

️✔️security

️✔️latest

️✔️3.10

️✔️3.9

️✔️servicebus

️✔️latest

️✔️3.10

️✔️3.9

️✔️serviceconnector

️✔️latest

️✔️3.10

️✔️3.9

️✔️servicefabric

️✔️latest

️✔️3.10

️✔️3.9

️✔️signalr

️✔️latest

️✔️3.10

️✔️3.9

️✔️sql

️✔️latest

️✔️3.10

️✔️3.9

️✔️sqlvm

️✔️latest

️✔️3.10

️✔️3.9

️✔️storage

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️synapse

️✔️latest

️✔️3.10

️✔️3.9

️✔️telemetry

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

️✔️util

️✔️latest

️✔️3.10

️✔️3.9

️✔️vm

️✔️2018-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2019-03-01-hybrid

️✔️3.10

️✔️3.9

️✔️2020-09-01-hybrid

️✔️3.10

️✔️3.9

️✔️latest

️✔️3.10

️✔️3.9

[yonzhan]

Cloud enhancement

[TheOnlyWei]

Although the main post mentioned that a new nested group called dataplaneEndpoints is being supported, there should also be support for new suffixes and endpoints that is not passed by a cloudconfig.json file during az cloud register. Currently, Azure CLI is using a deprecated method to populate the hard-coded metadata which should be updated to read all endpoints and suffixes:

azure-cli/src/azure-cli/azure/cli/command_modules/cloud/custom.py

Line 69 in 091c6b1

_populate_from_metadata_endpoint(c, arm_endpoint)

The method is from this deprecated library:
https://github.com/Azure/msrestazure-for-python/blob/8e3def8c9927a349596e575de93df76de7a3a2b6/msrestazure/azure_cloud.py#L204

[jiasli]

there should also be support for new suffixes and endpoints that is not passed by a cloudconfig.json file during az cloud register.

If these new suffixes and endpoints are not passed by a cloudconfig.json file, how will Azure CLI even know them?

Currently, Azure CLI is using a deprecated method to populate the hard-coded metadata which should be updated to read all endpoints and suffixes.

We can certainly update _populate_from_metadata_endpoint and make no longer deprecated. That's why this PR copies this function from msrestazure to Azure CLI.

Reading all endpoints and suffixes is simply against our modeling principal which enforces all endpoints to be an attribute of the cloud object. Of course, we can leverage setattr to programmatically change the attributes of the cloud object, this defeats the purpose of modeling.

[TheOnlyWei]

@jiasli

If these new suffixes and endpoints are not passed by a cloudconfig.json file, how will Azure CLI even know them?

The new endpoints and suffixes will appear in the GET request to the metadata endpoint URL.

Reading all endpoints and suffixes is simply against our modeling principal which enforces all endpoints to be an attribute of the cloud object.

Is there a reason there is an exception made for reading all endpoints and suffixes from the cloudconfig.json but not from the metadata endpoint?
For example, the following code just adds all endpoints from cloudconfig.json:

azure-cli/src/azure-cli/azure/cli/command_modules/cloud/custom.py

Line 46 in 091c6b1

endpoints = cloud_args['endpoints']

[jiasli]

Is there a reason there is an exception made for reading all endpoints and suffixes from the cloudconfig.json but not from the metadata endpoint?

I didn't write that code, but I guess it is because cloudconfig.json is designed/used specifically for Azure CLI. On the other hand, metadata endpoint is merely a supplemental way to extract information we want.

[toefraz]

@jiasli Bumping this.

FYI 2022-09-01 is a breaking change for the metadata endpoint. Prior versions returned a JSON array of cloud endpoints for multiple environments. This version returns a single JSON object for the endpoints associated with the ARM instance being queried.

Is the goal to support both schemas with this for backwards compatibility?

[jiasli]

msrestazure current uses 2015-01-01 API version: https://management.azure.com/metadata/endpoints?api-version=2015-01-01

> az cloud register -n myazurestack --endpoint-resource-manager https://management.azure.com/ --debug
...
urllib3.connectionpool: https://management.azure.com:443 "GET //metadata/endpoints?api-version=2015-01-01 HTTP/1.1" 200 295