[Role] Migrate `azure-mgmt-authorization` SDK to Track 2 and bump API version to 2022-04-01 by jiasli · Pull Request #25452 · Azure/azure-cli

jiasli · 2023-02-14T08:26:26Z

Related command
az role

Description
Close #23372
Migrate azure-mgmt-authorization SDK to Track 2 and bump API Version to 2022-04-01:

role_assignments: 2020-04-01-preview -> 2022-04-01
role_definitions: 2018-01-01-preview -> 2022-04-01

Breaking changes in azure-mgmt-authorization SDK

For SDK version 0.61.0 (Track 1) -> 3.0.0 (Track 2):

1-1: definitions_client.config and assignments_client.config are changed to definitions_client._config and assignments_client._config.
1-2: custom_headers are changed to headers.
1-3: When a REST call fails, the error is changed from msrestazure.azure_exceptions.CloudError to azure.core.exceptions.HttpResponseError

For API version 2020-04-01-preview -> 2022-04-01:

2-1: RoleAssignmentsOperations changes list method to list_for_subscription.
2-2: RoleAssignmentCreateParameters.principal_type's default value is changed from None to User ([Role] RoleAssignmentProperties.principalType should not have default value User azure-rest-api-specs#21664)

For API version 2015-07-01:

3-1: ProviderOperationsMetadataOperations.get changes api_version from positional argument to keyword argument.

jiasli · 2023-02-14T08:32:07Z

src/azure-cli/azure/cli/command_modules/role/custom.py

    definitions_client = _auth_client_factory(cmd.cli_ctx, scope).role_definitions
    scope = _build_role_scope(resource_group_name, scope,
-                              definitions_client.config.subscription_id)
+                              definitions_client._config.subscription_id)


For breaking change 1-1, Track 2 now makes config a protected attribute as _config. I can't find a better way to extract subscription ID from the SDK client.

This pattern has already been used by resource command module:

azure-cli/src/azure-cli/azure/cli/command_modules/resource/custom.py

Lines 3551 to 3552 in 1b62d20

subscriptionId=serialize.url(

"self._config.subscription_id", self.rcf.resources._config.subscription_id, 'str'),

Another option is to let get_mgmt_service_client return the subscription ID during client creation:

azure-cli/src/azure-cli-core/azure/cli/core/commands/client_factory.py

Lines 50 to 51 in 21266e0

def get_mgmt_service_client(cli_ctx, client_or_resource_type, subscription_id=None, api_version=None,

aux_subscriptions=None, aux_tenants=None, credential=None, **kwargs):

but this is such a big breaking change that it will affect almost all command modules and extension which use get_mgmt_service_client to create a client.

This change was mentioned in doc: https://github.com/Azure/azure-cli/blob/dev/doc/track_2_migration_guidance.md#obtaining-subscription

However, the doc is not accurate as the subscription used to create the client (possibly via --subscription) may not be the same as the current subscription. Assuming the subscription used to create the client being the same as the current subscription can introduce bugs very difficult to track down.

yonzhan · 2023-02-14T08:40:22Z

Migrate azure-mgmt-authorization SDK to Track 2

jiasli · 2023-02-15T03:13:18Z

Affected modules: vm aro iot acs resource

jiasli · 2023-02-15T03:56:37Z

I accidentally discovered a problem with ams module due to breaking change 2-1, when searching for occurrences of 2020-04-01-preview API.

In 2022-04-01, RoleAssignmentsOperations changes list method to list_for_subscription:

https://github.com/Azure/azure-rest-api-specs/blob/b02ad2011daebdaa4ffc1b0b338181a464d49c47/specification/authorization/resource-manager/Microsoft.Authorization/preview/2020-04-01-preview/authorization-RoleAssignmentsCalls.json#L392-L397

    "/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/roleAssignments": {
      "get": {
        "tags": [
          "RoleAssignments"
        ],
        "operationId": "RoleAssignments_List",

https://github.com/Azure/azure-rest-api-specs/blob/495363bc011ce917f579adc1a5209073565d37f4/specification/authorization/resource-manager/Microsoft.Authorization/stable/2022-04-01/authorization-RoleAssignmentsCalls.json#L37-L42

    "/subscriptions/{subscriptionId}/providers/Microsoft.Authorization/roleAssignments": {
      "get": {
        "tags": [
          "RoleAssignments"
        ],
        "operationId": "RoleAssignments_ListForSubscription",

so this line will fail:

azure-cli/src/azure-cli/azure/cli/command_modules/ams/operations/sp.py

Line 299 in 8058c09

assignments = list(assignments_client.list(filter=f))

However, the retry decorator has a bug:

azure-cli/src/azure-cli/azure/cli/command_modules/ams/operations/sp.py

Lines 245 to 258 in 8058c09

    
           for retry_time in range(0, _RETRY_TIMES): 
        
               try: 
        
                   return func(*args, **kwargs) 
        
                   break  # pylint: disable=unreachable 
        
               except Exception as ex:  # pylint: disable=broad-except 
        
                   if retry_time < _RETRY_TIMES: 
        
                       time.sleep(5) 
        
                       logger.warning('Retrying %s creation: %s/%s', kwargs['entity_name_string'], 
        
                                      retry_time + 1, _RETRY_TIMES) 
        
                   else: 
        
                       logger.warning("Creating %s failed for appid. Trace followed:\n%s", 
        
                                      kwargs['entity_name_string'], ex.response.headers if hasattr(ex, 'response') else ex) 
        
                       # pylint: disable=no-member 
        
                       raise

retry_time will always be < _RETRY_TIMES, so line 255 will never be hit and the error will be silenced.

See the doc for range: https://docs.python.org/3/library/stdtypes.html#typesseq-range

jiasli · 2023-02-15T04:31:22Z

src/azure-cli/azure/cli/command_modules/resource/custom.py

    auth_client = _authorization_management_client(cmd.cli_ctx)
    if version == '2015-07-01':
-        return auth_client.provider_operations_metadata.get(resource_provider_namespace, version)
+        return auth_client.provider_operations_metadata.get(resource_provider_namespace, api_version=version)


This is to handle breaking change 3-1.

jiasli · 2023-02-15T08:51:43Z

I discovered a problem with iot module:

azure-cli/src/azure-cli/azure/cli/command_modules/iot/custom.py

Lines 588 to 590 in 9ef066f

    
           create = client.iot_hub_resource.begin_create_or_update(resource_group_name, hub_name, hub_description, polling=True) 
        
           if identity_role and identity_scopes: 
        
               create.add_done_callback(identity_assignment)

polling=True causes SDK to start a separate thread to do the LRO polling. Even if identity_assignment fails, it fails in that separate thread, so the main thread running the CLI command still succeeds, which is not expected.

jiasli · 2023-02-16T02:24:31Z

src/azure-cli-core/azure/cli/core/commands/arm.py

                                          parameters=parameters)
                break
-            except CloudError as ex:
+            except HttpResponseError as ex:


This is to handle breaking change 1-3.

jiasli · 2023-02-16T02:24:51Z

src/azure-cli/azure/cli/command_modules/acs/_roleassignments.py

    )
    properties = RoleAssignmentProperties(role_definition_id=role_id, principal_id=object_id)
-    return assignments_client.create(scope, assignment_name, properties, custom_headers=custom_headers)
+    return assignments_client.create(scope, assignment_name, properties, headers=custom_headers)


This is to handle breaking change 1-2.

jiasli · 2023-02-16T02:25:29Z

src/azure-cli/azure/cli/command_modules/role/_multi_api_adaptor.py

+        # In 2022-04-01 API, principal_type is by default 'User', so we have to explicitly set it to None if we can't
+        # resolve principal type from Graph
+        # https://github.com/Azure/azure-rest-api-specs/issues/21664
+        parameters = RoleAssignmentCreateParameters(
+            role_definition_id=role_id, principal_id=object_id, principal_type=assignee_principal_type,
+            description=description, condition=condition, condition_version=condition_version)


This is to handle breaking change 2-2.

evelyn-ys · 2023-02-16T06:55:07Z

src/azure-cli-core/azure/cli/core/profiles/_shared.py

+        ResourceType.MGMT_AUTHORIZATION: SDKProfile('2022-04-01', {
            'classic_administrators': '2015-06-01',
-            'role_definitions': '2018-01-01-preview',
+            'role_definitions': '2022-04-01',


Suggested change

'role_definitions': '2022-04-01',

We can simply remove this line if role_definitions uses the same API version with default one~

role_definitions frequently diverges from role_assignments:

azure-cli/src/azure-cli/azure/cli/command_modules/role/_multi_api_adaptor.py

Lines 62 to 65 in 5f40c5e

# 2015-07-01 RoleDefinition: flattened, RoleAssignment: unflattened

# 2018-01-01-preview RoleDefinition: flattened

# 2020-04-01-preview RoleAssignment: flattened

# Get property_name from properties if the model is unflattened.

so it is merely a placeholder.

role_definitions will again diverge from role_assignments: #26577

… version to 2022-04-01 (Azure#25452)

jiasli added 2 commits February 14, 2023 16:04

code

898f5a3

role tests

3639f9d

ghost requested a review from yonzhan February 14, 2023 08:26

ghost added the Auto-Assign Auto assign by bot label Feb 14, 2023

ghost assigned jiasli Feb 14, 2023

ghost added this to the Feb 2023 (2023-03-07) milestone Feb 14, 2023

ghost added the RBAC az role label Feb 14, 2023

Add issue link

c25dcd5

jiasli commented Feb 14, 2023

View reviewed changes

jiasli changed the title ~~[Role] Migrate azure-mgmt-authorization SDK to Track 2 and bump API Version to 2022-04-01~~ [Role] Migrate azure-mgmt-authorization SDK to Track 2 and bump API Version to 2022-04-01 Feb 14, 2023

jiasli added 2 commits February 14, 2023 17:00

test_provider_operation

ee43004

pylint

032e64d

jiasli commented Feb 15, 2023

View reviewed changes

acs

ce4e4f5

jiasli added 8 commits February 15, 2023 19:08

RoleAssignmentCreateParameters

4dda3fc

iot

c43a62d

aro

1a2d0ac

resource

2ffad8b

vm

ce22054

acs-pylint

d185743

pylint

422fa12

credscan

8c8bbbf

jiasli commented Feb 16, 2023

View reviewed changes

jiasli mentioned this pull request Feb 16, 2023

[ams] ams's retry decorator never fails #25474

Closed

jiasli changed the title ~~[Role] Migrate azure-mgmt-authorization SDK to Track 2 and bump API Version to 2022-04-01~~ [Role] Migrate azure-mgmt-authorization SDK to Track 2 and bump API version to 2022-04-01 Feb 16, 2023

jiasli requested a review from FumingZhang February 16, 2023 03:14

jiasli mentioned this pull request Feb 16, 2023

[iot] identity_assignment is called on another thread #25479

Closed

jiasli requested a review from bebound February 16, 2023 05:55

wangzelin007 previously approved these changes Feb 16, 2023

View reviewed changes

calvinhzy previously approved these changes Feb 16, 2023

View reviewed changes

jiasli mentioned this pull request Feb 16, 2023

{Packaging} Bump paramiko to 3.0.0 #25472

Merged

3 tasks

evelyn-ys reviewed Feb 16, 2023

View reviewed changes

evelyn-ys previously approved these changes Feb 16, 2023

View reviewed changes

jiasli added 3 commits February 16, 2023 16:32

Merge remote-tracking branch 'azure/dev' into role-track2

fa84b6b

acs iot

6fa073c

vm

35a3147

jiasli dismissed stale reviews from evelyn-ys, calvinhzy, wangzelin007, and zhoxing-ms via 35a3147 February 16, 2023 10:22

zhoxing-ms approved these changes Feb 17, 2023

View reviewed changes

kairu-ms approved these changes Feb 17, 2023

View reviewed changes

evelyn-ys approved these changes Feb 17, 2023

View reviewed changes

necusjz approved these changes Feb 17, 2023

View reviewed changes

jiasli merged commit 261c6d5 into Azure:dev Feb 17, 2023

jiasli deleted the role-track2 branch February 17, 2023 02:42

FumingZhang mentioned this pull request Feb 17, 2023

{AKS} Fix test recording test_aks_create_and_update_with_http_proxy_config Azure/azure-cli-extensions#5890

Merged

3 tasks

jiasli mentioned this pull request Mar 6, 2023

{AMS} Use authorization Track 2 SDK and fix retry decorator #25688

Merged

FumingZhang mentioned this pull request Apr 13, 2023

{AKS} Fix role assignment failure caused by the breaking change of default API version bump of the auth SDK Azure/azure-cli-extensions#6203

Merged

3 tasks

jiasli mentioned this pull request May 9, 2023

RBAC commands not using Python SDK models correctly #26362

Open

jiasli mentioned this pull request Aug 3, 2023

Azure CLI 2.51.0 cannot be released to Homebrew #27047

Closed

avgale pushed a commit to avgale/azure-cli that referenced this pull request Aug 24, 2023

[Role] Migrate azure-mgmt-authorization SDK to Track 2 and bump API…

8774243

… version to 2022-04-01 (Azure#25452)

	subscriptionId=serialize.url(
	"self._config.subscription_id", self.rcf.resources._config.subscription_id, 'str'),

	def get_mgmt_service_client(cli_ctx, client_or_resource_type, subscription_id=None, api_version=None,
	aux_subscriptions=None, aux_tenants=None, credential=None, **kwargs):

	# 2015-07-01 RoleDefinition: flattened, RoleAssignment: unflattened
	# 2018-01-01-preview RoleDefinition: flattened
	# 2020-04-01-preview RoleAssignment: flattened
	# Get property_name from properties if the model is unflattened.

Conversation

jiasli commented Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiasli Feb 14, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiasli Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yonzhan commented Feb 14, 2023

Uh oh!

jiasli commented Feb 15, 2023

Uh oh!

jiasli commented Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiasli Feb 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiasli commented Feb 15, 2023

Uh oh!

jiasli Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

jiasli Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

jiasli Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

evelyn-ys Feb 16, 2023

Choose a reason for hiding this comment

Uh oh!

jiasli Feb 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiasli Aug 3, 2023

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

jiasli commented Feb 14, 2023 •

edited

Loading

jiasli Feb 14, 2023 •

edited

Loading

jiasli Feb 15, 2023 •

edited

Loading

jiasli commented Feb 15, 2023 •

edited

Loading

jiasli Feb 15, 2023 •

edited

Loading

jiasli Feb 16, 2023 •

edited

Loading