azure-cli 2.50.0 pins subscriptions to v2019_11_01

[sodul]

Describe the bug

Due to the very large size of installing the azure CLI in docker container we use the https://github.com/clumio-code/azure-sdk-trim tool save hundreds of MBs. This tool trims the older SDKs since in most case they are useless and just a waste of space for the end users.

Related command

az login

Errors

ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: No module named 'azure.mgmt.resource.subscriptions.v2019_11_01'
Traceback (most recent call last):
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 663, in execute
    raise ex
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 726, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 697, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 333, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/command_modules/profile/custom.py", line 139, in login
    subscriptions = profile.login(
                    ^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/_profile.py", line 174, in login
    subscriptions = subscription_finder.find_using_specific_tenant(tenant, credential)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/cli/core/_profile.py", line 807, in find_using_specific_tenant
    subscriptions = client.subscriptions.list()
                    ^^^^^^^^^^^^^^^^^^^^
  File "/root/.pyenv/versions/3.11.4/lib/python3.11/site-packages/azure/mgmt/resource/subscriptions/_subscription_client.py", line 158, in subscriptions
    from .v2019_11_01.operations import SubscriptionsOperations as OperationClass
ModuleNotFoundError: No module named 'azure.mgmt.resource.subscriptions.v2019_11_01'

Issue script & Debug output

see above

Expected behavior

The azure CLI should not depend on 4y old APIs and work with the latest APIs.

Environment Summary

> az --version
azure-cli                         2.50.0

core                              2.50.0
telemetry                          1.0.8

Dependencies:
msal                              1.22.0
azure-mgmt-resource             23.1.0b2

Python location '/root/.pyenv/versions/3.11.4/bin/python3'
Extensions directory '/root/.azure/cliextensions'

Python (Darwin) 3.11.4 (main, Jun 27 2023, 17:18:33) [Clang 14.0.3 (clang-1403.0.22.14.1)]

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[jiasli]

Hi @sodul, I saw you have commented on the "Azure Python SDK huge size" topic in Azure/azure-sdk-for-python#17801 and Azure/azure-sdk-for-python#11149.

The recommended installation method is to follow the official document https://learn.microsoft.com/en-us/cli/azure/install-azure-cli to install pre-built packages, i.e. MSI, DEB, RPM and docker image, instead of from PyPI or source code. A script scripts/trim_sdk.py is called while building these packages in order to remove unused API versions:

azure-cli/scripts/trim_sdk.py

Lines 6 to 10 in b9fbd0e

	"""
	This script trims Python SDKs by
	- Removing aio folders
	- Removing unused API versions
	"""

azure-cli/scripts/release/rpm/azure-cli.spec

Line 52 in 5f52c83

%{python_cmd} %{repo_path}/scripts/trim_sdk.py

azure-cli/scripts/release/debian/build.sh

Line 55 in 5f52c83

$WORKDIR/python_env/bin/python3 ${WORKDIR}/scripts/trim_sdk.py

azure-cli/Dockerfile

Line 55 in 5f52c83

RUN ./scripts/install_full.sh && python ./scripts/trim_sdk.py \

We are still improving the docker image size in #25801.

---

As for API versions, without explicit feature request from service team, we don't regularly bump API versions that Azure CLI uses in order to maintain its maximum backward compatibility. So, keeping only the latest version is a very fragile solution and will almost certainly cause failures.

[sodul]

@jiasli Unfortunately installing the standalone version on a container is not a good option for us due to the size.

We build a container that has both the az cli and the SDK since our CD pipelines depend on both. Installing the CLI as a separate package would increase the size of our container by several hundred MB, which is why we install it as a python package so that we do not end up with 2 copies of python, and 2 copies of the SDK.

It is nice to see that the CLI now calls scripts/trim_sdk.py, which mirror the idea from this script: https://github.com/clumio-code/azure-sdk-trim/blob/main/azure_sdk_trim/azure_sdk_trim.py.

I will take a look at trim_sdk.py and see if it is a viable option for us, or will update azure_sdk_trim.py to detect if the cli exist and discover what specific versions it tries to import so that they do not get trimmed. The azure_sdk_trim.py has worked very well for us for the past 2 year, up until azure-cli 2.50.0.