{CI} Restore az self-test

[jiasli]

Symptom

CI failure https://dev.azure.com/azure-sdk/public/_build/results?buildId=801336&view=logs&j=34631516-3f1c-501b-e428-6ea670d20b1c&t=4ecab084-061a-520a-6791-4f5413067b6d&l=1816

  File "/opt/az/azcliextensions/arcappliance/azext_arcappliance/pkg/resource/logic/util.py", line 5, in <module>
    from jsonschema import validate, ValidationError
  File "/opt/az/azcliextensions/azure-iot/jsonschema/__init__.py", line 33, in <module>
    __version__ = get_distribution(__name__).version
  ...
  File "/opt/hostedtoolcache/Python/3.6.13/x64/lib/python3.6/site-packages/pkg_resources/__init__.py", line 784, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema' distribution was not found and is required by the application

Root cause

This can also be reproed locally. (You may need to rename ~/cliextensions/azure-iot to aazure-iot to make it appear earlier than arcappliance, because os.listdir's result is not sorted.)

az extension add -n azure-iot
az extension add -n arcappliance
az self-test

Same error problem can be reproed with only azure-iot.

❌ This doesn't work:

import sys
import pkg_resources

sys.path.append(r'C:\Users\me\.azure\cliextensions\azure-iot')
print(pkg_resources.working_set.by_key.get('jsonschema'))
disk = pkg_resources.get_distribution('jsonschema')

None
Traceback (most recent call last):
  File "D:\cli\testproj\demo.py", line 8, in <module>
    print(pkg_resources.get_distribution('jsonschema'))
  ...
  File "D:\cli\py39\lib\site-packages\pkg_resources\__init__.py", line 772, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'jsonschema' distribution was not found and is required by the application

✔ This works:

import sys

sys.path.append(r'C:\Users\me\.azure\cliextensions\azure-iot')

import pkg_resources
print(pkg_resources.working_set.by_key.get('jsonschema'))
print(pkg_resources.get_distribution('jsonschema'))

jsonschema 3.0.2
jsonschema 3.0.2

This is because the change on sys.path doesn't take effect if pkg_resources is already imported.

CLI imports pkg_resources first:

azure-cli/src/azure-cli-core/azure/cli/core/__init__.py

Line 625 in 0df58e9

from pkg_resources import parse_version

Then it changes sys.path:

azure-cli/src/azure-cli-core/azure/cli/core/__init__.py

Line 337 in 0df58e9

sys.path.append(ext_dir)

As for why this issue doesn't happen when installed separately:

• When only loading azure-iot, azure-iot/jsonschema/__init__.py is not loaded at all (but the 🐛 is there). When azure-iot really uses jsonschema, azure-iot/azext_iot/common/utility.py:415 has its workaround to update pkg_resources.working_set:

  def ensure_pkg_resources_entries():
      import pkg_resources
  
      from azure.cli.core.extension import get_extension_path
      from azext_iot.constants import EXTENSION_NAME
  
      extension_path = get_extension_path(EXTENSION_NAME)
      if extension_path not in pkg_resources.working_set.entries:
          pkg_resources.working_set.add_entry(extension_path)
  
      return

• When only loading arcappliance, it loads arcappliance/jsonschema/__init__.py, which uses the new importlib.metadata and works fine
• When they are installed together, both azure-iot and arcappliance are loaded to sys.path. If azure-iot is loaded first, arcappliance loads jsonschema from azure-iot/jsonschema/__init__.py, thus bypassing the workaround and hit the 🐛

ℹ The above conclusion is derived by adding

import traceback
traceback.print_stack()

to

• pkg_resources/__init__.py
• azure-iot/jsonschema/__init__.py
• arcappliance/jsonschema/__init__.py

Then run az self-test to observe the result.

Actually, az self-test should not load all extensions together

It is possible for CLI to load all extensions together - when command index has not been built yet.

Proposed solution

Fix the azure-iot extension by using the latest jsonschema which uses importlib.metadata(python-jsonschema/jsonschema@6a7155a, python-jsonschema/jsonschema@fe772c3)

Mitigation

This PR restores az self-test and blocks azure-iot until jsonschema is bumped to the latest version.

[yonzhan]

CI

[yonzhan]

LGTM

[fengzhou-msft]

Can you submit an issue to https://github.com/azure/azure-iot-cli-extension for bumping the jsonschema version.

[jiasli]

Sure. Created issue Azure/azure-iot-cli-extension#321.