az container exec fails with python websocket error in Azure DevOps hosted agent

[ghelyar]

Describe the bug

Command Name
az container exec

Errors:

File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\websocket/_app.py", line 345, in _callback

Unfortunately, that's the entire error.

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

- task: AzureCLI@2
  inputs:
    azureSubscription: ...
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
	  az group create --location LOCATION --name RG_NAME
	  az container create -g RG_NAME -n ACI_NAME -f aci.yml
	  az container exec -n ACI_NAME --container-name CONTAINER_NAME --exec-command "ls"

Expected Behavior

Environment Summary

Azure DevOps hosted agent: windows-2019

C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd" --version"
azure-cli                         2.23.0 *

core                              2.23.0 *
telemetry                          1.0.6

Extensions:
azure-devops                      0.18.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.8.9 (tags/v3.8.9:a743f81, Apr  6 2021, 13:22:56) [MSC v.1928 32 bit (Intel)]

Additional Context

This was working on 2021-05-24 with 2.23.0, and started failing on 2021-05-25, the day 2.24.0 was released, but it's still using 2.23.0 according to the logs.

After the agent updated to 2.24.0, I started seeing an error with az container create, and not even getting this far.

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @macolso.

Issue Details

---

Describe the bug

Command Name
az container exec

Errors:

File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\websocket/_app.py", line 345, in _callback

To Reproduce:

Steps to reproduce the behavior. Note that argument values have been redacted, as they may contain sensitive information.

- task: AzureCLI@2
  inputs:
    azureSubscription: ...
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
	  az group create --location LOCATION --name RG_NAME
	  az container create -g RG_NAME -n ACI_NAME -f aci.yml
	  az container exec -n ACI_NAME --container-name CONTAINER_NAME --exec-command "ls"

Expected Behavior

Environment Summary

Azure DevOps hosted agent: windows-2019

C:\Windows\system32\cmd.exe /D /S /C ""C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin\az.cmd" --version"
azure-cli                         2.23.0 *

core                              2.23.0 *
telemetry                          1.0.6

Extensions:
azure-devops                      0.18.0

Python location 'C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Program Files\Common Files\AzureCliExtensionDirectory'

Python (Windows) 3.8.9 (tags/v3.8.9:a743f81, Apr  6 2021, 13:22:56) [MSC v.1928 32 bit (Intel)]

Additional Context

This was working on 2021-05-24 with 2.23.0, and started failing on 2021-05-25, the day 2.24.0 was released, but it's still using with 2.23.0 according to the logs.

Author:	ghelyar
Assignees:	-
Labels:	Container Instances, Service Attention
Milestone:	-

[yonzhan]

route to service team

[jiasli]

az container exec seems to be another issue. Could you share the --debug log?

[ghelyar]

TLDR

2021-06-03T11:07:01.4586179Z ERROR: websocket: error from callback <function _on_ws_msg at 0x03E86928>: write() argument must be str, not bytes
2021-06-03T11:07:01.4756691Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\websocket/_app.py", line 345, in _callback
2021-06-03T11:07:01.4758356Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/container/custom.py", line 643, in _on_ws_msg

Full log

2021-06-03T11:06:59.1289221Z DEBUG: cli.knack.cli: Command arguments: ['container', 'exec', '--debug', '--verbose', '-n', '***', '--container-name', '***', '--exec-command', 'ls']
2021-06-03T11:06:59.1290233Z DEBUG: cli.knack.cli: __init__ debug log:
2021-06-03T11:06:59.1290813Z Cannot enable color.
2021-06-03T11:06:59.1291306Z DEBUG: cli.knack.cli: Event: Cli.PreExecute []
2021-06-03T11:06:59.1292238Z DEBUG: cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x03042C88>, <function OutputProducer.on_global_arguments at 0x030B7B20>, <function CLIQuery.on_global_arguments at 0x030CE730>]
2021-06-03T11:06:59.1529498Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
2021-06-03T11:06:59.1530590Z DEBUG: cli.azure.cli.core: Modules found from index for 'container': ['azure.cli.command_modules.container']
2021-06-03T11:06:59.1531542Z DEBUG: cli.azure.cli.core: Loading command modules:
2021-06-03T11:06:59.1532588Z DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands
2021-06-03T11:06:59.1571080Z DEBUG: cli.azure.cli.core: container                 0.004         1        11
2021-06-03T11:06:59.1572114Z DEBUG: cli.azure.cli.core: Total (1)                 0.004         1        11
2021-06-03T11:06:59.1586922Z DEBUG: cli.azure.cli.core: These extensions are not installed and will be skipped: ['azext_ai_examples', 'azext_next']
2021-06-03T11:06:59.1587952Z DEBUG: cli.azure.cli.core: Loading extensions:
2021-06-03T11:06:59.1588617Z DEBUG: cli.azure.cli.core: Name                  Load Time    Groups  Commands  Directory
2021-06-03T11:06:59.1590092Z DEBUG: cli.azure.cli.core: Total (0)                 0.000         0         0  
2021-06-03T11:06:59.1590822Z DEBUG: cli.azure.cli.core: Loaded 1 groups, 11 commands.
2021-06-03T11:06:59.1591636Z DEBUG: cli.azure.cli.core: Found a match in the command table.
2021-06-03T11:06:59.1592244Z DEBUG: cli.azure.cli.core: Raw command  : container exec
2021-06-03T11:06:59.1592837Z DEBUG: cli.azure.cli.core: Command table: container exec
2021-06-03T11:06:59.1593533Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x0345FB68>]
2021-06-03T11:06:59.1596085Z DEBUG: cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to 'D:\a\_temp\.azclitask\commands\2021-06-03.11-06-59.container_exec.5432.log'.
2021-06-03T11:06:59.1597258Z INFO: az_command_data_logger: command args: container exec --debug --verbose -n {} --container-name {} --exec-command {}
2021-06-03T11:06:59.1598427Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x034A1BB0>, <function register_global_query_examples_argument.<locals>.register_query_examples at 0x034BB2B0>]
2021-06-03T11:06:59.3656392Z INFO: cli.knack.commands: Configured default '***' for arg resource_group_name
2021-06-03T11:06:59.3657499Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
2021-06-03T11:06:59.3658322Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x034BB2F8>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x034BB388>]
2021-06-03T11:06:59.3669214Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
2021-06-03T11:06:59.3671148Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
2021-06-03T11:06:59.3679389Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x030B7B68>, <function CLIQuery.handle_query_parameter at 0x030CE778>, <function register_global_query_examples_argument.<locals>.handle_example_parameter at 0x034BB268>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x034BB340>]
2021-06-03T11:06:59.3692932Z DEBUG: cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ContainerInstanceManagementClient
2021-06-03T11:06:59.4203124Z DEBUG: msrest.universal_http.requests: Configuring retry: max_retries=4, backoff_factor=0.8, max_backoff=90
2021-06-03T11:06:59.4208531Z DEBUG: cli.azure.cli.core.commands.client_factory: Adding custom headers to the client:
2021-06-03T11:06:59.4209806Z DEBUG: cli.azure.cli.core.commands.client_factory:     'x-ms-client-request-id': 'd0f1f334-c45b-11eb-b09d-000d3aadd7ca'
2021-06-03T11:06:59.4210647Z DEBUG: cli.azure.cli.core.commands.client_factory:     'CommandName': 'container exec'
2021-06-03T11:06:59.4211669Z DEBUG: cli.azure.cli.core.commands.client_factory:     'ParameterSetName': '--debug --verbose -n --container-name --exec-command'
2021-06-03T11:06:59.4212861Z DEBUG: cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ContainerInstanceManagementClient
2021-06-03T11:06:59.4282130Z DEBUG: msrest.universal_http.requests: Configuring retry: max_retries=4, backoff_factor=0.8, max_backoff=90
2021-06-03T11:06:59.4286930Z DEBUG: cli.azure.cli.core.commands.client_factory: Adding custom headers to the client:
2021-06-03T11:06:59.4288001Z DEBUG: cli.azure.cli.core.commands.client_factory:     'x-ms-client-request-id': 'd0f1f334-c45b-11eb-b09d-000d3aadd7ca'
2021-06-03T11:06:59.4288790Z DEBUG: cli.azure.cli.core.commands.client_factory:     'CommandName': 'container exec'
2021-06-03T11:06:59.4289553Z DEBUG: cli.azure.cli.core.commands.client_factory:     'ParameterSetName': '--debug --verbose -n --container-name --exec-command'
2021-06-03T11:06:59.4290352Z DEBUG: cli.azure.cli.core.commands.client_factory: Getting management service client client_type=ContainerInstanceManagementClient
2021-06-03T11:06:59.4350794Z DEBUG: msrest.universal_http.requests: Configuring retry: max_retries=4, backoff_factor=0.8, max_backoff=90
2021-06-03T11:06:59.4355692Z DEBUG: cli.azure.cli.core.commands.client_factory: Adding custom headers to the client:
2021-06-03T11:06:59.4357520Z DEBUG: cli.azure.cli.core.commands.client_factory:     'x-ms-client-request-id': 'd0f1f334-c45b-11eb-b09d-000d3aadd7ca'
2021-06-03T11:06:59.4359159Z DEBUG: cli.azure.cli.core.commands.client_factory:     'CommandName': 'container exec'
2021-06-03T11:06:59.4361721Z DEBUG: cli.azure.cli.core.commands.client_factory:     'ParameterSetName': '--debug --verbose -n --container-name --exec-command'
2021-06-03T11:06:59.4363624Z DEBUG: cli.azure.cli.core.adal_authentication: AdalAuthentication.signed_session invoked by Track 1 SDK
2021-06-03T11:06:59.4364517Z DEBUG: cli.azure.cli.core._profile: Retrieving token from ADAL for resource 'https://management.core.windows.net/'
2021-06-03T11:06:59.4365405Z DEBUG: cli.azure.cli.core.util: attempting to read file D:\a\_temp\.azclitask\accessTokens.json as utf-8-sig
2021-06-03T11:06:59.4372553Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - Authority:Performing instance discovery: ...
2021-06-03T11:06:59.4375540Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - Authority:Performing static instance discovery
2021-06-03T11:06:59.4376460Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - Authority:Authority validated via static instance discovery
2021-06-03T11:06:59.4377511Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - TokenRequest:Getting token with client credentials.
2021-06-03T11:06:59.4378264Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - TokenRequest:No user_id passed for cache query
2021-06-03T11:06:59.4379971Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - CacheDriver:finding with query keys: {'_clientId': '...'}
2021-06-03T11:06:59.4381055Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - CacheDriver:Looking for potential cache entries: {'_clientId': '...'}
2021-06-03T11:06:59.4382368Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - CacheDriver:Found 0 potential entries.
2021-06-03T11:06:59.4424071Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443
2021-06-03T11:06:59.5832432Z DEBUG: urllib3.connectionpool: https://login.microsoftonline.com:443 "POST /***/oauth2/token HTTP/1.1" 200 1466
2021-06-03T11:06:59.5850010Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - OAuth2Client:Get Token Server returned this correlation_id: d0c08458-cde8-4f56-9c77-28c50f86d044
2021-06-03T11:06:59.5856093Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - CacheDriver:Adding entry AccessTokenId: b'gKzaoMTfnOwoc3ZK9XvlU5Fh4QKOlITVbu9xjkbq0+c='
2021-06-03T11:06:59.5859584Z DEBUG: adal-python: d0c08458-cde8-4f56-9c77-28c50f86d044 - CacheDriver:Added entry is MRRT
2021-06-03T11:06:59.5860819Z DEBUG: msrest.http_logger: Request URL: 'https://management.azure.com/subscriptions/8ee1393a-b2dd-467d-a8b6-5e901aad0fb9/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***?api-version=2018-10-01'
2021-06-03T11:06:59.5861654Z DEBUG: msrest.http_logger: Request method: 'GET'
2021-06-03T11:06:59.5862267Z DEBUG: msrest.http_logger: Request headers:
2021-06-03T11:06:59.5862831Z DEBUG: msrest.http_logger:     'Accept': 'application/json'
2021-06-03T11:06:59.5863401Z DEBUG: msrest.http_logger:     'accept-language': 'en-US'
2021-06-03T11:06:59.5864215Z DEBUG: msrest.http_logger:     'User-Agent': 'python/3.8.9 (Windows-10-10.0.17763-SP0) msrest/0.6.21 VSTS_4a3f3fcc-6518-4256-b75f-8cb6957e6842_build_1988_0 msrest_azure/0.6.3 azure-mgmt-containerinstance/1.5.0 Azure-SDK-For-Python AZURECLI/2.24.0 (MSI)'
2021-06-03T11:06:59.5865018Z DEBUG: msrest.http_logger: Request body:
2021-06-03T11:06:59.5865525Z DEBUG: msrest.http_logger: None
2021-06-03T11:06:59.5866083Z DEBUG: msrest.universal_http: Configuring redirects: allow=True, max=30
2021-06-03T11:06:59.5866708Z DEBUG: msrest.universal_http: Configuring request: timeout=100, verify=True, cert=None
2021-06-03T11:06:59.5867497Z DEBUG: msrest.universal_http: Configuring proxies: ''
2021-06-03T11:06:59.5868101Z DEBUG: msrest.universal_http: Evaluate proxies against ENV settings: True
2021-06-03T11:06:59.5902129Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
2021-06-03T11:06:59.8711702Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "GET /subscriptions/8ee1393a-b2dd-467d-a8b6-5e901aad0fb9/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***?api-version=2018-10-01 HTTP/1.1" 200 None
2021-06-03T11:06:59.8726033Z DEBUG: msrest.http_logger: Response status: 200
2021-06-03T11:06:59.8727123Z DEBUG: msrest.http_logger: Response headers:
2021-06-03T11:06:59.8727933Z DEBUG: msrest.http_logger:     'Cache-Control': 'no-cache'
2021-06-03T11:06:59.8730035Z DEBUG: msrest.http_logger:     'Pragma': 'no-cache'
2021-06-03T11:06:59.8730897Z DEBUG: msrest.http_logger:     'Transfer-Encoding': 'chunked'
2021-06-03T11:06:59.8732187Z DEBUG: msrest.http_logger:     'Content-Type': 'application/json; charset=utf-8'
2021-06-03T11:06:59.8733522Z DEBUG: msrest.http_logger:     'Content-Encoding': 'gzip'
2021-06-03T11:06:59.8734704Z DEBUG: msrest.http_logger:     'Expires': '-1'
2021-06-03T11:06:59.8735829Z DEBUG: msrest.http_logger:     'Vary': 'Accept-Encoding,Accept-Encoding'
2021-06-03T11:06:59.8736734Z DEBUG: msrest.http_logger:     'x-ms-request-id': 'ukwest:cee3acac-cd10-4f06-8556-9427ecfed540'
2021-06-03T11:06:59.8737604Z DEBUG: msrest.http_logger:     'x-ms-ratelimit-remaining-subscription-reads': '11995'
2021-06-03T11:06:59.8738307Z DEBUG: msrest.http_logger:     'x-ms-correlation-request-id': '520b8939-1ad6-494a-9b15-ba08b552d48e'
2021-06-03T11:06:59.8739865Z DEBUG: msrest.http_logger:     'x-ms-routing-request-id': 'WESTEUROPE:20210603T110659Z:520b8939-1ad6-494a-9b15-ba08b552d48e'
2021-06-03T11:06:59.8740823Z DEBUG: msrest.http_logger:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
2021-06-03T11:06:59.8741708Z DEBUG: msrest.http_logger:     'X-Content-Type-Options': 'nosniff'
2021-06-03T11:06:59.8742382Z DEBUG: msrest.http_logger:     'Date': 'Thu, 03 Jun 2021 11:06:59 GMT'
2021-06-03T11:06:59.8743006Z DEBUG: msrest.http_logger: Response content:
2021-06-03T11:06:59.8753978Z DEBUG: msrest.http_logger: {"properties":{"provisioningState":"Succeeded","containers":[***
2021-06-03T11:06:59.8831674Z DEBUG: cli.azure.cli.core.adal_authentication: AdalAuthentication.signed_session invoked by Track 1 SDK
2021-06-03T11:06:59.8832807Z DEBUG: cli.azure.cli.core._profile: Retrieving token from ADAL for resource 'https://management.core.windows.net/'
2021-06-03T11:06:59.8838843Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - Authority:Performing instance discovery: ...
2021-06-03T11:06:59.8842002Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - Authority:Performing static instance discovery
2021-06-03T11:06:59.8843025Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - Authority:Authority validated via static instance discovery
2021-06-03T11:06:59.8844033Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - TokenRequest:Getting token with client credentials.
2021-06-03T11:06:59.8844758Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - TokenRequest:No user_id passed for cache query
2021-06-03T11:06:59.8846372Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - CacheDriver:finding with query keys: {'_clientId': '...'}
2021-06-03T11:06:59.8847174Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - CacheDriver:Looking for potential cache entries: {'_clientId': '...'}
2021-06-03T11:06:59.8847920Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - CacheDriver:Found 0 potential entries.
2021-06-03T11:06:59.8880386Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): login.microsoftonline.com:443
2021-06-03T11:06:59.9798271Z DEBUG: urllib3.connectionpool: https://login.microsoftonline.com:443 "POST /***/oauth2/token HTTP/1.1" 200 1466
2021-06-03T11:06:59.9815064Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - OAuth2Client:Get Token Server returned this correlation_id: 6b685992-4494-41d1-a9dd-c781ac288aef
2021-06-03T11:06:59.9824998Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - CacheDriver:Adding entry AccessTokenId: b'QifPDWLaNlAqkAQ9pn2bZ38SoEju7p/Pbskwi6X7VcE='
2021-06-03T11:06:59.9826266Z DEBUG: adal-python: 6b685992-4494-41d1-a9dd-c781ac288aef - CacheDriver:Added entry is MRRT
2021-06-03T11:06:59.9827652Z DEBUG: msrest.http_logger: Request URL: 'https://management.azure.com/subscriptions/8ee1393a-b2dd-467d-a8b6-5e901aad0fb9/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***/containers/***/exec?api-version=2018-10-01'
2021-06-03T11:06:59.9828728Z DEBUG: msrest.http_logger: Request method: 'POST'
2021-06-03T11:06:59.9829260Z DEBUG: msrest.http_logger: Request headers:
2021-06-03T11:06:59.9829837Z DEBUG: msrest.http_logger:     'Accept': 'application/json'
2021-06-03T11:06:59.9830445Z DEBUG: msrest.http_logger:     'Content-Type': 'application/json; charset=utf-8'
2021-06-03T11:06:59.9831043Z DEBUG: msrest.http_logger:     'accept-language': 'en-US'
2021-06-03T11:06:59.9831621Z DEBUG: msrest.http_logger:     'Content-Length': '59'
2021-06-03T11:06:59.9832432Z DEBUG: msrest.http_logger:     'User-Agent': 'python/3.8.9 (Windows-10-10.0.17763-SP0) msrest/0.6.21 VSTS_4a3f3fcc-6518-4256-b75f-8cb6957e6842_build_1988_0 msrest_azure/0.6.3 azure-mgmt-containerinstance/1.5.0 Azure-SDK-For-Python AZURECLI/2.24.0 (MSI)'
2021-06-03T11:06:59.9833234Z DEBUG: msrest.http_logger: Request body:
2021-06-03T11:06:59.9833843Z DEBUG: msrest.http_logger: {"command": "ls", "terminalSize": {"rows": 20, "cols": 80}}
2021-06-03T11:06:59.9834514Z DEBUG: msrest.universal_http: Configuring redirects: allow=True, max=30
2021-06-03T11:06:59.9835142Z DEBUG: msrest.universal_http: Configuring request: timeout=100, verify=True, cert=None
2021-06-03T11:06:59.9835758Z DEBUG: msrest.universal_http: Configuring proxies: ''
2021-06-03T11:06:59.9836388Z DEBUG: msrest.universal_http: Evaluate proxies against ENV settings: True
2021-06-03T11:06:59.9865942Z DEBUG: urllib3.connectionpool: Starting new HTTPS connection (1): management.azure.com:443
2021-06-03T11:07:00.6927717Z DEBUG: urllib3.connectionpool: https://management.azure.com:443 "POST /subscriptions/8ee1393a-b2dd-467d-a8b6-5e901aad0fb9/resourceGroups/***/providers/Microsoft.ContainerInstance/containerGroups/***/containers/***/exec?api-version=2018-10-01 HTTP/1.1" 200 None
2021-06-03T11:07:00.6938300Z DEBUG: msrest.http_logger: Response status: 200
2021-06-03T11:07:00.6939320Z DEBUG: msrest.http_logger: Response headers:
2021-06-03T11:07:00.6939962Z DEBUG: msrest.http_logger:     'Cache-Control': 'no-cache'
2021-06-03T11:07:00.6940578Z DEBUG: msrest.http_logger:     'Pragma': 'no-cache'
2021-06-03T11:07:00.6941524Z DEBUG: msrest.http_logger:     'Transfer-Encoding': 'chunked'
2021-06-03T11:07:00.6942370Z DEBUG: msrest.http_logger:     'Content-Type': 'application/json; charset=utf-8'
2021-06-03T11:07:00.6943005Z DEBUG: msrest.http_logger:     'Content-Encoding': 'gzip'
2021-06-03T11:07:00.6943587Z DEBUG: msrest.http_logger:     'Expires': '-1'
2021-06-03T11:07:00.6944189Z DEBUG: msrest.http_logger:     'Vary': 'Accept-Encoding,Accept-Encoding'
2021-06-03T11:07:00.6945084Z DEBUG: msrest.http_logger:     'x-ms-request-id': 'ukwest:b7df44b3-3a58-43be-bd0e-09d4c870d6ca'
2021-06-03T11:07:00.6946733Z DEBUG: msrest.http_logger:     'x-ms-ratelimit-remaining-subscription-writes': '1198'
2021-06-03T11:07:00.6947496Z DEBUG: msrest.http_logger:     'x-ms-correlation-request-id': '4bd24a94-fc56-4104-a829-7fb303696fef'
2021-06-03T11:07:00.6948699Z DEBUG: msrest.http_logger:     'x-ms-routing-request-id': 'WESTEUROPE:20210603T110700Z:4bd24a94-fc56-4104-a829-7fb303696fef'
2021-06-03T11:07:00.6949520Z DEBUG: msrest.http_logger:     'Strict-Transport-Security': 'max-age=31536000; includeSubDomains'
2021-06-03T11:07:00.6950430Z DEBUG: msrest.http_logger:     'X-Content-Type-Options': 'nosniff'
2021-06-03T11:07:00.6951340Z DEBUG: msrest.http_logger:     'Date': 'Thu, 03 Jun 2021 11:06:59 GMT'
2021-06-03T11:07:00.6952347Z DEBUG: msrest.http_logger: Response content:
2021-06-03T11:07:00.6953377Z DEBUG: msrest.http_logger: {"webSocketUri":"wss://sbzr2uadnhdhkgh.ukwest.atlas.cloudapp.azure.com:19390/execstream/sessionId/8e5e9578-c005-4881-a78a-6887d15846b7?api-version=1.0&rows=20&cols=80","password":"***"}
2021-06-03T11:07:01.2985555Z ERROR: websocket: error from callback <function _on_ws_msg at 0x03E86928>: write() argument must be str, not bytes
2021-06-03T11:07:01.4423910Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\websocket/_app.py", line 345, in _callback
2021-06-03T11:07:01.4424692Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/container/custom.py", line 643, in _on_ws_msg
2021-06-03T11:07:01.4425437Z ERROR: websocket: error from callback <function _on_ws_msg at 0x03E86928>: write() argument must be str, not bytes
2021-06-03T11:07:01.4584689Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\websocket/_app.py", line 345, in _callback
2021-06-03T11:07:01.4585448Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/container/custom.py", line 643, in _on_ws_msg
2021-06-03T11:07:01.4586179Z ERROR: websocket: error from callback <function _on_ws_msg at 0x03E86928>: write() argument must be str, not bytes
2021-06-03T11:07:01.4756691Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\websocket/_app.py", line 345, in _callback
2021-06-03T11:07:01.4758356Z   File "D:\a\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/command_modules/container/custom.py", line 643, in _on_ws_msg
2021-06-03T11:07:01.4760120Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnTransformResult [<function _resource_group_transform at 0x03497658>, <function _x509_from_base64_to_hex_transform at 0x034976A0>]
2021-06-03T11:07:01.4761709Z DEBUG: cli.knack.cli: Event: CommandInvoker.OnFilterResult []
2021-06-03T11:07:01.4762563Z DEBUG: cli.knack.cli: Event: Cli.SuccessfulExecute []
2021-06-03T11:07:01.4763331Z DEBUG: cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x0345FC88>]
2021-06-03T11:07:01.4764267Z INFO: az_command_data_logger: exit code: 0
2021-06-03T11:07:01.4765010Z INFO: cli.__main__: Command ran in 2.570 seconds (init: 0.221, invoke: 2.349)
2021-06-03T11:07:01.5288744Z INFO: telemetry.save: Save telemetry record of length 2960 in cache
2021-06-03T11:07:01.5290733Z WARNING: telemetry.check: Negative: The D:\a\_temp\.azclitask\telemetry.txt was modified at 2021-06-03 11:02:27.654020, which in less than 600.000000 s

2021-06-03T11:07:01.6541026Z ##[debug]$LASTEXITCODE: 0

I am no Python expert but it looks like in websocket.WebSocketApp on_message is called as a fallback on opcodes and sometimes the data decoded from utf-8, and sometimes isn't, depending on the opcode, which makes it seem a bit unsafe. Maybe using on_data and looking at the opcode would be safer, but again I'm no Python expert.

[zhoxing-ms]

This issue seems duplicate with #18251, @bitcloud could you please help to have a look~

[bitcloud]

the websocket could also return utf-8 but the cli does not handle that properly. The PR should take care of that.