az deployment and az stack always call https://aka.ms/BicepLatestRelease

[NJBBruins]

Describe the bug

We use az-cli in our pipelines. Our agents run in Azure and due to security reasons all internet acces is blocked. This causes problems when running az deployment of az stack with the option --parameters and a .bicepparam file as input.

To reproduce the problem i made this docker file:

FROM ubuntu:24.04

RUN apt update
RUN apt upgrade -y
RUN apt install -y curl git jq libicu74

# Create agent user and set up home directory
RUN useradd -m -d /home/agent agent

# Install latest bicep
ADD --chmod=700 --chown=agent:agent https://github.com/Azure/bicep/releases/latest/download/bicep-linux-x64 /usr/local/bin/bicep

# Install Azure-cli
RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

WORKDIR /azp/

RUN echo "\
targetScope = 'subscription'\n\
param resourceGroupName string\n\
\n\
resource resourceGroup 'Microsoft.Resources/resourceGroups@2024-03-01' = {\n\
  name: resourceGroupName\n\
  location: 'westeurope'\n\
}"\
>> ./resourceGroup.bicep

RUN echo "\
using './resourceGroup.bicep'\n\
\n\
param resourceGroupName = 'my-resource-group-name'"\
>>./resourceGroup.bicepparam

RUN chown -R agent:agent /azp /home/agent
USER agent

RUN az config set bicep.check_version=False
RUN az config set bicep.use_binary_from_path=True
RUN az config set bicep.use_binary_from_path.if_found_in_ci=true
RUN az extension add --name init

ENV AZURE_CORE_COLLECT_TELEMETRY=false
ENV AZURE_BICEP_USE_BINARY_FROM_PATH=true
ENV AZURE_BICEP_CHECK_VERSION=false

ENTRYPOINT [ "/bin/bash" ]

Save this as dockerfile and build the image with the following command: docker build --tag az .
Run the container with the command docker run -it az

Then in the docker container run this command to deploy a resource group az deployment sub create -l westeurope --parameters resourceGroup.bicepparam --debug

Part of the output of az deployment sub create -l westeurope --parameters resourceGroup.bicepparam --debug:

cli.azure.cli.core: init                      0.002         1         1  /home/agent/.azure/cliextensions/init
cli.azure.cli.core: Total (1)                 0.002         1         1  
cli.azure.cli.core: Loaded 1193 groups, 4698 commands.
cli.azure.cli.core: Updated command index in 0.004 seconds.
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f3a71116ca0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/agent/.azure/commands/2024-08-29.12-40-56.deployment_sub_create.16.log'.
az_command_data_logger: command args: deployment sub create -l {} --parameters {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f3a71143d80>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f3a7116b240>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f3a7116b380>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs [<function _documentdb_deprecate at 0x7f3a6f51f6a0>]
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f3a71f2a200>, <function CLIQuery.handle_query_parameter at 0x7f3a71d43ce0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f3a7116b2e0>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
urllib3.connectionpool: https://aka.ms:443 "GET /BicepLatestRelease HTTP/1.1" 301 0
urllib3.connectionpool: Starting new HTTPS connection (1): downloads.bicep.azure.com:443
urllib3.connectionpool: https://downloads.bicep.azure.com:443 "GET /releases/latest HTTP/1.1" 200 40365
cli.azure.cli.command_modules.resource._bicep: Generated download URL https://downloads.bicep.azure.com/v0.29.47/bicep-linux-x64. from system Linux, machine x86_64, release tag v0.29.47 and target platform None.
cli.azure.cli.command_modules.resource._bicep: The configuration value of bicep.use_binary_from_path has been set to 'false'.
cli.azure.cli.command_modules.resource._bicep: Successfully installed Bicep CLI to /home/agent/.azure/bin/bicep
cli.azure.cli.command_modules.resource._bicep: Current value of "use_binary_from_path": true.
cli.azure.cli.command_modules.resource._bicep: Using Bicep CLI from PATH. Bicep CLI version 0.29.47 (132ade51bc)

In the output there is a call to https://aka.ms/BicepLatestRelease this is fine when there is an internet connection but this gives an Error when it can't reach https://aka.ms/BicepLatestRelease

Checkout these four lines in the output:

cli.azure.cli.command_modules.resource._bicep: The configuration value of bicep.use_binary_from_path has been set to 'false'.
cli.azure.cli.command_modules.resource._bicep: Successfully installed Bicep CLI to /home/agent/.azure/bin/bicep
cli.azure.cli.command_modules.resource._bicep: Current value of "use_binary_from_path": true.
cli.azure.cli.command_modules.resource._bicep: Using Bicep CLI from PATH. Bicep CLI version 0.29.47 (132ade51bc)

First it says bicep.use_binary_from_path is set to false and than it says it is set to true.

We where expecting no call to https://aka.ms/BicepLatestRelease because use_binary_from_path is set to true and also check_version is set to false

output of az config get:

{
  "bicep": [
    {
      "name": "use_binary_from_path",
      "source": "AZURE_BICEP_USE_BINARY_FROM_PATH",
      "value": "true"
    },
    {
      "name": "check_version",
      "source": "AZURE_BICEP_CHECK_VERSION",
      "value": "false"
    },
    {
      "name": "use_binary_from_path.if_found_in_ci",
      "source": "/home/agent/.azure/config",
      "value": "true"
    }
  ],
  "cloud": [
    {
      "name": "name",
      "source": "/home/agent/.azure/config",
      "value": "AzureCloud"
    }
  ]
}

Related command

az deployment sub create -l westeurope --parameters resourceGroup.bicepparam --debug same problem with az stack

Errors

 cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f58c591af20>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.873 seconds (init: 0.200, invoke: 0.674)

Issue script & Debug output

 az deployment sub create -l westeurope --parameters resourceGroup.bicepparam --debug
cli.knack.cli: Command arguments: ['deployment', 'sub', 'create', '-l', 'westeurope', '--parameters', 'resourceGroup.bicepparam', '--debug']
cli.knack.cli: __init__ debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0x7f58c66aff60>, <function OutputProducer.on_global_arguments at 0x7f58c6656160>, <function CLIQuery.on_global_arguments at 0x7f58c6693c40>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'deployment': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name                  Load Time    Groups  Commands
cli.azure.cli.core: resource                  0.268        51       231
cli.azure.cli.core: Total (1)                 0.268        51       231
cli.azure.cli.core: Loaded 51 groups, 231 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command  : deployment sub create
cli.azure.cli.core: Command table: deployment sub create
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0x7f58c591aca0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/home/agent/.azure/commands/2024-08-29.11-57-54.deployment_sub_create.3090.log'.
az_command_data_logger: command args: deployment sub create -l {} --parameters {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument.<locals>.add_subscription_parameter at 0x7f58c5947d80>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument.<locals>.add_ids_arguments at 0x7f58c596f240>, <function register_cache_arguments.<locals>.add_cache_arguments at 0x7f58c596f380>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0x7f58c6656200>, <function CLIQuery.handle_query_parameter at 0x7f58c6693ce0>, <function register_ids_argument.<locals>.parse_ids_arguments at 0x7f58c596f2e0>]
urllib3.connectionpool: Starting new HTTPS connection (1): aka.ms:443
cli.azure.cli.core.azclierror: Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 404, in _make_request
    self._validate_conn(conn)
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1060, in _validate_conn
    conn.connect()
  File "/opt/az/lib/python3.11/site-packages/urllib3/connection.py", line 419, in connect
    self.sock = ssl_wrap_socket(
                ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/ssl.py", line 517, in wrap_socket
    return self.sslsocket_class._create(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/opt/az/lib/python3.11/ssl.py", line 1382, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: [SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/requests/adapters.py", line 667, in send
    resp = conn.urlopen(
           ^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/connectionpool.py", line 801, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/urllib3/util/retry.py", line 594, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/_bicep.py", line 200, in get_bicep_latest_release_tag
    response = requests.get("https://aka.ms/BicepLatestRelease", verify=_requests_verify)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/requests/adapters.py", line 698, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)')))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 701, in _run_job
    result = cmd_copy(params)
             ^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 334, in __call__
    return self.handler(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
    return op(**command_args)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/custom.py", line 525, in deploy_arm_template_at_subscription_scope
    return _deploy_arm_template_at_subscription_scope(cmd=cmd,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/custom.py", line 549, in _deploy_arm_template_at_subscription_scope
    deployment_properties = _prepare_deployment_properties_unmodified(cmd, 'subscription', template_file=template_file,
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/custom.py", line 1150, in _prepare_deployment_properties_unmodified
    template_content, template_spec_id, bicepparam_json_content = _parse_bicepparam_file(cmd, template_file, parameters)
                                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/custom.py", line 1073, in _parse_bicepparam_file
    ensure_bicep_installation(cmd.cli_ctx, stdout=False)
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/_bicep.py", line 129, in ensure_bicep_installation
    release_tag = release_tag if release_tag else get_bicep_latest_release_tag()
                                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/command_modules/resource/_bicep.py", line 204, in get_bicep_latest_release_tag
    raise ClientRequestError(f"Error while attempting to retrieve the latest Bicep version: {err}.")
azure.cli.core.azclierror.ClientRequestError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/arm.py", line 109, in handle_template_based_exception
    raise CLIError(ex.inner_exception.error.message)
                   ^^^^^^^^^^^^^^^^^^
AttributeError: 'ClientRequestError' object has no attribute 'inner_exception'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/az/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
    cmd_result = self.invocation.execute(args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 664, in execute
    raise ex
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 731, in _run_jobs_serially
    results.append(self._run_job(expanded_arg, cmd_copy))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/__init__.py", line 723, in _run_job
    return cmd_copy.exception_handler(ex)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/az/lib/python3.11/site-packages/azure/cli/core/commands/arm.py", line 114, in handle_template_based_exception
    raise CLIError(ex)
knack.util.CLIError: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aa.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))).

cli.azure.cli.core.azclierror: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))).
az_command_data_logger: Error while attempting to retrieve the latest Bicep version: HTTPSConnectionPool(host='aka.ms', port=443): Max retries exceeded with url: /BicepLatestRelease (Caused by SSLError(SSLEOFError(8, '[SSL: UNEXPECTED_EOF_WHILE_READING] EOF occurred in violation of protocol (_ssl.c:1006)'))).
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0x7f58c591af20>]
az_command_data_logger: exit code: 1
cli.__main__: Command ran in 0.873 seconds (init: 0.200, invoke: 0.674)

Expected behavior

We where expecting no call to https://aka.ms/BicepLatestRelease because use_binary_from_path is set to true and also check_version is set to false

Environment Summary

az --version
azure-cli 2.63.0

core 2.63.0
telemetry 1.1.0

Extensions:
init 0.1.0

Dependencies:
msal 1.30.0
azure-mgmt-resource 23.1.1

Python location '/opt/az/bin/python3'
Extensions directory '/home/agent/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Jul 31 2024, 03:40:14) [GCC 13.2.0]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.

Bicep CLI version 0.29.47 (132ade51bc)

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[github-actions]

Here are some similar issues that might help you. Please check if they can solve your problem.

• AZ Cli SSL verification failure with proxy root cert (Python works) #29018