bicep on linux arm64 installation and InsufficientExecutionStackException

[KrisSimon]

Describe the bug

On a linux aarch64 platform we faced two issues:

1. installation with az bicep install does not detect the correct architecture for the binary. Workaround is this:

if [ "$(arch)" == "aarch64" ]; then \
    az bicep install --target-platform linux-arm64; \
    else \
    az bicep install; \
    fi;

2. after installing the version:

$ az bicep version
Bicep CLI version 0.26.170 (e9abaf16a5)

So far so good, but when we lint a very short main.bicep we get an exception:

$ touch main.bicep
$ az bicep lint -f main.bicep 

ERROR: Unhandled exception. System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space.
   at System.Runtime.CompilerServices.RuntimeHelpers.EnsureSufficientExecutionStack()
   at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node)
   at Bicep.Core.Parsing.Parser.Program()
   at Bicep.Core.Workspaces.SourceFileFactory.CreateBicepFile(Uri fileUri, String fileContents)
   at Bicep.Core.Workspaces.SourceFileFactory.TryCreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
   at Bicep.Core.Workspaces.SourceFileFactory.CreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
   at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResult(Uri fileUri, ArtifactReference moduleReference)
   at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResultWithCaching(Uri fileUri, ArtifactReference reference)
   at Bicep.Core.Workspaces.SourceFileGroupingBuilder.PopulateRecursive(Uri fileUri, ArtifactReference reference, ImmutableHashSet`1 sourceFilesToRebuild, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager)
   at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(Uri entryFileUri, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager, ImmutableHashSet`1 sourceFilesToRebuild)
   at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(IFileResolver fileResolver, IModuleDispatcher moduleDispatcher, IConfigurationManager configurationManager, IReadOnlyWorkspace workspace, Uri entryFileUri, IFeatureProviderFactory featuresFactory, Boolean forceModulesRestore)
   at Bicep.Core.BicepCompiler.CreateCompilationWithoutRestore(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean markAllForRestore)
   at Bicep.Core.BicepCompiler.CreateCompilation(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean skipRestore, Boolean forceRestore)
   at Bicep.Cli.Commands.LintCommand.RunAsync(LintArguments args)
   at Bicep.Cli.Program.RunAsync(String[] args, CancellationToken cancellationToken)
   at Bicep.Cli.Program.<>c__DisplayClass3_0.<<Main>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Bicep.Cli.Program.RunWithCancellationAsync(Func`2 runFunc)
   at Bicep.Cli.Program.Main(String[] args)
   at Bicep.Cli.Program.<Main>(String[] args)

Related command

az bicep lint

Errors

ERROR: Unhandled exception. System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space.
at System.Runtime.CompilerServices.RuntimeHelpers.EnsureSufficientExecutionStack()
at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node)
at Bicep.Core.Parsing.Parser.Program()
at Bicep.Core.Workspaces.SourceFileFactory.CreateBicepFile(Uri fileUri, String fileContents)
at Bicep.Core.Workspaces.SourceFileFactory.TryCreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileFactory.CreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResult(Uri fileUri, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResultWithCaching(Uri fileUri, ArtifactReference reference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.PopulateRecursive(Uri fileUri, ArtifactReference reference, ImmutableHashSet1 sourceFilesToRebuild, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(Uri entryFileUri, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager, ImmutableHashSet1 sourceFilesToRebuild)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(IFileResolver fileResolver, IModuleDispatcher moduleDispatcher, IConfigurationManager configurationManager, IReadOnlyWorkspace workspace, Uri entryFileUri, IFeatureProviderFactory featuresFactory, Boolean forceModulesRestore)
at Bicep.Core.BicepCompiler.CreateCompilationWithoutRestore(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean markAllForRestore)
at Bicep.Core.BicepCompiler.CreateCompilation(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean skipRestore, Boolean forceRestore)
at Bicep.Cli.Commands.LintCommand.RunAsync(LintArguments args)
at Bicep.Cli.Program.RunAsync(String[] args, CancellationToken cancellationToken)
at Bicep.Cli.Program.<>c__DisplayClass3_0.<

b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Bicep.Cli.Program.RunWithCancellationAsync(Func`2 runFunc)
at Bicep.Cli.Program.Main(String[] args)
at Bicep.Cli.Program.(String[] args)

Issue script & Debug output

cli.knack.cli: Command arguments: ['bicep', 'lint', '-f', 'main.bicep', '--debug']
cli.knack.cli: init debug log:
Enable color in terminal.
cli.knack.cli: Event: Cli.PreExecute []
cli.knack.cli: Event: CommandParser.OnGlobalArgumentsCreate [<function CLILogging.on_global_arguments at 0xffff83fc00e0>, <function OutputProducer.on_global_arguments at 0xffff83eb6480>, <function CLIQuery.on_global_arguments at 0xffff83eeff60>]
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableCreate []
cli.azure.cli.core: Modules found from index for 'bicep': ['azure.cli.command_modules.resource']
cli.azure.cli.core: Loading command modules:
cli.azure.cli.core: Name Load Time Groups Commands
cli.azure.cli.core: resource 0.053 51 228
cli.azure.cli.core: Total (1) 0.053 51 228
cli.azure.cli.core: Loaded 51 groups, 228 commands.
cli.azure.cli.core: Found a match in the command table.
cli.azure.cli.core: Raw command : bicep lint
cli.azure.cli.core: Command table: bicep lint
cli.knack.cli: Event: CommandInvoker.OnPreCommandTableTruncate [<function AzCliLogging.init_command_file_logging at 0xffff820640e0>]
cli.azure.cli.core.azlogging: metadata file logging enabled - writing logs to '/root/.azure/commands/2024-04-16.09-06-58.bicep_lint.59.log'.
az_command_data_logger: command args: bicep lint -f {} --debug
cli.knack.cli: Event: CommandInvoker.OnPreArgumentLoad [<function register_global_subscription_argument..add_subscription_parameter at 0xffff82085d00>]
cli.knack.cli: Event: CommandInvoker.OnPostArgumentLoad []
cli.knack.cli: Event: CommandInvoker.OnPostCommandTableCreate [<function register_ids_argument..add_ids_arguments at 0xffff820d04a0>, <function register_cache_arguments..add_cache_arguments at 0xffff820d05e0>]
cli.knack.cli: Event: CommandInvoker.OnCommandTableLoaded []
cli.knack.cli: Event: CommandInvoker.OnPreParseArgs []
cli.knack.cli: Event: CommandInvoker.OnPostParseArgs [<function OutputProducer.handle_output_argument at 0xffff83eb6520>, <function CLIQuery.handle_query_parameter at 0xffff83bdc040>, <function register_ids_argument..parse_ids_arguments at 0xffff820d0540>]
cli.azure.cli.command_modules.resource._bicep: Current value of "use_binary_from_path": false.
cli.azure.cli.command_modules.resource._bicep: Bicep CLI installation path: /root/.azure/bin/bicep
cli.azure.cli.command_modules.resource._bicep: Bicep CLI installed: True.
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/resource/_bicep.py", line 312, in _run_command
process.check_returncode()
File "/usr/local/lib/python3.11/subprocess.py", line 502, in check_returncode
raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['/root/.azure/bin/bicep', 'lint', 'main.bicep']' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 664, in execute
raise ex
File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 731, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 701, in _run_job
result = cmd_copy(params)
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/init.py", line 334, in call
return self.handler(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/core/commands/command_operation.py", line 121, in handler
return op(**command_args)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/resource/custom.py", line 4608, in lint_bicep_file
output = run_bicep_command(cmd.cli_ctx, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/resource/_bicep.py", line 107, in run_bicep_command
return _run_command(installation_path, args, custom_env)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/azure/cli/command_modules/resource/_bicep.py", line 328, in _run_command
raise UnclassifiedUserFault(error_msg)
azure.cli.core.azclierror.UnclassifiedUserFault: Unhandled exception. System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space.
at System.Runtime.CompilerServices.RuntimeHelpers.EnsureSufficientExecutionStack()
at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node)
at Bicep.Core.Parsing.Parser.Program()
at Bicep.Core.Workspaces.SourceFileFactory.CreateBicepFile(Uri fileUri, String fileContents)
at Bicep.Core.Workspaces.SourceFileFactory.TryCreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileFactory.CreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResult(Uri fileUri, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResultWithCaching(Uri fileUri, ArtifactReference reference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.PopulateRecursive(Uri fileUri, ArtifactReference reference, ImmutableHashSet1 sourceFilesToRebuild, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(Uri entryFileUri, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager, ImmutableHashSet1 sourceFilesToRebuild)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(IFileResolver fileResolver, IModuleDispatcher moduleDispatcher, IConfigurationManager configurationManager, IReadOnlyWorkspace workspace, Uri entryFileUri, IFeatureProviderFactory featuresFactory, Boolean forceModulesRestore)
at Bicep.Core.BicepCompiler.CreateCompilationWithoutRestore(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean markAllForRestore)
at Bicep.Core.BicepCompiler.CreateCompilation(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean skipRestore, Boolean forceRestore)
at Bicep.Cli.Commands.LintCommand.RunAsync(LintArguments args)
at Bicep.Cli.Program.RunAsync(String[] args, CancellationToken cancellationToken)
at Bicep.Cli.Program.<>c__DisplayClass3_0.<

b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Bicep.Cli.Program.RunWithCancellationAsync(Func`2 runFunc)
at Bicep.Cli.Program.Main(String[] args)
at Bicep.Cli.Program.(String[] args)

cli.azure.cli.core.azclierror: Unhandled exception. System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space.
at System.Runtime.CompilerServices.RuntimeHelpers.EnsureSufficientExecutionStack()
at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node)
at Bicep.Core.Parsing.Parser.Program()
at Bicep.Core.Workspaces.SourceFileFactory.CreateBicepFile(Uri fileUri, String fileContents)
at Bicep.Core.Workspaces.SourceFileFactory.TryCreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileFactory.CreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResult(Uri fileUri, ArtifactReference moduleReference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResultWithCaching(Uri fileUri, ArtifactReference reference)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.PopulateRecursive(Uri fileUri, ArtifactReference reference, ImmutableHashSet1 sourceFilesToRebuild, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(Uri entryFileUri, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager, ImmutableHashSet1 sourceFilesToRebuild)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(IFileResolver fileResolver, IModuleDispatcher moduleDispatcher, IConfigurationManager configurationManager, IReadOnlyWorkspace workspace, Uri entryFileUri, IFeatureProviderFactory featuresFactory, Boolean forceModulesRestore)
at Bicep.Core.BicepCompiler.CreateCompilationWithoutRestore(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean markAllForRestore)
at Bicep.Core.BicepCompiler.CreateCompilation(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean skipRestore, Boolean forceRestore)
at Bicep.Cli.Commands.LintCommand.RunAsync(LintArguments args)
at Bicep.Cli.Program.RunAsync(String[] args, CancellationToken cancellationToken)
at Bicep.Cli.Program.<>c__DisplayClass3_0.<

b__0>d.MoveNext()
--- End of stack trace from previous location ---
at Bicep.Cli.Program.RunWithCancellationAsync(Func2 runFunc) at Bicep.Cli.Program.Main(String[] args) at Bicep.Cli.Program.<Main>(String[] args) az_command_data_logger: Unhandled exception. System.InsufficientExecutionStackException: Insufficient stack to continue executing the program safely. This can happen from having too many functions on the call stack or function on the stack using too much stack space. at System.Runtime.CompilerServices.RuntimeHelpers.EnsureSufficientExecutionStack() at Bicep.Core.Syntax.SyntaxVisitor.Visit(SyntaxBase node) at Bicep.Core.Parsing.Parser.Program() at Bicep.Core.Workspaces.SourceFileFactory.CreateBicepFile(Uri fileUri, String fileContents) at Bicep.Core.Workspaces.SourceFileFactory.TryCreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference) at Bicep.Core.Workspaces.SourceFileFactory.CreateSourceFile(Uri fileUri, String fileContents, ArtifactReference moduleReference) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResult(Uri fileUri, ArtifactReference moduleReference) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.GetFileResolutionResultWithCaching(Uri fileUri, ArtifactReference reference) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.PopulateRecursive(Uri fileUri, ArtifactReference reference, ImmutableHashSet1 sourceFilesToRebuild, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager)
at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(Uri entryFileUri, IFeatureProviderFactory featuresFactory, IConfigurationManager configurationManager, ImmutableHashSet1 sourceFilesToRebuild) at Bicep.Core.Workspaces.SourceFileGroupingBuilder.Build(IFileResolver fileResolver, IModuleDispatcher moduleDispatcher, IConfigurationManager configurationManager, IReadOnlyWorkspace workspace, Uri entryFileUri, IFeatureProviderFactory featuresFactory, Boolean forceModulesRestore) at Bicep.Core.BicepCompiler.CreateCompilationWithoutRestore(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean markAllForRestore) at Bicep.Core.BicepCompiler.CreateCompilation(Uri bicepUri, IReadOnlyWorkspace workspace, Boolean skipRestore, Boolean forceRestore) at Bicep.Cli.Commands.LintCommand.RunAsync(LintArguments args) at Bicep.Cli.Program.RunAsync(String[] args, CancellationToken cancellationToken) at Bicep.Cli.Program.<>c__DisplayClass3_0.<<Main>b__0>d.MoveNext() --- End of stack trace from previous location --- at Bicep.Cli.Program.RunWithCancellationAsync(Func2 runFunc)
at Bicep.Cli.Program.Main(String[] args)
at Bicep.Cli.Program.(String[] args)
cli.knack.cli: Event: Cli.PostExecute [<function AzCliLogging.deinit_cmd_metadata_logging at 0xffff82064360>]
az_command_data_logger: exit code: 1
cli.main: Command ran in 0.974 seconds (init: 0.130, invoke: 0.845)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.client: Accumulated 0 events. Flush the clients.
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 5947 in cache
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/usr/local/bin/python /usr/local/lib/python3.11/site-packages/azure/cli/telemetry/init.py /root/.azure"
telemetry.process: Return from creating process
telemetry.main: Finish creating telemetry upload process.

Expected behavior

Should lint the file as on amd64 images.

Environment Summary

azure-cli                         2.59.0

core                              2.59.0
telemetry                          1.1.0

Dependencies:
msal                              1.27.0
azure-mgmt-resource             23.1.0b2

Python location '/usr/local/bin/python'
Extensions directory '/root/.azure/cliextensions'

Python (Linux) 3.11.8 (main, Mar 16 2024, 09:26:53) [GCC 13.2.1 20231014]

Legal docs and information: aka.ms/AzureCliLegal


Your CLI is up-to-date.

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[KrisSimon]

A litte bit more background about the broken microsoft image mcr.microsoft.com/azure-cli::

Doing this image to fix the aarch64 bug does not work like described above:

# Own version of Microsoft's `az` container
# ======================================================================================================================
# This container is needed, because of a bug in Bicep that does not detect the target system correctly
# See:
#   - https://github.com/Azure/azure-cli/issues/24810
#   - https://github.com/Azure/azure-cli/issues/25105
#   - https://github.com/Azure/azure-cli/pull/27194
#   - https://github.com/Azure/azure-cli/issues/27333
#   - https://github.com/Azure/azure-cli/issues/27429
# ======================================================================================================================
ARG AZ_IMAGE_VERSION
FROM mcr.microsoft.com/azure-cli:${AZ_IMAGE_VERSION} as base
# ----------------------------------------------------------------------------------------------------------------------

# Install packages; Update system
RUN apk update; apk upgrade; apk add curl

# Prepare system
WORKDIR /tmp
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
RUN mkdir -p /root/.azure/

# patched Bicep, because of: https://github.com/Azure/azure-cli/issues/24810
RUN if [ "$(arch)" == "aarch64" ]; then \
    az bicep install --target-platform linux-arm64; \
    else \
    az bicep install; \
    fi;

# Verify installation
RUN az --version
RUN az bicep version

WORKDIR /project
CMD ["bash"]

Doing a own ubuntu image seams to work so far:

ARG BASEIMAGE=ubuntu:22.04
FROM ${BASEIMAGE} as base
# ----------------------------------------------------------------------------------------------------------------------

# Install packages; Update system
RUN apt update; \
    apk upgrade -y; \
    apt install -y \
      curl

# Prepare system
WORKDIR /tmp
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=1
RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash

# patched Bicep, because of: https://github.com/Azure/azure-cli/issues/24810
RUN TARGET=""; \
    if [ "$(arch)" = "aarch64" ]; then \
    TARGET="--target-platform linux-arm64"; \
    fi; \
    az bicep install ${TARGET}

# Verify installation
RUN az --version
RUN az bicep version

WORKDIR /project
CMD ["bash"]

Not only does the Bicep installer not work as expected, but also the "official" Azure Cli image from Microsoft does not seem to be suitable for use in production.