Storage account: Deployment fails in main (Identity issue)

[AlexanderSehr]

[ahmadabdalla]

That's interesting. Although there is an issue with the principal ID as I'll detail below, but there's another thing interesting here as this was working for about a week without issues. When testing locally, I can see that the ARM API is not returning an exception in this case. The $res variable contains data. Tested using our New-ModuleDeployment and regular New-AzResourceGroupDeployment, To narrow if its a bug in our script. Turns out its not.

PS> New-AzResourceGroupDeployment -TemplateFile '.\arm\Microsoft.Storage\storageAccounts\deploy.bicep' -TemplateParameterFile '.\arm\Microsoft.Storage\storageAccounts\.parameters\parameters.json' -ResourceGroupName validation-rg

DeploymentName          : deploy
ResourceGroupName       : validation-rg
ProvisioningState       : Failed
Timestamp               : 12/10/2021 5:15:26 AM
Mode                    : Incremental
TemplateLink            : 
Parameters              : 
                          Name                                     Type                       Value     
                          =======================================  =========================  ==========
                          name                                     String                     sxxazsaweux001
                          location                                 String                     westeurope
                          roleAssignments                          Array                      [

The ProvisioningState is returned as Failed. So the deployment is not pushing this out as a failure. If you look at the history of this module deployment, this has started failing when we introduced the principal ID as output. I've tried getting the principal ID using the CLI to that same storage account and indeed it does not exist. So its either an issue with Storage account that it doesn't provide it although it does indeed exist. Or it isn't supported.

Get-AzStorageAccount -Name sxxazsaweux001 -ResourceGroupName validation-rg | Select-Object -ExpandProperty Identity

PrincipalId TenantId Type                         UserAssignedIdentities
----------- -------- ----                         ----------------------
                     SystemAssigned, UserAssigned

So I think there are two issues here:

• The deployment does not fail when this issue exists, which was the case for the past week until this last run on main, hence the issue was not detected.
  • So I thought it might be related to the Az.Resources Module (5.1.0) but not 100% sure as its the same version didn't fail yesterday, but locally I'm on an old version (4.4.1) and updated to (5.1.0) but still not popping up.
  • 5.1.0 was released 3 days ago and we were using 4.4.1 before that so something must have changed in the last 2 days to cause this to pop up now.
  • Nevertheless, I think our New-ModuleDeployment should look inside the $res to see the ProvisioningState if it has failed as well. So we need a ticket for that
• Also we need to remove the principal ID as output from the storage module as it isn't returned in the first place

Thoughts?
@MrMCake @MariusStorhaug

[ahmadabdalla]

Further testing scenario: Used the min.parameters.json, and set to use systemAssignedIdentity to true, and it worked.. It returned the output... Then I introduced the userAssignedIdentity and then the issue popped up again. So for the storage account identity issue. I think that's where the problem is.. I am interested to know your thoughts.

[AlexanderSehr]

Good catch, thanks @ahmadabdalla. @MariusStorhaug, any chance you could check this out?

If first thought it's related to the existing properties reference we talked about before, but it's actually not. Probably we need a dummy value? Still weird

[senavar]

Just to add, I ran the storage account workflow for my branch when I did the updates for Diagnostic Settings. My PR was just added 2 days ago with no issues during deployment. The behavior seems to be intermittent, sometimes it works other times it doesn't. The workflow ran all green in my branch for all 3 parameter files:

[ahmadabdalla]

Something like this before the return $deploymentName should be able to pick up the edge case:

if ($res.ProvisioningState -eq 'Failed') {
      throw ($res | Select-Object -Property DeploymentName, ProvisioningState, Timestamp | Out-String)
      return
}

Result:

[DBG]: PS C:\Users\ahmadabdalla\github\azure\ResourceModules> 
     |  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     |
DeploymentName                       ProvisioningState Timestamp
--------------                       ----------------- ---------
storageAccounts-20211211T0812292848Z Failed            12/10/2021 9:32:43 PM

It is truly an edge case as running it again doesn't always get into that state.
The Storage Account itself when having both SystemIdentity + UserIdentity is still an issue, although the documentation doesn't restrict this.

[eriqua]

The issue is not limited to Storage accounts but also other resources. See e.g. the RSV here

See related issue #730

Note It's also flaky. The error may not be experienced in the next run (reason why issue #730 was initially closed).