Virtual Network module fails when deploying to a network that has existing workloads in it

[ahmadabdalla]

If the virtual network already contains workloads in it, attempting to redeploy the vnet results in failure.

Description

The reason is that we are defining the subnet resources outside of the virtual network resource, and that tries to delete all the subnets in the virtual network, which fails. Although we are defining the subnets as a child resource. This doesn't work with this type of resource as all the subnets need to be in the VNET resource.

This was discovered when running the dependency pipeline (in a fork) and had the following error:

     | 3:53:28 PM - The deployment 'virtualNetworks-20220309T1503008054Z' failed with error(s). Showing 1 out of 1 error(s).
Status Message: Subnet crawl-az-subnet-x-001 is in use by /subscriptions/20d6fbfe-b049-471c-95af-1369d14d0d45/resourceGroups/validation-rg/providers/Microsoft.Network/networkInterfaces/adp-crawl-vm-01-nic-01/ipConfigurations/ipconfig01 and cannot be deleted. In order to delete the subnet, delete all the resources within the subnet. See aka.ms/deletesubnet. (Code: InUseSubnetCannotBeDeleted)
CorrelationId: b833c292-0cba-4699-91b1-3213358bbb78

Additional references.

Looking up this issue, I can see it already raised here Azure/bicep-types-az#1687 with the recommendation to have it in the single vnet resource Azure/bicep-types-az#1687 and this is also still an open item here Azure/azure-quickstart-templates#2786

Steps to reproduce

1. Create a VNET using the module
2. Create a workload in that VNET
3. Redeploy the VNET module

Screenshots

During a deployment, this is what happens inside the VNET (when it doesn't have resources)

However, this does solve it (AS A WORKAROUND) and a roll back to the PR #1081

We need to discuss on:

• Removing the child resource subnet
• Align the subnet properties to the ARM documentation https://docs.microsoft.com/en-us/azure/templates/microsoft.network/virtualnetworks/subnets?tabs=bicep#subnetpropertiesformat
• What are we forgoing from user experience and flexibility if we do that

[AlexanderSehr]

Hey @ahmadabdalla, we should definitely not roll back 1:1 as I removed it one purpose due to an issue that was caused by keeping it (most notably e.g. that the original implementation would clash with ALZ policies). I'd suggest we either find a way to reference 'existing' subnets and pass them in (e.g. with an existing reference if that doesn't throw an exception if empty), or we must duplicate the child resource's property (which would be the worst case scenario).

[ahmadabdalla]

@MrMCake what is the clash you're referring to? The subnet being embedded in the vnet resource?

[AlexanderSehr]

Hey @ahmadabdalla, the issue was that the subnets would be deployed without the NSG property and hence remove it. The choice was to either add all subnets properties (and duplicate them), or, remove the subnet property alltogether. I did the later as it seemed to logical choice at time time (not knowing it would remove the child templates).
Let's have a call about this some time.