Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
The `'Test-NamePrefixAvailability'` function provides you with the capability to test if the [`namePrefix`](./The%20CI%20environment%20-%20Token%20replacement.md#optional-local-custom-tokens), specified in the [`settings.json`](https://github.com/Azure/ResourceModules/blob/main/settings.json) file, conflicts with any existing resource.
The `'Test-NamePrefixAvailability'` function provides you with the capability to test if the [`namePrefix`](./The%20CI%20environment%20-%20Token%20replacement#optional-local-custom-tokens), specified in the [`settings.json`](https://github.com/Azure/ResourceModules/blob/main/settings.json) file, conflicts with any existing resource.

---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -345,7 +345,7 @@ This section will explain what is required to publish the modules to [Azure Arti
1. If you chose the feed to be project-scoped, you will need the Project Build Service account to have `Contributor` access to publish to the Azure Artifacts feed. To set this, follow the [Pipeline permission](https://docs.microsoft.com/en-us/azure/devops/artifacts/feeds/feed-permissions?view=azure-devops#pipelines-permissions) steps.

#### Implementation Guidance
Each `./azuredevops/modulePipelines` yaml pipeline already calls [`/.azuredevops/pipelineTemplates/jobs.publishModule.yml`](https://github.com/Azure/ResourceModules/blob/main/.azuredevops/pipelineTemplates/jobs.publishModule.yml). This YAML template contains a method to `Publish module to artifacts feed` via [`utilities\pipelines\resourcePublish\Publish-ModuleToUniversalArtifactFeed.ps1`](https://github.com/Azure/ResourceModules/blob/main/utilities\pipelines\resourcePublish\Publish-ModuleToUniversalArtifactFeed.ps1).
Each `./azuredevops/modulePipelines` YAML pipeline already calls [`/.azuredevops/pipelineTemplates/jobs.publishModule.yml`](https://github.com/Azure/ResourceModules/blob/main/.azuredevops/pipelineTemplates/jobs.publishModule.yml). This YAML template contains a method to `Publish module to artifacts feed` via [`utilities\pipelines\resourcePublish\Publish-ModuleToUniversalArtifactFeed.ps1`](https://github.com/Azure/ResourceModules/blob/main/utilities\pipelines\resourcePublish\Publish-ModuleToUniversalArtifactFeed.ps1).

</details>

Expand All @@ -357,9 +357,9 @@ In order to successfully deploy and test all modules in your desired environment

The repository comes with a platform pipeline, i.e., the '*dependencies pipeline*', that deploys a set of Azure services such as Virtual Networks and Key Vaults (along with dummy secrets) to be used by the module pipeline tests.

Run the dependencies pipeline by following instructions provided in the specific [Dependencies pipeline usage](./The%20CI%20environment%20-%20Pipeline%20usage.md#operate-the-dependencies-pipeline) section.
Run the dependencies pipeline by following instructions provided in the specific [Dependencies pipeline usage](./The%20CI%20environment%20-%20Pipeline%20usage#operate-the-dependencies-pipeline) section.

> **Note**: For details about the dependencies pipeline design, please refer to the dedicated [Dependencies pipeline design](./The%20CI%20environment%20-%20Pipeline%20design.md#dependencies-pipeline) section.
> **Note**: For details about the dependencies pipeline design, please refer to the dedicated [Dependencies pipeline design](./The%20CI%20environment%20-%20Pipeline%20design#dependencies-pipeline) section.

## 4.1 Manual Dependencies

Expand Down
4 changes: 2 additions & 2 deletions docs/wiki/The CI environment - Deployment validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ The intention of this test is to **fail fast**, before getting to the later depl

This step performs the actual Azure deployments using each available & configured module parameter file. The purpose of this step is to prove the module can be deployed in different configurations based on the different parameters provided. Deployments for the different variants happen in parallel.

If any of these parallel deployments require multiple/different/specific resource instances already present, these resources are deployed by the [dependencies pipeline](./The%20CI%20environment%20-%20Pipeline%20design.md#dependencies-pipeline). E.g., for the Azure Firewall to be tested with multiple configurations, the dependencies pipeline deploys multiple VNET instances, with a dedicated "AzureFirewallSubnet" in each.
If any of these parallel deployments require multiple/different/specific resource instances already present, these resources are deployed by the [dependencies pipeline](./The%20CI%20environment%20-%20Pipeline%20design#dependencies-pipeline). E.g., for the Azure Firewall to be tested with multiple configurations, the dependencies pipeline deploys multiple VNET instances, with a dedicated "AzureFirewallSubnet" in each.

The parameter files used in this stage should ideally cover as many configurations as possible to validate the template flexibility, i.e., to verify that the module can cover multiple scenarios in which the given Azure resource may be used. Using the example of the CosmosDB module, we may want to have one parameter file for the minimum amount of required parameters, one parameter file for each CosmosDB type to test individual configurations, and at least one parameter file testing the supported extension resources such as RBAC & diagnostic settings.

Expand All @@ -50,7 +50,7 @@ The removal step is triggered after the deployment completes. It removes all res
- Make sure to keep the validation subscription cost as low as possible.
- Run test deployments from scratch at every run.

However, the removal step can be skipped in case further investigation on the deployed resource is needed. This can be controlled when running the module pipeline leveraging [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design.md#module-pipeline-inputs).
However, the removal step can be skipped in case further investigation on the deployed resource is needed. This can be controlled when running the module pipeline leveraging [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design#module-pipeline-inputs).

### How it works

Expand Down
12 changes: 6 additions & 6 deletions docs/wiki/The CI environment - Pipeline design.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ The purpose of each module pipeline is twofold:
1. **Validation**: To ensure the modules hosted by the CARML library are valid and can perform the intended deployments.
1. **Publishing**: To publish _versioned_ and already validated modules to one or multiple target locations, from where they can be referenced by solutions consuming them.

As such, each pipeline can be mapped to `Phases 1 and 2` described in the [Deployment flow](./The%20context%20-%20CARML%20CI%20environment.md#deployment-flow) section.
As such, each pipeline can be mapped to `Phases 1 and 2` described in the [Deployment flow](./The%20context%20-%20CARML%20CI%20environment#deployment-flow) section.

<img src="./media/CIEnvironment/pipelineDesign.png" alt="Pipeline phases" height="500">

Expand Down Expand Up @@ -84,7 +84,7 @@ Technical documentation for each composite action, such as required input and ou
<img src="./media/CIEnvironment/pipelinePhasesADO.png" alt="Pipeline phases ADO" height="300">

Azure DevOps pipelines map each pipeline phase to a dedicated pipeline template, to maximize code reusability.
The mapping to the specific yaml template file is provided below:
The mapping to the specific YAML template file is provided below:

| Template Name | Pipeline phase |
| - | - |
Expand Down Expand Up @@ -169,7 +169,7 @@ This group of resources has a dependency only on the resource group which will h
> **Note**: The object ID of the \[user assigned identity] is needed by several dependency parameter files. However, before running the dependencies pipeline for the first time, the \[user assigned identity] resource does not exist yet, thus its object ID is unknown. For this reason, instead of the object ID value, some dependency parameter files contain the `"<<msiPrincipalId>>"` token, for which the correct value is retrieved and replaced by the pipeline at runtime.
1. Shared image gallery and definition: These resources are leveraged by the \[image template] resource.
1. Route table: This resource is leveraged by the virtual network subnet dedicated to test \[SQL managed instance].
>**Note**: This resource is deployed and configured only if sqlmi dependency resources are enabled.
>**Note**: This resource is deployed and configured only if SQL-MI dependency resources are enabled.
1. Route table: This resource is leveraged by a test subnet deployment of the \[Virtual Network] module.
1. Action group: This resource is leveraged by \[activity log alert] and \[metric alert] resources.
1. Application security group: This resource is leveraged by the \[network security group] resource.
Expand All @@ -187,7 +187,7 @@ This group of resources has a dependency on one or more resources in the group a
- '_adp-\<<namePrefix\>>-az-nsg-x-ase_': NSG with required network security rules to be leveraged by the \[app service environment] subnet.
- '_adp-\<<namePrefix\>>-az-nsg-x-bastion_': NSG with required network security rules to be leveraged by the \[bastion host] subnet.
- '_adp-\<<namePrefix\>>-az-nsg-x-sqlmi_': NSG with required network security rules to be leveraged by the \[sql managed instance] subnet.
>**Note**: This resource is deployed and configured only if sqlmi dependency resources are enabled.
>**Note**: This resource is deployed and configured only if SQL-MI dependency resources are enabled.
- '_adp-\<<namePrefix\>>-az-nsg-x-001_': default NSG leveraged by all other subnets.
1. Application insight: This resource supports monitoring, hence it has a dependency on the \[storage account], \[log analytics workspace] and \[event hub] deployed in the group above. This resource is leveraged by the \[machine learning service] resource.
1. Automation account: This resource supports monitoring, hence it has a dependency on the \[storage account], \[log analytics workspace] and \[event hub] deployed in the group above. This resource is leveraged by the \[log analytics workspace] resource.
Expand All @@ -202,7 +202,7 @@ This group of resources has a dependency on one or more resources in the group a
- '_adp-\<<namePrefix\>>-az-kv-x-001_': KV with required secrets, keys, certificates and access policies to be leveraged by all resources requiring access to a Key Vault key, secret and/or certificate, i.e., \[application gateway], \[azure NetApp file], \[azure SQL server], \[disk encryption set], \[machine learning service], \[virtual machine], \[virtual machine scale set], \[virtual network gateway connection].
- '_adp-\<<namePrefix\>>-az-kv-x-pe_': KV to be leveraged by the \[private endpoint] resource.
- '_adp-\<<namePrefix\>>-az-kv-x-sqlmi_': KV with required secrets, keys and access policies to be leveraged by the \[SQL managed instance] resource.
>**Note**: This resource is deployed and configured only if sqlmi dependency resources are enabled.
>**Note**: This resource is deployed and configured only if SQL-MI dependency resources are enabled.
>**Note**: This resource has a global scope name.
1. Recovery services vault: This resource supports monitoring, hence it has a dependency on the \[storage account], \[log analytics workspace] and \[event hub] deployed in the group above. This resource is leveraged by the \[virtual machine] resource when backup is enabled.

Expand All @@ -220,7 +220,7 @@ This group of resources has a dependency on one or more resources in the groups
>**Note**: This resource is deployed and configured only if sqlmi dependency resources are enabled.
- '_adp-\<<namePrefix\>>-az-vnet-x-001_': Hosting multiple subnets to be leveraged by \[virtual machine], \[virtual machine scale set], \[service bus], \[azure NetApp files], \[azure bastion], \[private endpoints], \[app service environment] and \[application gateway] resources.
1. Azure Image Builder template: This resource triggers the build and distribution of a VHD in a storage account. The VHD file is copied to a known storage account blob container and leveraged by \[compute disks] and \[compute images] resources.
>**Note**: This resource is deployed and configured only if the 'Enable deployment of a vhd stored in a blob container' option is selected.
>**Note**: This resource is deployed and configured only if the 'Enable deployment of a VHD stored in a blob container' option is selected.

#### **5th level resources**

Expand Down
6 changes: 3 additions & 3 deletions docs/wiki/The CI environment - Pipeline usage.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ To run the dependencies pipeline you can perform the following steps:

By default, i.e., by keeping the optional flags disables, the dependencies pipeline may take up to 30 minutes to run.

> **Note:** For details about the dependencies pipeline design, please refer to the dedicated [Dependencies pipeline design](./The%20CI%20environment%20-%20Pipeline%20design.md#dependencies-pipeline) section.
> **Note:** For details about the dependencies pipeline design, please refer to the dedicated [Dependencies pipeline design](./The%20CI%20environment%20-%20Pipeline%20design#dependencies-pipeline) section.

## Add a new dependency

Expand Down Expand Up @@ -92,7 +92,7 @@ This section focuses on _GitHub_ Actions & Workflows.

<img src="./media/CIEnvironment/gHtriggerPipeline.png" alt="Run workflow" height="350">

>**Note**: Depending on the pipeline you selected you may have additional input parameters you can provide aside from the branch. An outline can be found in the [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design.md#module-pipeline-inputs) section.
>**Note**: Depending on the pipeline you selected you may have additional input parameters you can provide aside from the branch. An outline can be found in the [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design#module-pipeline-inputs) section.

### Register a workflow

Expand Down Expand Up @@ -124,7 +124,7 @@ This section focuses on _Azure DevOps_ pipelines.

<img src="./media/CIEnvironment/pipelineStart3.png" alt="Pipeline start step 3" height="400">

>**Note**: Depending on the pipeline you selected you may have additional input parameters you can provide aside from the branch. An outline can be found in the [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design.md#module-pipeline-inputs) section.
>**Note**: Depending on the pipeline you selected you may have additional input parameters you can provide aside from the branch. An outline can be found in the [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design#module-pipeline-inputs) section.

### Register a pipeline

Expand Down
2 changes: 1 addition & 1 deletion docs/wiki/The CI environment - Publishing.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The publishing phase concludes each module's pipeline. If all previous tests suc
- _[Azure DevOps universal packages](https://docs.microsoft.com/en-us/azure/devops/artifacts/concepts/feeds?view=azure-devops)_.
> Note: this is only available if using Azure DevOps pipelines.

Besides the publishing phase's runtime, there is also the possibility to set the switch `Publish prerelease module`. This switch makes it possible to publish a prerelease version in every workflow run that is not based on `main` or `master`. This can be controlled when running the module pipeline leveraging [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design.md#module-pipeline-inputs).
Besides the publishing phase's runtime, there is also the possibility to set the switch `Publish prerelease module`. This switch makes it possible to publish a prerelease version in every workflow run that is not based on `main` or `master`. This can be controlled when running the module pipeline leveraging [Module pipeline inputs](./The%20CI%20environment%20-%20Pipeline%20design#module-pipeline-inputs).

> **Note**<br>
> The `version` used for publishing any artifact is the same for all three target locations, which reduces the maintenance effort.
Expand Down
2 changes: 1 addition & 1 deletion docs/wiki/The CI environment - Static validation.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ The following activities are performed by the [`arm/.global/global.module.tests.
- **Deployment template tests** check the template's structure and elements for errors as well as consistency matters, e.g.:
- template file (or the built bicep template) converts from JSON and has all expected properties
- variable names are camelCase
- the minimum set of outputs is returned (see [module design](./The%20library%20-%20Module%20design.md#outputs))
- the minimum set of outputs is returned (see [module design](./The%20library%20-%20Module%20design#outputs))
- **Module (readme) documentation** contains all required sections, e.g.:
- is not empty
- contains all the mandatory sections
Expand Down
4 changes: 2 additions & 2 deletions docs/wiki/The CI environment - Token replacement.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ This section provides details on the tokens replacement functionality that enabl

Tokens allow you to test deploying modules in your own environment (i.e., using tokens for your naming conventions), or apply other customizations to your resources (i.e., injecting a subscription ID inside a Resource ID string).

The [module pipelines](./The%20CI%20environment%20-%20Pipeline%20design.md#module-pipelines) leverage a token replacement function that enables parameter files to contain tokens (i.e., `<<subscriptionId>>`, `<<tenantId>>`) instead of using static values. This helps with the following:
The [module pipelines](./The%20CI%20environment%20-%20Pipeline%20design#module-pipelines) leverage a token replacement function that enables parameter files to contain tokens (i.e., `<<subscriptionId>>`, `<<tenantId>>`) instead of using static values. This helps with the following:

- Allows the repository to be portable without having static values from where it was cloned.
- Enables dynamic updates of the tokens from single locations without having to modify all files.
Expand Down Expand Up @@ -69,7 +69,7 @@ Once the Key Vault is deployed, you'll notice that the Key Vault name in Azure w

The token prefix `'<<'` and suffix `'>>'` in the above example are also configurable in the [Settings.json](https://github.com/Azure/ResourceModules/blob/main/settings.json) file.

The solution comes with one predefined local token `namePrefix`. When validating modules through the CI environment, you must update it to a custom value as described in the [Update default nameprefix](./Getting%20started%20-%20Scenario%202%20Onboard%20module%20library%20and%20CI%20environment.md#31-update-default-nameprefix) paragraph. This is done to avoid conflicts with resources requiring a globally unique name, such as storage accounts or Key Vaults.
The solution comes with one predefined local token `namePrefix`. When validating modules through the CI environment, you must update it to a custom value as described in the [Update default nameprefix](./Getting%20started%20-%20Scenario%202%20Onboard%20module%20library%20and%20CI%20environment#31-update-default-nameprefix) paragraph. This is done to avoid conflicts with resources requiring a globally unique name, such as storage accounts or Key Vaults.

> **Note**: Do not store sensitive information in this location as they will be present in your Git History. Follow best [practices and guidelines](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/best-practices#security-recommendations-for-parameters) on how to handle secrets in template deployments.

Expand Down