An Azure DevOps pipeline allows a list of repeatable tasks to be executed in an Azure environment such as creating a virtual machine. There are many options when considering automating Azure deployment using Azure DevOps pipelines (https://dev.azure.com). The goal of this document is outline common scripting language choices and provide detail in how to use them starting with the simplest and working towards more complex structures. Examples will be provided both as importable pipelines and step by step instructions to get you running right away. One concept that will be highlighted is creating modular pipelines that maximize code reuse from an associated github repo.
Before starting any automation project It’s important to understand the scope of your automation. If your system will eventually be supported by another group, you should also understand their skill level and support capability.
When considering scope some questions you may ask are whether you are automating a complete Azure environment that includes networking, resource group, compute, storage, and configuration/code OR you are only automating a subset of these components.
Another important topic is understanding whether the scripting you plan to use will use either imperative or declarative syntax as described in detail here. In summary, imperative syntax lists specifically how a task will be completed AND declarative allows assessment of the current state to determine if it is compliant and then executes the appropriate changes to bring it into compliance. Using imperative syntax scripts will execute a set of commands without assessment.
Below is a list of common Microsoft scripting languages used for infrastructure activities (network, resource groups, VMs) in Azure DevOps pipelines and the syntax type used:
| Microsoft script | Syntax Type |
|---|---|
| Powershell | Imperative |
| Azure Resource Manager (ARM) templates | Declarative |
| Powershell DSC | Declarative |
These scripting languages can be used in combinations to achieve the optimal automation covering tasks such as modification/creation of Azure objects, VM extensions, and guest VM configuration. Below is a list of common combinations:
| Microsoft script structure | How they are used |
|---|---|
| Powershell only | Azure objects, VM extensions and guest VM config |
| ARM templates | Azure objects and VM extensions |
| ARM templates calling Powershell DSC | ARM for azure objects and VM extensions calling DSC for guest VM config |
| ARM templates calling sub-ARM templates and Powershell DSC | ARM for azure objects and VM extensions calling DSC for guest VM config. Sub-ARM templates provide more modularity. |
| Powershell and Powershell DSC | Powershell for azure objects and VM extensions and DSC for guest VM config |
| Combination | Script language chosen for specific actions |
If you add in third party scripting options, you have a long list of choices and combinations. Understanding your available options will improve your pipeline design for specific situations.
Let’s start with a discussion of the simplest which is Powershell only. More complex structures will be detailed in additional sections.
Powershell provides a simple imperative method to deploy automation from Azure DevOps.
As an example of how to use Powershell in Azure Pipelines we’ll deploy a virtual machine into Azure with an associated resource group and virtual network. To follow this example, you will need the following available:
-
Azure Subscription (http://portal.azure.com)
- Permission will be required to create objects and create a service connection (if not already available)
-
Azure DevOps account (http://dev.azure.com)
If you don’t already have an Azure subscription, trial information can be found here: https://azure.microsoft.com/en-us/offers/ms-azr-0044p.
If you don’t already have an Azure DevOps account, trial information can be
found here:
https://azure.microsoft.com/en-us/services/devops
The example will be simpler if your Azure Subscription and Azure DevOps account are registered under the same Microsoft account.
If you would like to try the example, first download the github content from https://github.com/jriekse5555/InfrastructureAsCode-Powershell by using the Clone or download button as shown below.
Once this is done unzip the resulting file. We’ll use this later.
Next, sign into to an Azure DevOps account (https://dev.azure.com). If you don’t have one already there is a trial option using a Microsoft account such as outlook.com.
Next, you’ll need to create a project if don’t have one already.
- If you want to create a new project, click the Create Project button on the right-side of the screen
- Then name the project and click Create.
- You should now be on the main page of your new project
Next, we’ll initialize the Repository to hold files
- Click Repos on the left and then Files
- On the bottom of the right-pane click Initialize
When working with Azure DevOps Pipelines the two primary types of pipelines are build and release pipelines. Conceptually build pipelines are for building/preparation activities such as compiling code or copying necessary content from the DevOps repository to a storage account. Release pipelines are for deploying or releasing content into the environment. Release pipelines should be used for activities such as deploying code or virtual machines into the environment. Release pipelines provide layered version control using a base release pipeline and specific instantiations called releases.
For this example, you can either import a release pipeline from the associated github repository OR create a release pipeline step-by-step.
Regardless of whether you are planning to import the release pipeline or create one, you need to create a blank release pipeline first. The Azure DevOps portal does not allow importing a release pipeline if none already exist and if you are creating one from scratch this is the first step.
Within the Azure DevOps portal, click on Pipelines, then click Releases
If this is the first release pipeline you’ve created click the New Pipeline button on the right
If you’ve created a release pipeline previously in the middle-pane click the New button and choose New release pipeline
On the next screen choose the Empty Pipeline at the top of the choices
Close the option for changing the Stage Name with the X in the right-corner
Save the changes to your release pipeline with the Save button in the right-corner
Next, choose whether you will import or create and follow only that section.
With a blank release pipeline created and saved, navigate to the Releases section
If you’ve created a release pipeline previously in the middle-pane click the New button and choose Import release pipeline
Browse to the location of the release pipeline to import. You should have downloaded the github repository in the first example step. The proper release pipeline is on the following path in this repo.
InfrastructureAsCode-Powershell\Azure_DevOps_Pipelines_Infrastructure_BuildingBlocks\Import_AzureDevOpsPipelines\Release_Pipelines\Example_Deploy_VM_with_Powershell.json
First, lets's resolve an error related to a security feature that was recently added. Click the Variables tab. Then click the lock icon to change the local password into a secure variable as shown below:
You can now save the pipeline normally.
Next, click the Tasks button or the Stage 1 link
Click on the first step that needs attention
In the right-pane, note the agent that is used to run the release. You can change this to use the hosted 2019 version.
You cannot save the release pipeline until all errors have been resolved.
Next, click on any remaining task steps with errors. You may be missing a service connection in each of the remaining steps. You may see the following in the right-pane.
You will need a service connection between Azure DevOps and Azure. The service connection is similar to a service account and provides the permission to alter the Azure fabric from Azure DevOps.
If you already have a service connection available with appropriate permissions, choose it.
If you are using Azure DevOps for the first time and have not set a service connection up previously, click the drop-down to determine if an Azure subscription is listed. If a subscription is shown select it, then click the button Authorize to allow the connection.
If an Azure subscription is not available or you want to use a different one, click on the Manage link. Then choose the following:
Then
The basic service connection screen will appear. Next click the full version link to enter the necessary details. Setting this up is beyond the scope of this document. A service connection is required to continue with this example.
Resolve any remaining tasks steps and then Save the release pipeline.
Review the imported pipeline variables to ensure no changes are required to work in your existing environment
The release pipeline should be complete.
Continue to the section Test the release pipeline. You can skip Scenario 2 which creates a release pipeline from scratch.
A blank release pipeline should be available based on the previous instructions. If you don’t already have it open for editing, please navigate to it and click Edit. Review the previous instructions if necessary.
Click on the name of your release pipeline and type in a new name of Example_Deploy_VM_with_ Powershell
Click on any white space on the page to set the new name
Next, set the variables that will be used throughout the pipeline. Click the Variables button along the ribbon
Note that you are in the Pipeline variables section. This section is for variables for this specific pipeline. For variable reuse across pipelines, variable groups can be created in the Library.
For this example, create the following Pipeline variables by typing them in:
| Variable Name | Value |
|---|---|
| localPass | P@ssw0rd123456 |
| localUser | localUser |
| Location | EastUS |
| Network | BuildingBlocks |
| ResourceGroup | BuildingBlocks |
| Subnet | Subnet1 |
| VMName | BuildingBlocks |
| VMSize | Standard_B2s |
The localPass variable can be locked to hide the value:
Change any variable values if required to accommodate your Azure environment.
This is a good time to save your work.
Click the Tasks button to start creating task steps.
Click on Agent job.
In the right-pane, note the agent that is used to run the release. You can change this to use the hosted 2019 version.
Click the + sign on the right of Agent Job to create a task.
Type in Azure Powershell in the search box and click Add on the resulting task.
Click on the new task
In the right-pane, several settings will need to be set.
Change the Display Name to Create Resource Group. If you want to deploy to an existing resource group, you could skip this step and set the variable appropriately.
Next, you will choose or configure the service connection between Azure DevOps and Azure. The service connection is similar to a service account and provides the permission to alter the Azure fabric from Azure DevOps.
If you already have a service connection available with appropriate permissions, choose it.
If you are using Azure DevOps for the first time and have not set a service connection up previously, click the drop-down to determine if an Azure subscription is listed. If a subscription is shown select it, then click the button Authorize to allow the connection.
If an Azure subscription is not available or you want to use a different one, click on the Manage link. Then choose the following:
Then
The basic service connection screen will appear. Next click the full version link to enter the necessary details. Setting this up is beyond the scope of this document. A service connection is required to continue with this example.
Returning to configuring the first task step, you should now have a service connection set.
Choose the inline Powershell option.
Paste in the Powershell content to create a Resource Group using the pipeline variables.
#Note all variables with the syntax "$()" are DevOps variables that need to be predefined
# Create Resource Group
New-AzureRMResourceGroup -Name "$(ResourceGroup)" -Location "$(Location)"
Choose the option to use the latest Powershell version.
The first step is done. Save the pipeline
Right-click in the middle of the finished task step and choose Clone task(s).
Click on the copied task step. Change the name to Create Virtual Network and Subnet.
Paste in the Powershell content to create the virtual network and subnet using the pipeline variables.
#Note all variables with the syntax "$()" are DevOps variables that need to be predefined
# Create Virtual Network
$virtualNetwork = New-AzureRMVirtualNetwork `
-ResourceGroupName "$(ResourceGroup)" `
-Location "$(Location)" `
-Name "$(Network)" `
-AddressPrefix 11.0.0.0/16
#Create Subnet
Add-AzureRMVirtualNetworkSubnetConfig `
-Name "Subnet1" `
-AddressPrefix 11.0.0.0/24 `
-VirtualNetwork $virtualNetwork
#Associate subnet with virtual network
$virtualNetwork | Set-AzureRMVirtualNetwork
This step is done. Save the pipeline
Right-click in the middle of the finished task step and choose Clone task(s).
Click on the copied task step. Change the name to Create Virtual Network and Subnet.
Paste in the Powershell content.
#Note all variables with the syntax "$()" are DevOps variables that need to be predefined
# Creates variable for NIC name using DevOps variable for VMName with 'NIC'
suffix
$NICName = "$(VMName)" + "NIC"
# Gets Subnet ID for NIC creation
$vnet = Get-AzsureRMVirtualNetwork -Name "$(Network)" -ResourceGroupName
"$(ResourceGroup)"
$SubnetObject = $vnet.Subnets[0].Id
# Creates NIC using previous variables and DevOps variables
$NIC = New-AzureRmNetworkInterface -Name $NICName -ResourceGroupName
"$(ResourceGroup)" -Location "$(Location)" -SubnetId $SubnetObject -Force
This step is done. Save the pipeline
Right-click in the middle of the finished task step and choose Clone task(s).
Click on the copied task step. Change the name to Create Virtual Network and Subnet.
Paste in the Powershell content.
#Adds a public IP to a NIC
#Note all variables with the syntax "$()" are DevOps variables that need to
be predefined
#Sets up Public IP object name
$PublicIPName = "Public-" + "$(VMName)"
#Creates Public IP object
$PublicIP = New-AzureRmPublicIpAddress -Name $PublicIPName -ResourceGroupName
"$(ResourceGroup)" -AllocationMethod Dynamic -Location "$(Location)" -Force
#Sets NIC Name that was created previously
$NICName = "$(VMName)" + "NIC"
#Retrieves NIC object from name
$NIC = Get-AzureRmNetworkInterface -ResourceGroupName "$(ResourceGroup)"
-Name $NICName
#Assign public IP object to NIC
$nic.IpConfigurations[0].PublicIpAddress = $PublicIP
#Updates NIC object
$NIC | Set-AzureRmNetworkInterface
This step is done. Save the pipeline
Right-click in the middle of the finished task step and choose Clone task(s).
Click on the copied task step. Change the name to Create Virtual Network and Subnet.
Paste in the Powershell content to create the virtual network and subnet using the pipeline variables.
$localPass = ConvertTo-SecureString "$(localPass)" -AsPlainText -Force
$NICName = "$(VMName)" + "NIC"
$NIC = Get-AzureRmNetworkInterface -ResourceGroupName "$(ResourceGroup)" -Name $NICName
$Credential = New-Object System.Management.Automation.PSCredential ("$(localUser)", $localPass)
$VirtualMachine = New-AzureRmVMConfig -VMName "$(VMName)" -VMSize
"$(VMSize)"
$VirtualMachine = Set-AzureRmVMOperatingSystem -VM $VirtualMachine -Windows
-ComputerName "$(VMName)" -Credential $Credential -ProvisionVMAgent
-EnableAutoUpdate
$VirtualMachine = Add-AzureRmVMNetworkInterface -VM $VirtualMachine -Id
$NIC.Id
$VirtualMachine = Set-AzureRmVMSourceImage -VM $VirtualMachine -PublisherName
'MicrosoftWindowsServer' -Offer 'WindowsServer' -Skus '2016-Datacenter' -Version
'latest'
$VirtualMachine = Set-AzureRmVMOSDisk -VM $VirtualMachine -CreateOption
'FromImage' -StorageAccountType 'Standard_LRS' -Name "$(VMName)-osdisk"
$VirtualMachine = Add-AzureRmVMDataDisk -VM $VirtualMachine -Lun 0
-CreateOption 'Empty' -Name "$(VMName)-datadisk1" -StorageAccountType
'Standard_LRS' -Caching None -DiskSizeinGB 127
$VirtualMachine = Set-AzureRmVMBootDiagnostics -VM $VirtualMachine -Disable
New-AzureRmVM -ResourceGroupName "$(ResourceGroup)" -Location "$(Location)" -VM $VirtualMachine -Verbose
This step is done. Save the pipeline
You should now have a completed release pipeline created by either import or step-by-step.
A release pipeline can be deployed from several portal locations.
From the release pipeline edit view you can click the Create a release button in the top-right corner.
From the release pipeline overview…
Either way with result in a final screen, click the Create button
The deployment can be monitored within Azure DevOps.
Clicking the link after the release is created will show the status. Note that the powershell commands in this example may generate errors upon multiple reruns if objects like resource groups or virtual networks are already present. This can be overcome by disabling steps that have already run or by other methods.
If you hover you the middle status widget you can select the Logs button.
You should soon see a successful deployment and can verify the objects exist within Azure.
If you navigate to the resource group holding the objects you should the objects.
By clicking on the virtual machine, you can Connect to the virtual machine via RDP using the local admin and password configured in the variables
Click Download RDP File
Open the RDP file
Click Connect
Type in the credentials from the variables. Click OK
Click Yes
You should be successfully logged in.
Thanks for using the example. Feel free to send any feedback to jriekse5555@hotmail.com.
If you are interested in customizing the example further the associated github repository https://github.com/jriekse5555/InfrastructureAsCode-Powershell has content that may be useful.
Example 2: Azure Deployment with Azure DevOps – Leveraging a Combination of Modular Techniques with a Release Pipeline
Original Technical Design by Mike Dzikowski
Moving on from the previous simple example, the next example will use several more advanced declarative techniques executed serially by the pipeline. This will continue to allow a high degree of customization using the simple graphical pipeline interface and offer a higher degree of insurance that the desired state is reached.
This pipeline will use a series of modular ARM templates that contain an output section which allows the necessary variable to be passed back to the pipeline. A powershell script is used to convert the passed in variable into a format that can be used as an input to later modular ARM templates used later in the pipeline. You can examine the completed pipeline to see how this is done.
This example will also deploy a VM and will instead use individual ARM templates for the virtual network, public IP, virtual network interface and virtual machine. A release pipeline will be used.
Here is a graphic of the pipeline that will be constructed:
As you did in the previoius example you can either import an example pipeline or create it from scratch. These instructions will only detail the import process. If you would like to build it from scratch, you can first import the pipeline and then use it as a guide to build the design from scratch.
This example has a dependency on the resource group created in the first example so please ensure this is completed before starting.
Follow the steps from the previous example to import the example pipeline. It's located in the same location as the previous example within the git repository that was downloaded.
Once the import is completed follow these steps to resolve the errors with the pipeline. This pipeline reads from the source git repository to avoid the additional complexity of copying files to an Azure DevOps Repository or other location.
After the import it will look like the screenshot below. You will need to resolve missing information in both the artifact and the default Stage.
First, lets's resolve an error related to a security feature that was recently added. Click the Variables tab. Then click the lock icon to change the local password into a secure variable as shown below:
You can now save the pipeline normally.
Next, resolve the error with the artifact by clicking it.
Click Manage
Click New service connection and then choose GitHub
Next, we'll go to the stages section to resolve remaining issues. Click on the red information icon to start resolving the errors.
Add a hosted agent to run the pipeline as shown below and Save the pipeline:
Next, go to the Variable section and lock the default password. Then Save the pipeline.
To deploy the pipeline from the release pipeline edit view click the Create a release button in the top-right corner.
After completion you should see a 2nd VM created. You should be able to log into it with the same credentials as the first chapter. Don't forgot to deallocate the VMs if you are not actively using them to avoid compute charges. The minimal storage costs will still apply.
Example 3: Azure Deployment with Azure DevOps – Leveraging a Single ARM Template with a Release Pipeline
In the next and final section we'll go over the most popular deployment method which is a single ARM template called in a release pipeline. Using a single ARM template keeps the code in one place, allows creating multiple copies of resources using a copyindex function, and allows linking to DSC using linked ARM templates. This method is the most common Microsoft technology used in professional settings.
The one potential drawback to this approach when developing and considering the lifecycle of a solution, is the skill level required to create and change ARM templates. Especially in scenarios where its desirable that a solution be transitioned to an operational team, its good to use the best combination of techniques to keep its maintanability in line with their skillset.
Here is a graphic of the pipeline that will be constructed:
As you can see it only has one step which is to call the ARM template!
As you did in the previoius example you can either import an example pipeline or create it from scratch. These instructions will only detail the import process. If you would like to build it from scratch, you can first import the pipeline and then use it as a guide to build the design from scratch.
This example has a dependency on the resource group created in the first example so please ensure this is completed before starting.
Follow the steps from the previous example to import the example pipeline. It's located in the same location as the previous example within the git repository that was downloaded.
Once the import is completed follow these steps to resolve the errors with the pipeline. This pipeline reads from the source git repository to avoid the additional complexity of copying files to an Azure DevOps Repository or other location.
After the import it will look like the screenshot below. You will need to resolve missing information in both the artifact and the default Stage.
If you completed the example 2 the artifact will automatically connect and only errors in the Stages section will be shown. If you didn't complete example 2, refer to the instructions there on connecting to the github repo.
Next, lets's resolve an error related to a security feature that was recently added. Click the Variables tab. Then click the lock icon to change the local password into a secure variable as shown below:
You can now save the pipeline normally.
Next, click on the error shown in the Stages section to bring up the details.
As you did in the last section select the hosted agent to run the pipeline. Here's a screenshot of an appropriate choice:
The pipeline is now ready to run which you can start at any time.
Take a look at how the parameters are passed from the Variables tab into the ARM template.
It's common to see a parameters template file used to also pass parameters into the primary ARM template file. In this example the Override template parameters field is used as the sole source of parameters.
Also, take a look at the construction of the ARM template which is located in the github repository. If you plan to use ARM templates you will need to understand the details well to customize them to your needs.
This concludes this introductory series which I hope you have found worthwhile. There are many more advanced areas you can explore further if you are interested.
To assist with finding your pipelines quickly you may wish to mark them as favorites.
The following steps show you how to mark your pipelines as favorites.
The folder icon has all the pipelines. The first icon only has recent pipelines.
Click on your pipeline and then mark it as a favorite.
