AzDO Migration - Move off self-hosted pools, build script improvements, naming cleanups by jimson-msft · Pull Request #47 · microsoft/do-client

jimson-msft · 2021-07-01T16:24:02Z

This is part 1 of the migration work to use 1ES Hosted agents:

Migrate to using microsoft-hosted agents and 1es hosted agents in all pipeline yaml files
Converge to using a single bootstrap script
Various cleanup around paths and naming conventions

TODO:
Native builds are using a 1ES hosted image of a blank Ubuntu 18.04 image, we can use ImageFactory artifacts to preinstall our dependencies to reduce COGS
Docker builds are using Microsoft-hosted agents which contain docker preinstalled, this is because trying to install and use docker on a blank ubuntu 18.04 image leads to permissions issues with the az pipelines agent calling the docker cli

See: https://deliveryoptimization.visualstudio.com/client/_build
All runs succesful

Migrate to using hosted agents/1es hosted pool Modify bootstrap script Various cleanup around paths and naming conventions

jimson-msft · 2021-07-01T16:29:13Z

build/bootstrap.sh

@@ -0,0 +1,220 @@
+#! /bin/bash


Part of the motivation behind this script change is to be able to invoke it as an image factory artifact to help us provision our dependencies when constructing a managed image #Closed

jimson-msft · 2021-07-01T17:31:00Z

azure-pipelines/build/templates/dosdkcpp-docker-steps.yml


 parameters:
- name: targetOsArch    # example: debian10-arm32
+- name: targetOS        # example: ubuntu18.04


ubuntu18.04

nit: ubuntu1804, same with line 7 #Resolved

jimson-msft · 2021-07-01T17:31:43Z

azure-pipelines/publishing/github-release.yml

 - none

-pool: dotestlab
+pool: 1es_hosted_pool_ubuntu_1804


This should actually be using the msft hosted agent, until 1ES machine can support docker #Resolved

jimson-msft · 2021-07-01T17:32:08Z

build/bootstrap.sh

+# This script handles provisioning of the Delivery Optimization client components on supported platforms
+###
+
+# bootstrap scripts will exit immediately if a command exits with a non-zero status


can remove this comment #Closed

Maybe keep it and shorten it to 'Treat all command failures as fatal'

shishirb-MSFT · 2021-07-01T17:38:53Z

build/docker/debian10/arm32/Dockerfile

 WORKDIR /code

-ENTRYPOINT [ "/bin/bash", "-c" ]
+ENTRYPOINT [ "/bin/bash", "-c"]


]

nit: revert the change (spacing) #Closed

jimson-msft · 2021-07-01T17:43:28Z

build/bootstrap.sh

+    cat <<EOM
+$(basename $0) - Script to setup development environments for Delivery Optimization
+Usage: $(basename $0) --platform <platform to install for> --install <install command>
+    --platform          # Platform to provision, supported platforms: Ubuntu1804, Ubuntu 2004, Debian9, Debian 10. Default is Ubuntu1804


nit: space #Closed

jimson-msft · 2021-07-01T17:44:35Z

build/bootstrap.sh

+    exit 1
+}
+
+function parseArgs() {


function parseArgs()

not the most familiar with shell scripting, but I think the () can be removed here, usage() can also be a function to stay consistent #Resolved

shishirb-MSFT · 2021-07-01T17:46:27Z

build/bootstrap.sh

+                echo -e "[INFO] Platform set to: ${PLATFORM}"
+            else 
+                echo -e "[ERROR] Unsupported platform: ${PLATFORM}"
+                exit


exit

Exit with an error? #Closed

shishirb-MSFT · 2021-07-01T17:47:23Z

build/bootstrap.sh

+    cat <<EOM
+$(basename $0) - Script to setup development environments for Delivery Optimization
+Usage: $(basename $0) --platform <platform to install for> --install <install command>
+    --platform          # Platform to provision, supported platforms: Ubuntu1804, Ubuntu 2004, Debian9, Debian 10. Default is Ubuntu1804


Platform to provision, supported platforms: Ubuntu1804, Ubuntu 2004, Debian9, Debian 10. Default is Ubuntu1804

Do these comments get printed to the terminal? If not, consider changing them to normal strings that get printed. #Closed

Yes they get printed

Ok, then change the casing on the platform strings (all lowercase).

jimson-msft · 2021-07-01T17:50:20Z

build/bootstrap.sh

+#! /bin/bash
+
+###
+# This script handles provisioning of the Delivery Optimization client components on supported platforms


modify this to read "...handles provisiong of build environment for the Delivery Optimization..." #Resolved

shishirb-MSFT · 2021-07-01T18:20:03Z

build/bootstrap.sh

+        echo "[INFO] Platform check succesful"
+    else
+        echo "[WARNING] No platform supplied, using default: Ubuntu1804"
+        PLATFORM = "ubuntu1804"


Consider not having a default. It is important enough to explicitly specify. #Closed

shishirb-MSFT · 2021-07-01T18:22:46Z

build/bootstrap.sh

+
+    # Most target platforms do not natively have a version of cpprest that supports url-redirection
+    # Build and install v2.10.16 as it's the earliest version which supports url-redirection
+    mkdir /tmp/cpprestsdk


mkdir /tmp/cpprestsdk

Script will fail if directory already exists. I've been bitten by this. Consider deleting the directory if exists and then continue with creating the dir.

Same for /tmp/gtest and other custom dirs we use. #Closed

shishirb-MSFT · 2021-07-01T18:23:22Z

build/bootstrap.sh

+$(basename $0) - Script to setup development environments for Delivery Optimization
+Usage: $(basename $0) --platform <platform to install for> --install <install command>
+    --platform          # Platform to provision, supported platforms: Ubuntu1804, Ubuntu 2004, Debian9, Debian 10. Default is Ubuntu1804
+    --install           # Which command to run, supported commands: builddependencies, developertools, containertools, qemu, all. Default is All


All

all (lowercase) #Closed

shishirb-MSFT · 2021-07-01T18:34:24Z

azure-pipelines/build/templates/dopapt-docker-steps.yml

+  inputs:
+    script: 'sudo ./bootstrap.sh --platform ${{parameters.targetOS}} --install containertools'
+    workingDirectory: '$(Build.SourcesDirectory)/build'
+  displayName: 'Install Docker'


displayName: 'Install Docker'

Why does only the plugin arm build require docker to be installed? #Closed

Oops, forgot to remove

shishirb-MSFT · 2021-07-01T18:35:52Z

azure-pipelines/build/templates/dosdkcpp-native-steps.yml

  displayName: 'Install libdeliveryoptimization Debian Package'

- task: PythonScript@0
-  condition: eq('${{parameters.skipTests}}', false)


Why remove this condition? No need to build tests in release mode. #Closed

oops, good catch

jimson-msft · 2021-07-01T18:35:58Z

build/bootstrap.sh

+{
+    parseArgs "$@"
+
+


nit: extra space #Resolved

jimson-msft · 2021-07-02T17:42:13Z

Oops, need to undo #Resolved

Refers to: azure-pipelines/build/templates/doclient-lite-docker-steps.yml:26 in 7933105. [](commit_id = 7933105, deletion_comment = True)

shishirb-MSFT · 2021-07-02T19:59:07Z

azure-pipelines/build/templates/doclient-lite-docker-steps.yml

  type: string

 steps:
+- task: Docker@2


I think all the steps from login to install & register qemu can be moved to a new template and referenced from do-docker-jobs.yml.

Not urgent, feel free to open a task. #Closed

Well, the registering of QEMU will happen during image provisioning (once this script is uploaded as an artifact to Image Factory), which leaves just the login and pull steps, which could be templatized further, but at that point I'm not sure if it's worth it for two tasks

Agreed, may not be worth it for just a couple of steps.

shishirb-MSFT

jimson-msft and others added 2 commits July 1, 2021 09:09

User/jimson/test hosted pool (#46)

ff7789a

Migrate to using hosted agents/1es hosted pool Modify bootstrap script Various cleanup around paths and naming conventions

cleanup and modify some comments

7933105

jimson-msft requested a review from a team as a code owner July 1, 2021 16:24

jimson-msft changed the title ~~User/jimson/test hosted pool (#46)~~ AzDO Migration - Move off self-hosted pools, build script improvements, naming cleanups Jul 1, 2021

jimson-msft commented Jul 1, 2021

View reviewed changes

shishirb-MSFT reviewed Jul 1, 2021

View reviewed changes

jimson-msft commented Jul 1, 2021

View reviewed changes

shishirb-MSFT reviewed Jul 1, 2021

View reviewed changes

jimson-msft commented Jul 1, 2021

View reviewed changes

shishirb-MSFT reviewed Jul 1, 2021

View reviewed changes

jimson-msft commented Jul 1, 2021

View reviewed changes

build/bootstrap.sh

{

parseArgs "$@"

Copy link

Contributor Author

jimson-msft Jul 1, 2021 •

edited

Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: extra space #Resolved

jimson-msft added 2 commits July 1, 2021 16:58

resolve pr comments

da811eb

fix tabs

8accb2b

fix

f6ca315

shishirb-MSFT reviewed Jul 2, 2021

View reviewed changes

shishirb-MSFT approved these changes Jul 2, 2021

View reviewed changes

jimson-msft merged commit f1a3b5d into feature/azdo_migration Jul 2, 2021

jimson-msft deleted the user/jimson/azdo_migration branch July 2, 2021 20:55

Conversation

jimson-msft commented Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimson-msft Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021 • edited by shishirb-MSFT Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Platform to provision, supported platforms: Ubuntu1804, Ubuntu 2004, Debian9, Debian 10. Default is Ubuntu1804

Uh oh!

jimson-msft Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 1, 2021

Choose a reason for hiding this comment

Uh oh!

jimson-msft commented Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited by shishirb-MSFT

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

shishirb-MSFT Jul 1, 2021 •

edited

Loading

jimson-msft Jul 1, 2021 •

edited

Loading

jimson-msft commented Jul 2, 2021 •

edited

Loading

shishirb-MSFT Jul 2, 2021 •

edited by jimson-msft

Loading