Migrate azdo resources to 1ES managed pool #51

jimson-msft · 2021-07-14T17:06:16Z

No description provided.

…s, naming cleanups (#47) * User/jimson/test hosted pool (#46) Migrate to using hosted agents/1es hosted pool Modify bootstrap script Various cleanup around paths and naming conventions

shishirb-MSFT · 2021-07-14T18:45:08Z

azure-pipelines/build/templates/doclient-lite-docker-steps.yml


 parameters:
- name: targetOsArch    # example: debian10-arm32
+- name: targetOS        # example: ubuntu1804


targetOS

targetOS seems to be unused, no? #Resolved

In all the docker and native steps templates.

Thanks good catch - after removing the install script steps, targetOS is now unused. Need to remove from here as well as the non-template yaml for docker as well.

shishirb-MSFT · 2021-07-14T18:50:18Z

azure-pipelines/publishing/github-release.yml

 - none

-pool: dotestlab
+#pool: 1es_hosted_pool_ubuntu_1804


I think we are ready to switch this pipeline also to the 1ES pool right? #Closed

jimson-msft · 2021-07-14T19:00:40Z

build/bootstrap.sh

+# 1. Using 'usermod -aG docker $USER' hangs the az pipeline agent on 'newgrp docker', which is required for the permission changes to take effect
+# 2. The rootless install succeeds, but azdo pipeline agent still reports permissions failure when calling into docker daemon
+# See if we can leverage someone elses docker install task from Image Factory when provisioning the 1ES managed image
+# After docker is callable on a 1ES managed image, we can swap all pipelines to use 1ES hosted pool instead of the microsoft hosted agent, and we can remove usage of the bootstrap script


This comment can now be deleted #Closed

The lines above, about permissions issue calling into docker daemon, can also be removed, right?

How did you resolve it?

Yes, I added comment on codeflow - but I think on github it only highlights the bottom-most line. The whole comment block can be removed.

So this comment came out of my first PR for this work, where I was using an Ubuntu1804 base image with no dependencies installed. The issue was that when trying to install docker via pipeline step - the current user is running as the self-hosted agent, with network service permissions. As a result - the pipeline agent would fail to perform certain commands, like pulling from a remote container registry.
When running this script as an ImageFactory artifact - docker gets installed with the right permissions to have the docker CLI callable through the pipeline agent's user.

So the tl;dr is that it self-resolved once I did the install as a part of image provisioning rather than trying to install docker as a pipeline step.

jimson-msft · 2021-07-14T19:28:36Z

build/bootstrap.sh

+$(basename $0) - Script to setup development environments for Delivery Optimization
+Usage: $(basename $0) --platform <platform to install for> --install <install command>
+    --platform          # Platform to provision, supported platforms: ubuntu1804, ubuntu2004, debian9, debian 10. Required
+    --install           # Which command to run, supported commands: builddependencies, developertools, containertools, qemu, all. Default is all


dependencies

fix this to just read 'build' #Closed

shishirb-MSFT · 2021-07-14T22:04:50Z

azure-pipelines/build/templates/doclient-lite-docker-steps.yml

+  inputs:
+    command: login
+    containerRegistry: doclientcontainerregistry
+    repository: $(parameters.targetOsArch)


$(parameters.targetOsArch)

You will need to rename the repositories on azure portal now to change from dash to underscore. #Resolved

The github release yaml is still using dash (ubuntu1804-x64).

IIRC, pipeline job names do not accept dash which is the reason we had the OS and arch as separate and used dash/underscore as appropriate.

Correct, saw the dash issue in pipeline job names as well. Let's use the underscore convention everywhere.

Re: repositories on azure portal - do you mean the container repository? Yeah those names will need changing then

renamed all container repositories to all use underscore, and just use a single parameter targetOsArch

shishirb-MSFT · 2021-07-14T22:08:31Z

azure-pipelines/build/doclient-lite-docker.yml

  parameters:
-    targetOS: 'debian10'
+    targetOsArch: 'debian9_arm32'
    targetArch: 'arm32'


targetArch: 'arm32'

If you don't go back to separating the OS and arch, remove targetArch here. #Resolved

missed this one

shishirb-MSFT · 2021-07-15T00:01:52Z

    targetOsArch: 'ubuntu1804-x64'

dash to underscore in all jobs here too.

In reply to: 880287518

In reply to: 880287518 #Resolved

Refers to: azure-pipelines/publishing/github-release.yml:31 in fb134bb. [](commit_id = fb134bb, deletion_comment = False)

shishirb-MSFT

revoking review

shishirb-MSFT

* Use CredScanSuppressions.json instead of inline comments (#15) * Add support for additional build types (#28) Add support for additional build types * Fix dockerfile for Deb10 (#29) * Support Ubuntu 20.04 arm64 builds and fix boost log linker issue (#31) * Update readme with section detailing install (#33) * Return specific errors from creating and accessing the destination file path (#35) * Provide better errors for dest file path. Use POSIX APIs. * Fix tests to account for this. * deliveryoptimization_set_iot_connection_string returns specific error codes * Fix WIL link * Cleanup download in all cases, avoid overwriting existing files (#36) * SDK: Ensure download is cleaned up in all cases * Agent: Avoid overwriting existing destination file * Agent: Fix file path logging in cancel event * Check binary size growth in azure pipelines (#38) * Check runs only for minsizerel flavor. * Baseline number obtained from local dev build. * Binary size increase of 5% or more results in build failure. Think carefully before increasing the baseline at any time. * SDK: Prepare to remove cpprestsdk dependency (#39) * CPortFinder provides port only, address constructed by CHttpClient. * Remove unused CHttpClient SendRequest method that supported request body. * Simplify SendRequest. Use boost ptree. * SDK: Remove cpprestsdk, use custom code with boost.asio (#40) * Custom, minimal, http parser implementation, on top of Boost.Asio. * Fix minsizerel build warning in class download. * Re-enable some tests. * Strip unneeded symbols from minsizerel binaries (#41) * Bootstrap only required boost libs * Strip unneeded symbols from minsizerel binaries. * Fix deb10 build. boost::asio::io_service backcompat. * Update limits for binary size check * Agent: Move log location to /var/log (#43) * This removes usage of /var/cache * Group write permissions needed only for the config path * Make strip_symbols optional (#44) * Some build environments do not have objcopy installed (like Yocto Zeus) * Agent: Update MCC fallback logic (#45) * Custom fallback logic: oscillate between MCC and original host when possible. * Ban MCC on fatal errors only. * 4xx errors are fatal for MCC also. * ConfigManager handling -1 fallback config. * Fix mcc manager tests not appending json files properly. * Agent: Reset retry delay on fallback from MCC * SDK: Add test to exercise new MCC fallback logic * Increment versions for release v0.7.0 (#50) agent, sdk: 0.5.0 plugin-apt: 0.3.0 * Fix github_release.yml (#52) * echo in Release Information task needs the text wrapped in quotes. * Use minsizerel for building all components. * Migrate azdo resources to 1ES managed pool (#51) * use ninja (#53) * Agent: Replace boost.log usage with custom logger (#54) * Agent: Remove boost.log requirement from test (#55) * SDK: Remove inline spaces in favor of nested namespaces (#57) Co-authored-by: shishirb-MSFT <50385517+shishirb-MSFT@users.noreply.github.com> Co-authored-by: Shishir Bhat <shishirb@microsoft.com>

jimson-msft added 2 commits July 2, 2021 13:55

AzDO Migration - Move off self-hosted pools, build script improvement…

f1a3b5d

…s, naming cleanups (#47) * User/jimson/test hosted pool (#46) Migrate to using hosted agents/1es hosted pool Modify bootstrap script Various cleanup around paths and naming conventions

Use 1ES Self-Hosted Images provisioned with dependencies (#49)

5660a50

jimson-msft requested a review from a team as a code owner July 14, 2021 17:06

shishirb-MSFT reviewed Jul 14, 2021

View reviewed changes

jimson-msft commented Jul 14, 2021

View reviewed changes

cleanup/resolve pr feedback

62ae4b8

jimson-msft commented Jul 14, 2021

View reviewed changes

jimson-msft added 3 commits July 14, 2021 12:29

revise help comment

9279497

use just 'targetosarch' now

fa6f9d1

fix

0a18a44

shishirb-MSFT reviewed Jul 14, 2021

View reviewed changes

fix

fb134bb

jimson-msft added 2 commits July 14, 2021 17:37

change to undescore

a7caf18

fix underscore in comments

cd4e3f1

shishirb-MSFT previously approved these changes Jul 15, 2021

View reviewed changes

Merge branch 'main' into feature/azdo_migration

e793b7e

shishirb-MSFT approved these changes Jul 15, 2021

View reviewed changes

jimson-msft merged commit 5fa3fb5 into main Jul 15, 2021

jimson-msft deleted the feature/azdo_migration branch July 15, 2021 17:19

Migrate azdo resources to 1ES managed pool #51

Migrate azdo resources to 1ES managed pool #51

Uh oh!

Conversation

jimson-msft commented Jul 14, 2021

Uh oh!

shishirb-MSFT Jul 14, 2021 • edited by jimson-msft Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 14, 2021 • edited by jimson-msft Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT Jul 14, 2021 • edited by jimson-msft Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimson-msft Jul 14, 2021

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT commented Jul 15, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shishirb-MSFT left a comment

Choose a reason for hiding this comment

Uh oh!

shishirb-MSFT left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shishirb-MSFT Jul 14, 2021 •

edited by jimson-msft

Loading

shishirb-MSFT Jul 14, 2021 •

edited

Loading

shishirb-MSFT Jul 14, 2021 •

edited

Loading

jimson-msft Jul 14, 2021 •

edited

Loading

shishirb-MSFT Jul 14, 2021 •

edited

Loading

jimson-msft Jul 14, 2021 •

edited

Loading

shishirb-MSFT Jul 14, 2021 •

edited by jimson-msft

Loading

jimson-msft Jul 14, 2021 •

edited

Loading

jimson-msft Jul 14, 2021 •

edited

Loading

shishirb-MSFT Jul 14, 2021 •

edited by jimson-msft

Loading

shishirb-MSFT commented Jul 15, 2021 •

edited

Loading