From 006b2171cae7da3f07e89760080aaa9f721857dd Mon Sep 17 00:00:00 2001 From: Ganga Mahesh Siddem Date: Sat, 2 Apr 2022 19:14:40 -0700 Subject: [PATCH 1/5] instructions for msi test validation --- README.md | 29 ++++++---- kubernetes/omsagent.yaml | 112 ++++++++++++++++++++++++++++++++++++--- 2 files changed, 126 insertions(+), 15 deletions(-) diff --git a/README.md b/README.md index f0fa40e53..13ca3a822 100644 --- a/README.md +++ b/README.md @@ -230,7 +230,7 @@ powershell -ExecutionPolicy bypass # switch to powershell if you are not on pow ``` ##### Developer Build optimizations -If you do not want to build the image from scratch every time you make changes during development,you can choose to build the docker images that are separated out by +If you do not want to build the image from scratch every time you make changes during development,you can choose to build the docker images that are separated out by * Base image and dependencies including agent bootstrap(setup.ps1) * Agent conf and plugin changes @@ -248,7 +248,7 @@ And then run the script to build the image consisting of code and conf changes. .\build-and-publish-dev-docker-image.ps1 -image /: # trigger build code and image and publish docker hub or acr ``` -For the subsequent builds, you can just run - +For the subsequent builds, you can just run - ``` .\build-and-publish-dev-docker-image.ps1 -image /: # trigger build code and image and publish docker hub or acr @@ -316,18 +316,29 @@ Navigate to Kubernetes directory and update the yamls with latest docker image o For DEV and PROD branches, automatically deployed latest yaml with latest agent image (which automatically built by the azure devops pipeline) onto CIDEV and CIPROD AKS clusters in build subscription. So, you can use CIDEV and CIPROD AKS cluster to validate E2E. Similarly, you can set up build and release pipelines for your feature branch. +# Testing MSI Auth Mode Using Yaml + 1. Enable Monitoring addon with Managed Idenity Auth Mode either using Portal or CLI + 2. Deploy [ARM template](./scripts/onboarding/aks/onboarding-using-msi-auth/) with enabled = false to create DCR, DCR-A and link the workspace to Portal + > Note - Make sure to update the parameter values in existingClusterParam.json file and have enabled = true in template file + `az deployment group create --resource-group --template-file ./existingClusterOnboarding.json --parameters @./existingClusterParam.json` + 3. Get the MSI token (which is valid for 24 hrs.) value via `kubectl get secrets -n kube-system omsagent-aad-msi-token -o=jsonpath='{.data.token}'` + 4. Disable Monitoring addon via `az aks disable-addons -a monitoring -g -n ` + 5. Uncomment MSI auth related yaml lines, replace all the placeholder values including the MSI token value in the omsagent.yaml + 6. Deploy the omsagent.yaml via `kubectl apply -f omsagent.yaml` + > Note: use the image toggle for release E2E validation + # E2E Tests ## For executing tests 1. Deploy the omsagent.yaml with your agent image. In the yaml, make sure `ISTEST` environment variable set to `true` if its not set already -2. Update the Service Principal CLIENT_ID, CLIENT_SECRET and TENANT_ID placeholder values and apply e2e-tests.yaml to execute the tests +2. Update the Service Principal CLIENT_ID, CLIENT_SECRET and TENANT_ID placeholder values and apply e2e-tests.yaml to execute the tests > Note: Service Principal requires reader role on log analytics workspace and cluster resource to query LA and metrics ``` - cd ~/Docker-Provider/test/e2e # based on your repo path - kubectl apply -f e2e-tests.yaml # this will trigger job to run the tests in sonobuoy namespace - kubectl get po -n sonobuoy # to check the pods and jobs associated to tests - ``` + cd ~/Docker-Provider/test/e2e # based on your repo path + kubectl apply -f e2e-tests.yaml # this will trigger job to run the tests in sonobuoy namespace + kubectl get po -n sonobuoy # to check the pods and jobs associated to tests + ``` 3. Download (sonobuoy)[https://github.com/vmware-tanzu/sonobuoy/releases] on your dev box to view the results of the tests ``` results=$(sonobuoy retrieve) # downloads tar file which has logs and test results @@ -338,9 +349,9 @@ For DEV and PROD branches, automatically deployed latest yaml with latest agent ## For adding new tests 1. Add the test python file with your test code under `tests` directory -2. Build the docker image, recommended to use ACR & MCR +2. Build the docker image, recommended to use ACR & MCR ``` - cd ~/Docker-Provider/test/e2e/src # based on your repo path + cd ~/Docker-Provider/test/e2e/src # based on your repo path docker login -u -p # login to acr docker build -f ./core/Dockerfile -t /: . docker push /: diff --git a/kubernetes/omsagent.yaml b/kubernetes/omsagent.yaml index 24db6f20f..85d0ffb6b 100644 --- a/kubernetes/omsagent.yaml +++ b/kubernetes/omsagent.yaml @@ -27,6 +27,11 @@ rules: - apiGroups: ["apps", "extensions", "autoscaling"] resources: ["replicasets", "deployments", "horizontalpodautoscalers"] verbs: ["list"] + # Uncomment below lines for MSI Auth Mode testing + # - apiGroups: [""] + # resources: ["secrets"] + # resourceNames: [ "omsagent-aad-msi-token" ] + # verbs: ["get", "watch"] - nonResourceURLs: ["/metrics"] verbs: ["get"] --- @@ -297,6 +302,16 @@ data: WSID: "VALUE_WSID" KEY: "VALUE_KEY" --- +# Uncomment below lines for MSI Auth Mode testing +# apiVersion: v1 +# kind: Secret +# metadata: +# name: omsagent-aad-msi-token +# namespace: kube-system +# type: Opaque +# data: +# token: "VALUE_MSI_TOKEN" +# --- apiVersion: apps/v1 kind: DaemonSet metadata: @@ -328,6 +343,41 @@ spec: - name: ndots value: "3" containers: + # Uncomment below lines for MSI Auth Mode testing + # - name: addon-token-adapter + # command: + # - /addon-token-adapter + # args: + # - --secret-namespace=kube-system + # - --secret-name=omsagent-aad-msi-token + # - --token-server-listening-port=8888 + # - --health-server-listening-port=9999 + # # Make sure this matching with version in AKS RP side + # image: mcr.microsoft.com/aks/msi/addon-token-adapter:master.220318.3 + # imagePullPolicy: IfNotPresent + # env: + # - name: AZMON_COLLECT_ENV + # value: "false" + # livenessProbe: + # httpGet: + # path: /healthz + # port: 9999 + # initialDelaySeconds: 10 + # periodSeconds: 60 + # resources: + # limits: + # cpu: 500m + # memory: 500Mi + # requests: + # cpu: 100m + # memory: 100Mi + # securityContext: + # capabilities: + # drop: + # - ALL + # add: + # - NET_ADMIN + # - NET_RAW - name: omsagent image: "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03172022" imagePullPolicy: IfNotPresent @@ -367,8 +417,9 @@ spec: value: "VALUE_USER_ASSIGNED_IDENTITY_CLIENT_ID_VALUE" - name: AZMON_CONTAINERLOGS_ONEAGENT_REGIONS value: "koreacentral,norwayeast,eastus2" - - name: USING_AAD_MSI_AUTH - value: "false" + # Uncomment below lines for MSI Auth Mode testing + # - name: USING_AAD_MSI_AUTH + # value: "true" securityContext: privileged: true ports: @@ -448,8 +499,9 @@ spec: # Update this with the user assigned msi client id for omsagent - name: USER_ASSIGNED_IDENTITY_CLIENT_ID value: "VALUE_USER_ASSIGNED_IDENTITY_CLIENT_ID_VALUE" - - name: USING_AAD_MSI_AUTH - value: "false" + # Uncomment below lines for MSI Auth Mode testing + # - name: USING_AAD_MSI_AUTH + # value: "true" securityContext: privileged: true volumeMounts: @@ -563,6 +615,41 @@ spec: spec: serviceAccountName: omsagent containers: + # Uncomment below lines for MSI Auth Mode testing + # - name: addon-token-adapter + # command: + # - /addon-token-adapter + # args: + # - --secret-namespace=kube-system + # - --secret-name=omsagent-aad-msi-token + # - --token-server-listening-port=8888 + # - --health-server-listening-port=9999 + # # Make sure this matching with version in AKS RP side + # image: mcr.microsoft.com/aks/msi/addon-token-adapter:master.220318.3 + # imagePullPolicy: IfNotPresent + # env: + # - name: AZMON_COLLECT_ENV + # value: "false" + # livenessProbe: + # httpGet: + # path: /healthz + # port: 9999 + # initialDelaySeconds: 10 + # periodSeconds: 60 + # resources: + # limits: + # cpu: 500m + # memory: 500Mi + # requests: + # cpu: 100m + # memory: 100Mi + # securityContext: + # capabilities: + # drop: + # - ALL + # add: + # - NET_ADMIN + # - NET_RAW - name: omsagent image: "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03172022" imagePullPolicy: IfNotPresent @@ -596,8 +683,9 @@ spec: # Add the below environment variable to true only in sidecar enabled regions, else set it to false - name: SIDECAR_SCRAPING_ENABLED value: "true" - - name: USING_AAD_MSI_AUTH - value: "false" + # Uncomment below lines for MSI Auth Mode testing + # - name: USING_AAD_MSI_AUTH + # value: "true" securityContext: privileged: true ports: @@ -776,6 +864,9 @@ spec: # Add this only for clouds that require cert bootstrapping # - name: REQUIRES_CERT_BOOTSTRAP # value: "true" + # Uncomment below lines for MSI Auth Mode testing + # - name: USING_AAD_MSI_AUTH + # value: "true" volumeMounts: - mountPath: C:\ProgramData\docker\containers name: docker-windows-containers @@ -798,6 +889,10 @@ spec: - mountPath: C:\etc\kubernetes\host name: azure-json-path readOnly: true + # Uncomment below lines for MSI Auth Mode testing + # - mountPath: C:\etc\IMDS-access-token + # name: imds-token + # readOnly: true livenessProbe: exec: command: @@ -855,3 +950,8 @@ spec: secret: secretName: omsagent-adx-secret optional: true + # Uncomment below lines for MSI Auth Mode testing + # - name: imds-token + # secret: + # secretName: omsagent-aad-msi-token + From 44d11b5df5b04aa7e034060d1d6e3a72845c8dea Mon Sep 17 00:00:00 2001 From: Ganga Mahesh Siddem Date: Mon, 4 Apr 2022 09:47:39 -0700 Subject: [PATCH 2/5] readme updates --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 13ca3a822..bd29e179a 100644 --- a/README.md +++ b/README.md @@ -317,15 +317,16 @@ Navigate to Kubernetes directory and update the yamls with latest docker image o For DEV and PROD branches, automatically deployed latest yaml with latest agent image (which automatically built by the azure devops pipeline) onto CIDEV and CIPROD AKS clusters in build subscription. So, you can use CIDEV and CIPROD AKS cluster to validate E2E. Similarly, you can set up build and release pipelines for your feature branch. # Testing MSI Auth Mode Using Yaml - 1. Enable Monitoring addon with Managed Idenity Auth Mode either using Portal or CLI + 1. Enable Monitoring addon with Managed Idenity Auth Mode either using Portal or CLI or Template 2. Deploy [ARM template](./scripts/onboarding/aks/onboarding-using-msi-auth/) with enabled = false to create DCR, DCR-A and link the workspace to Portal > Note - Make sure to update the parameter values in existingClusterParam.json file and have enabled = true in template file `az deployment group create --resource-group --template-file ./existingClusterOnboarding.json --parameters @./existingClusterParam.json` 3. Get the MSI token (which is valid for 24 hrs.) value via `kubectl get secrets -n kube-system omsagent-aad-msi-token -o=jsonpath='{.data.token}'` 4. Disable Monitoring addon via `az aks disable-addons -a monitoring -g -n ` - 5. Uncomment MSI auth related yaml lines, replace all the placeholder values including the MSI token value in the omsagent.yaml + 5. Uncomment MSI auth related yaml lines, replace all the placeholder values, MSI token value and image tag in the omsagent.yaml 6. Deploy the omsagent.yaml via `kubectl apply -f omsagent.yaml` - > Note: use the image toggle for release E2E validation + > Note: use the image toggle for release E2E validation + 7. valiade E2E # E2E Tests From c2dfa643870f9fed91ac5ce743a38edd1fd44e15 Mon Sep 17 00:00:00 2001 From: Ganga Mahesh Siddem Date: Mon, 4 Apr 2022 09:56:57 -0700 Subject: [PATCH 3/5] readme updates --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index bd29e179a..15486f3c1 100644 --- a/README.md +++ b/README.md @@ -317,6 +317,7 @@ Navigate to Kubernetes directory and update the yamls with latest docker image o For DEV and PROD branches, automatically deployed latest yaml with latest agent image (which automatically built by the azure devops pipeline) onto CIDEV and CIPROD AKS clusters in build subscription. So, you can use CIDEV and CIPROD AKS cluster to validate E2E. Similarly, you can set up build and release pipelines for your feature branch. # Testing MSI Auth Mode Using Yaml + 1. Enable Monitoring addon with Managed Idenity Auth Mode either using Portal or CLI or Template 2. Deploy [ARM template](./scripts/onboarding/aks/onboarding-using-msi-auth/) with enabled = false to create DCR, DCR-A and link the workspace to Portal > Note - Make sure to update the parameter values in existingClusterParam.json file and have enabled = true in template file From 5175539d25f60613bd5f42b2dc8c6f30355b687a Mon Sep 17 00:00:00 2001 From: Ganga Mahesh Siddem Date: Mon, 4 Apr 2022 10:05:38 -0700 Subject: [PATCH 4/5] readme updates --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 15486f3c1..7e2700cb7 100644 --- a/README.md +++ b/README.md @@ -320,14 +320,14 @@ For DEV and PROD branches, automatically deployed latest yaml with latest agent 1. Enable Monitoring addon with Managed Idenity Auth Mode either using Portal or CLI or Template 2. Deploy [ARM template](./scripts/onboarding/aks/onboarding-using-msi-auth/) with enabled = false to create DCR, DCR-A and link the workspace to Portal - > Note - Make sure to update the parameter values in existingClusterParam.json file and have enabled = true in template file + > Note - Make sure to update the parameter values in existingClusterParam.json file and have enabled = false in template file `az deployment group create --resource-group --template-file ./existingClusterOnboarding.json --parameters @./existingClusterParam.json` 3. Get the MSI token (which is valid for 24 hrs.) value via `kubectl get secrets -n kube-system omsagent-aad-msi-token -o=jsonpath='{.data.token}'` 4. Disable Monitoring addon via `az aks disable-addons -a monitoring -g -n ` 5. Uncomment MSI auth related yaml lines, replace all the placeholder values, MSI token value and image tag in the omsagent.yaml 6. Deploy the omsagent.yaml via `kubectl apply -f omsagent.yaml` > Note: use the image toggle for release E2E validation - 7. valiade E2E + 7. validate E2E for # E2E Tests From 92184a10602dd2298fe627da4f9f5c947d82cc22 Mon Sep 17 00:00:00 2001 From: Ganga Mahesh Siddem Date: Mon, 4 Apr 2022 10:10:23 -0700 Subject: [PATCH 5/5] readme updates --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 7e2700cb7..e7206c492 100644 --- a/README.md +++ b/README.md @@ -327,7 +327,7 @@ For DEV and PROD branches, automatically deployed latest yaml with latest agent 5. Uncomment MSI auth related yaml lines, replace all the placeholder values, MSI token value and image tag in the omsagent.yaml 6. Deploy the omsagent.yaml via `kubectl apply -f omsagent.yaml` > Note: use the image toggle for release E2E validation - 7. validate E2E for + 7. validate E2E for LA & Metrics data flows, and other scenarios # E2E Tests