Skip to content

Gangams/msi test instructions #739

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ganga1980 merged 5 commits into from
Apr 4, 2022
Merged

Gangams/msi test instructions #739

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
006b217
instructions for msi test validation
ganga1980 Apr 3, 2022
44d11b5
readme updates
ganga1980 Apr 4, 2022
c2dfa64
readme updates
ganga1980 Apr 4, 2022
5175539
readme updates
ganga1980 Apr 4, 2022
92184a1
readme updates
ganga1980 Apr 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 22 additions & 9 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,7 @@ powershell -ExecutionPolicy bypass # switch to powershell if you are not on pow
```

##### Developer Build optimizations
If you do not want to build the image from scratch every time you make changes during development,you can choose to build the docker images that are separated out by
If you do not want to build the image from scratch every time you make changes during development,you can choose to build the docker images that are separated out by
* Base image and dependencies including agent bootstrap(setup.ps1)
* Agent conf and plugin changes

Expand All @@ -248,7 +248,7 @@ And then run the script to build the image consisting of code and conf changes.
.\build-and-publish-dev-docker-image.ps1 -image <repo>/<imagename>:<imagetag> # trigger build code and image and publish docker hub or acr
```

For the subsequent builds, you can just run -
For the subsequent builds, you can just run -

```
.\build-and-publish-dev-docker-image.ps1 -image <repo>/<imagename>:<imagetag> # trigger build code and image and publish docker hub or acr
Expand Down Expand Up @@ -316,18 +316,31 @@ Navigate to Kubernetes directory and update the yamls with latest docker image o

For DEV and PROD branches, automatically deployed latest yaml with latest agent image (which automatically built by the azure devops pipeline) onto CIDEV and CIPROD AKS clusters in build subscription. So, you can use CIDEV and CIPROD AKS cluster to validate E2E. Similarly, you can set up build and release pipelines for your feature branch.

# Testing MSI Auth Mode Using Yaml

1. Enable Monitoring addon with Managed Idenity Auth Mode either using Portal or CLI or Template
2. Deploy [ARM template](./scripts/onboarding/aks/onboarding-using-msi-auth/) with enabled = false to create DCR, DCR-A and link the workspace to Portal
> Note - Make sure to update the parameter values in existingClusterParam.json file and have enabled = false in template file
`az deployment group create --resource-group <ResourceGroupName> --template-file ./existingClusterOnboarding.json --parameters @./existingClusterParam.json`
3. Get the MSI token (which is valid for 24 hrs.) value via `kubectl get secrets -n kube-system omsagent-aad-msi-token -o=jsonpath='{.data.token}'`
4. Disable Monitoring addon via `az aks disable-addons -a monitoring -g <rgName> -n <clusterName>`
5. Uncomment MSI auth related yaml lines, replace all the placeholder values, MSI token value and image tag in the omsagent.yaml
6. Deploy the omsagent.yaml via `kubectl apply -f omsagent.yaml`
> Note: use the image toggle for release E2E validation
7. validate E2E for LA & Metrics data flows, and other scenarios

# E2E Tests

## For executing tests

1. Deploy the omsagent.yaml with your agent image. In the yaml, make sure `ISTEST` environment variable set to `true` if its not set already
2. Update the Service Principal CLIENT_ID, CLIENT_SECRET and TENANT_ID placeholder values and apply e2e-tests.yaml to execute the tests
2. Update the Service Principal CLIENT_ID, CLIENT_SECRET and TENANT_ID placeholder values and apply e2e-tests.yaml to execute the tests
> Note: Service Principal requires reader role on log analytics workspace and cluster resource to query LA and metrics
```
cd ~/Docker-Provider/test/e2e # based on your repo path
kubectl apply -f e2e-tests.yaml # this will trigger job to run the tests in sonobuoy namespace
kubectl get po -n sonobuoy # to check the pods and jobs associated to tests
```
cd ~/Docker-Provider/test/e2e # based on your repo path
kubectl apply -f e2e-tests.yaml # this will trigger job to run the tests in sonobuoy namespace
kubectl get po -n sonobuoy # to check the pods and jobs associated to tests
```
3. Download (sonobuoy)[https://github.com/vmware-tanzu/sonobuoy/releases] on your dev box to view the results of the tests
```
results=$(sonobuoy retrieve) # downloads tar file which has logs and test results
Expand All @@ -338,9 +351,9 @@ For DEV and PROD branches, automatically deployed latest yaml with latest agent
## For adding new tests

1. Add the test python file with your test code under `tests` directory
2. Build the docker image, recommended to use ACR & MCR
2. Build the docker image, recommended to use ACR & MCR
```
cd ~/Docker-Provider/test/e2e/src # based on your repo path
cd ~/Docker-Provider/test/e2e/src # based on your repo path
docker login <acr> -u <user> -p <pwd> # login to acr
docker build -f ./core/Dockerfile -t <repo>/<imagename>:<imagetag> .
docker push <repo>/<imagename>:<imagetag>
Expand Down
112 changes: 106 additions & 6 deletions kubernetes/omsagent.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ rules:
- apiGroups: ["apps", "extensions", "autoscaling"]
resources: ["replicasets", "deployments", "horizontalpodautoscalers"]
verbs: ["list"]
# Uncomment below lines for MSI Auth Mode testing
# - apiGroups: [""]
# resources: ["secrets"]
# resourceNames: [ "omsagent-aad-msi-token" ]
# verbs: ["get", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
Expand Down Expand Up @@ -297,6 +302,16 @@ data:
WSID: "VALUE_WSID"
KEY: "VALUE_KEY"
---
# Uncomment below lines for MSI Auth Mode testing
# apiVersion: v1
# kind: Secret
# metadata:
# name: omsagent-aad-msi-token
# namespace: kube-system
# type: Opaque
# data:
# token: "VALUE_MSI_TOKEN"
# ---
apiVersion: apps/v1
kind: DaemonSet
metadata:
Expand Down Expand Up @@ -328,6 +343,41 @@ spec:
- name: ndots
value: "3"
containers:
# Uncomment below lines for MSI Auth Mode testing
# - name: addon-token-adapter
# command:
# - /addon-token-adapter
# args:
# - --secret-namespace=kube-system
# - --secret-name=omsagent-aad-msi-token
# - --token-server-listening-port=8888
# - --health-server-listening-port=9999
# # Make sure this matching with version in AKS RP side
# image: mcr.microsoft.com/aks/msi/addon-token-adapter:master.220318.3
# imagePullPolicy: IfNotPresent
# env:
# - name: AZMON_COLLECT_ENV
# value: "false"
# livenessProbe:
# httpGet:
# path: /healthz
# port: 9999
# initialDelaySeconds: 10
# periodSeconds: 60
# resources:
# limits:
# cpu: 500m
# memory: 500Mi
# requests:
# cpu: 100m
# memory: 100Mi
# securityContext:
# capabilities:
# drop:
# - ALL
# add:
# - NET_ADMIN
# - NET_RAW
- name: omsagent
image: "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03172022"
imagePullPolicy: IfNotPresent
Expand Down Expand Up @@ -367,8 +417,9 @@ spec:
value: "VALUE_USER_ASSIGNED_IDENTITY_CLIENT_ID_VALUE"
- name: AZMON_CONTAINERLOGS_ONEAGENT_REGIONS
value: "koreacentral,norwayeast,eastus2"
- name: USING_AAD_MSI_AUTH
value: "false"
# Uncomment below lines for MSI Auth Mode testing
# - name: USING_AAD_MSI_AUTH
# value: "true"
securityContext:
privileged: true
ports:
Expand Down Expand Up @@ -448,8 +499,9 @@ spec:
# Update this with the user assigned msi client id for omsagent
- name: USER_ASSIGNED_IDENTITY_CLIENT_ID
value: "VALUE_USER_ASSIGNED_IDENTITY_CLIENT_ID_VALUE"
- name: USING_AAD_MSI_AUTH
value: "false"
# Uncomment below lines for MSI Auth Mode testing
# - name: USING_AAD_MSI_AUTH
# value: "true"
securityContext:
privileged: true
volumeMounts:
Expand Down Expand Up @@ -563,6 +615,41 @@ spec:
spec:
serviceAccountName: omsagent
containers:
# Uncomment below lines for MSI Auth Mode testing
# - name: addon-token-adapter
# command:
# - /addon-token-adapter
# args:
# - --secret-namespace=kube-system
# - --secret-name=omsagent-aad-msi-token
# - --token-server-listening-port=8888
# - --health-server-listening-port=9999
# # Make sure this matching with version in AKS RP side
# image: mcr.microsoft.com/aks/msi/addon-token-adapter:master.220318.3
# imagePullPolicy: IfNotPresent
# env:
# - name: AZMON_COLLECT_ENV
# value: "false"
# livenessProbe:
# httpGet:
# path: /healthz
# port: 9999
# initialDelaySeconds: 10
# periodSeconds: 60
# resources:
# limits:
# cpu: 500m
# memory: 500Mi
# requests:
# cpu: 100m
# memory: 100Mi
# securityContext:
# capabilities:
# drop:
# - ALL
# add:
# - NET_ADMIN
# - NET_RAW
- name: omsagent
image: "mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod03172022"
imagePullPolicy: IfNotPresent
Expand Down Expand Up @@ -596,8 +683,9 @@ spec:
# Add the below environment variable to true only in sidecar enabled regions, else set it to false
- name: SIDECAR_SCRAPING_ENABLED
value: "true"
- name: USING_AAD_MSI_AUTH
value: "false"
# Uncomment below lines for MSI Auth Mode testing
# - name: USING_AAD_MSI_AUTH
# value: "true"
securityContext:
privileged: true
ports:
Expand Down Expand Up @@ -776,6 +864,9 @@ spec:
# Add this only for clouds that require cert bootstrapping
# - name: REQUIRES_CERT_BOOTSTRAP
# value: "true"
# Uncomment below lines for MSI Auth Mode testing
# - name: USING_AAD_MSI_AUTH
# value: "true"
volumeMounts:
- mountPath: C:\ProgramData\docker\containers
name: docker-windows-containers
Expand All @@ -798,6 +889,10 @@ spec:
- mountPath: C:\etc\kubernetes\host
name: azure-json-path
readOnly: true
# Uncomment below lines for MSI Auth Mode testing
# - mountPath: C:\etc\IMDS-access-token
# name: imds-token
# readOnly: true
livenessProbe:
exec:
command:
Expand Down Expand Up @@ -855,3 +950,8 @@ spec:
secret:
secretName: omsagent-adx-secret
optional: true
# Uncomment below lines for MSI Auth Mode testing
# - name: imds-token
# secret:
# secretName: omsagent-aad-msi-token