Gangams/release ciprod06112021

Documentation/OSMPrivatePreview/ReadMe.md

@@ -1,15 +1,17 @@
 Note - This is private preview. For any support issues, please reach out to us at [askcoin@microsoft.com](mailto:askcoin@microsoft.com). Please don't open a support ticket.
 
+This private preview supports Open Service Mesh on [AKS](https://docs.microsoft.com/azure/aks/servicemesh-osm-about) & Azure [Arc on k8s](http://docs.microsoft.com/azure/azure-arc/kubernetes/tutorial-arc-enabled-osm).
+
 # Azure Monitor Container Insights Open Service Mesh Monitoring
 
 Azure Monitor container insights now supporting preview of [Open Service Mesh(OSM)](https://docs.microsoft.com/azure/aks/servicemesh-osm-about) Monitoring. As part of this support, customer can:
 1.	Filter & view inventory of all the services that are part of your service mesh.
 2.	Visualize and monitor requests between services in your service mesh, with request latency, error rate & resource utilization by services.
-3.	Provides connection summary for OSM infrastructure running on AKS.
+3.	Provides connection summary for OSM infrastructure running on AKS or Azure Arc for k8s.
 
 ## How to onboard Container Insights OSM monitoring?
 OSM exposes Prometheus metrics which Container Insights can collect, for container insights agent to collect OSM metrics follow the following steps.
-
+### AKS
 1.  Follow this [link](https://docs.microsoft.com/en-us/azure/aks/servicemesh-osm-about?pivots=client-operating-system-linux#register-the-aks-openservicemesh-preview-feature) as a prereq before enabling the addon. 
 
 2.  Enable AKS OSM addon on your 
@@ -27,9 +29,29 @@ osm metrics enable --namespace "test1, test2"
      * Download the configmap from [here](https://github.com/microsoft/Docker-Provider/blob/ci_prod/kubernetes/container-azm-ms-osmconfig.yaml)
      * Add the namespaces you want to monitor in configmap `monitor_namespaces = ["namespace1", "namespace2"]`
      * Run the following kubectl command: kubectl apply -f<configmap_yaml_file.yaml>
-         * Example: `kubectl apply -f container-azm-ms-agentconfig.yaml`
+         * Example: `kubectl apply -f container-azm-ms-osmconfig.yaml`
 4. The configuration change can take upto 15 mins to finish before taking effect, and all omsagent pods in the cluster will restart. The restart is a rolling restart for all omsagent pods, not all restart at the same time.
 
+### Azure Arc for Kuberentes
+This section assumes that you already have your kubernetes distribution connected via Azure Arc. If not learn more [here.](https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster)
+
+1. Install Arc enabled Open Service mesh on your Arc cluster. Learn more [here](http://docs.microsoft.com/azure/azure-arc/kubernetes/tutorial-arc-enabled-osm#install-arc-enabled-open-service-mesh-osm-on-an-arc-enabled-kubernetes-cluster)
+2. Install Azure Monitor Container Insights on Arc. If not installed already. Learn more how to install [here](https://docs.microsoft.com/azure/azure-monitor/containers/container-insights-enable-arc-enabled-clusters)
+3. Ensure that prometheus_scraping is set to true in the OSM configmap.
+3. Ensure that the application namespaces that you wish to be monitored are onboarded to the mesh. Follow the guidance available [here.](http://docs.microsoft.com/azure/azure-arc/kubernetes/tutorial-arc-enabled-osm#onboard-namespaces-to-the-service-mesh)
+4. To enable namespace(s), download the osm client library [here](https://docs.microsoft.com/en-us/azure/aks/servicemesh-osm-about?pivots=client-operating-system-linux#osm-service-quotas-and-limits-preview) & then enable metrics on namespaces
+```bash
+# With osm
+osm metrics enable --namespace test
+osm metrics enable --namespace "test1, test2"
+
+```
+4. On your Azure Monitor Container Insights for Arc.
+ * Download the configmap from [here](https://github.com/microsoft/Docker-Provider/blob/ci_prod/kubernetes/container-azm-ms-osmconfig.yaml)
+     * Add the namespaces you want to monitor in configmap `monitor_namespaces = ["namespace1", "namespace2"]`
+     * Run the following kubectl command: kubectl apply -f<configmap_yaml_file.yaml>
+         * Example: `kubectl apply -f container-azm-ms-osmconfig.yaml`
+5. The configuration change can take upto 15 mins to finish before taking effect, and all omsagent pods in the cluster will restart. The restart is a rolling restart for all omsagent pods, not all restart at the same time.
 
 ## Validate the metrics flow
 1.	Query cluster's Log Analytics workspace InsightsMetrics table to see metrics are flowing or not
@@ -41,8 +63,9 @@ InsightsMetrics
 
 ## How to consume OSM monitoring dashboard?
 1.	Access your AKS cluster & Container Insights through this [link.](https://aka.ms/azmon/osmux)
-2.	Go to reports tab and access Open Service Mesh (OSM) workbook.
-3.	Select the time-range & namespace to scope your services. By default, we only show services deployed by customers and we exclude internal service communication. In case you want to view that you select Show All in the filter. Please note OSM is managed service mesh, we show all internal connections for transparency. 
+     *  For **Azure Arc for k8s**, access Container Insights through this [link.](https://aka.ms/azmon/osmarcux)
+3.	Go to reports tab and access Open Service Mesh (OSM) workbook.
+4.	Select the time-range & namespace to scope your services. By default, we only show services deployed by customers and we exclude internal service communication. In case you want to view that you select Show All in the filter. Please note OSM is managed service mesh, we show all internal connections for transparency. 
 
 ![alt text](https://github.com/microsoft/Docker-Provider/blob/saarorOSMdoc/Documentation/OSMPrivatePreview/Image1.jpg)
 ### Requests Tab
@@ -51,6 +74,8 @@ InsightsMetrics
 3.	You can view total requests, request error rate & P90 latency.
 4.	You can drill-down to destination and view trends for HTTP error/success code, success rate, Pods resource utilization, latencies at different percentiles.
 
+![image](https://user-images.githubusercontent.com/31900410/119195241-2e712000-ba39-11eb-8cb0-2d7d16e26d1b.png)
+
 ### Connections Tab
 1.	This tab provides you a summary of all the connections between your services in Open Service Mesh. 
 2.	Outbound connections: Total number of connections between Source and destination services.
@@ -68,4 +93,6 @@ InsightsMetrics
 2.	When source or destination is osmcontroller we show no latency & for internal services we show no resource utilization. 
 3.  When both prometheus scraping using pod annotations and OSM monitoring are enabled on the same set of namespaces, the default set of metrics (envoy_cluster_upstream_cx_total, envoy_cluster_upstream_cx_connect_fail, envoy_cluster_upstream_rq, envoy_cluster_upstream_rq_xx, envoy_cluster_upstream_rq_total, envoy_cluster_upstream_rq_time_bucket, envoy_cluster_upstream_cx_rx_bytes_total, envoy_cluster_upstream_cx_tx_bytes_total, envoy_cluster_upstream_cx_active) will be collected twice. You can follow [this](https://docs.microsoft.com/en-us/azure/azure-monitor/containers/container-insights-prometheus-integration#prometheus-scraping-settings) documentation to exclude these namespaces from pod annotation scraping using the setting monitor_kubernetes_pods_namespaces to work around this issue.
 
+4.  For monitoring on **Azure Arc on k8s** currently there is a separate link to access OSM workbook. We plan to have one single link to access workbook on both platforms by 10th June 2021.
+
 This is private preview, the goal for us is to get feedback. Please feel free to reach out to us at [askcoin@microsoft.com](mailto:askcoin@microsoft.com) for any feedback and questions!

ReleaseNotes.md

@@ -11,6 +11,24 @@ additional questions or comments.
 
 Note : The agent version(s) below has dates (ciprod<mmddyyyy>), which indicate the agent build dates (not release dates)
 
+### 06/11/2021 -
+##### Version microsoft/oms:ciprod06112021 Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod06112021 (linux)
+##### Version microsoft/oms:win-ciprod06112021 Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:win-ciprod06112021 (windows)
+ - Linux Agent
+   - Removal of base omsagent dependency 
+   - Using MDSD version 1.10.1  as base agent for all the supported LA data types
+   - Ruby version upgrade to 2.6 i.e. same version as windows agent
+   - Upgrade FluentD gem version to 1.12.2 
+   - All the Ruby Fluentd Plugins upgraded to v1 as per Fluentd guidance
+   - Fluent-bit tail plugin Mem_Buf_limit is configurable via ConfigMap
+ - Windows Agent
+   - CA cert changes for airgapped clouds
+   - Send perf metrics to MDM from windows daemonset
+   - FluentD gem version upgrade from 1.10.2 to 1.12.2 to make same version as Linux Agent
+  - Doc updates 
+   - README updates related to OSM preview release for Arc K8s
+   - README updates related to recommended alerts
+
 ### 05/20/2021 -
 ##### Version microsoft/oms:ciprod05202021 Version mcr.microsoft.com/azuremonitor/containerinsights/ciprod:ciprod05202021 (linux)
 ##### No Windows changes with this release, win-ciprod04222021 still current.

alerts/recommended_alerts_ARM/README.md

@@ -24,7 +24,7 @@ Completed job count|Calculates number of jobs completed more than six hours ago.
 
 ### How to enable with a Resource Manager template
 1. Download one or all of the available templates that describe how to create the alert.
-2. Create and use a [parameters file](https://review.docs.microsoft.com/azure/azure-resource-manager/templates/parameter-files) as a JSON to set the values required to create the alert rule.
+2. Create and use a [parameters file](https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/parameter-files) as a JSON to set the values required to create the alert rule.
 3. Deploy the template from the Azure portal, PowerShell, or Azure CLI.
 
 For step by step procedures on how to enable alerts via Resource manager, please go [here.](https://aka.ms/ci_alerts_arm)

build/common/installer/scripts/td-agent-bit-conf-customizer.rb

@@ -6,6 +6,7 @@
 @default_service_interval = "1"
 @default_buffer_chunk_size = "1"
 @default_buffer_max_size = "1"
+@default_mem_buf_limit = "10"
 
 def is_number?(value)
   true if Integer(value) rescue false
@@ -19,6 +20,7 @@ def substituteFluentBitPlaceHolders
     interval = ENV["FBIT_SERVICE_FLUSH_INTERVAL"]
     bufferChunkSize = ENV["FBIT_TAIL_BUFFER_CHUNK_SIZE"]
     bufferMaxSize = ENV["FBIT_TAIL_BUFFER_MAX_SIZE"]
+    memBufLimit = ENV["FBIT_TAIL_MEM_BUF_LIMIT"]
 
     serviceInterval = (!interval.nil? && is_number?(interval) && interval.to_i > 0 ) ? interval : @default_service_interval
     serviceIntervalSetting = "Flush         " + serviceInterval
@@ -32,8 +34,12 @@ def substituteFluentBitPlaceHolders
       tailBufferMaxSize = tailBufferChunkSize
     end
 
+    tailMemBufLimit = (!memBufLimit.nil? && is_number?(memBufLimit) && memBufLimit.to_i > 10) ? memBufLimit : @default_mem_buf_limit
+    tailMemBufLimitSetting = "Mem_Buf_Limit " + tailMemBufLimit + "m"
+
     text = File.read(@td_agent_bit_conf_path)
     new_contents = text.gsub("${SERVICE_FLUSH_INTERVAL}", serviceIntervalSetting)
+    new_contents = new_contents.gsub("${TAIL_MEM_BUF_LIMIT}", tailMemBufLimitSetting)
     if !tailBufferChunkSize.nil?
       new_contents = new_contents.gsub("${TAIL_BUFFER_CHUNK_SIZE}", "Buffer_Chunk_Size " + tailBufferChunkSize + "m")
     else

build/linux/installer/scripts/tomlparser-mdm-metrics-config.rb → build/common/installer/scripts/tomlparser-mdm-metrics-config.rb

@@ -1,9 +1,16 @@
 #!/usr/local/bin/ruby
 # frozen_string_literal: true
 
-require_relative "tomlrb"
+#this should be require relative in Linux and require in windows, since it is a gem install on windows
+@os_type = ENV["OS_TYPE"]
+if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0
+  require "tomlrb"
+else
+  require_relative "tomlrb"
+end
+
+require_relative "/etc/fluent/plugin/constants"
 require_relative "ConfigParseErrorLogger"
-require_relative "microsoft/omsagent/plugin/constants"
 
 @configMapMountPath = "/etc/config/settings/alertable-metrics-configuration-settings"
 @configVersion = ""
@@ -124,6 +131,10 @@ def populateSettingValuesFromConfigMap(parsedConfig)
   end
 end
 
+def get_command_windows(env_variable_name, env_variable_value)
+  return "[System.Environment]::SetEnvironmentVariable(\"#{env_variable_name}\", \"#{env_variable_value}\", \"Process\")" + "\n" + "[System.Environment]::SetEnvironmentVariable(\"#{env_variable_name}\", \"#{env_variable_value}\", \"Machine\")" + "\n"
+end
+
 @configSchemaVersion = ENV["AZMON_AGENT_CFG_SCHEMA_VERSION"]
 puts "****************Start MDM Metrics Config Processing********************"
 if !@configSchemaVersion.nil? && !@configSchemaVersion.empty? && @configSchemaVersion.strip.casecmp("v1") == 0 #note v1 is the only supported schema version, so hardcoding it
@@ -137,19 +148,37 @@ def populateSettingValuesFromConfigMap(parsedConfig)
   end
 end
 
-# Write the settings to file, so that they can be set as environment variables
-file = File.open("config_mdm_metrics_env_var", "w")
+if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0
+  # Write the settings to file, so that they can be set as environment variables in windows container
+  file = File.open("setmdmenv.ps1", "w")
 
-if !file.nil?
-  file.write("export AZMON_ALERT_CONTAINER_CPU_THRESHOLD=#{@percentageCpuUsageThreshold}\n")
-  file.write("export AZMON_ALERT_CONTAINER_MEMORY_RSS_THRESHOLD=#{@percentageMemoryRssThreshold}\n")
-  file.write("export AZMON_ALERT_CONTAINER_MEMORY_WORKING_SET_THRESHOLD=\"#{@percentageMemoryWorkingSetThreshold}\"\n")
-  file.write("export AZMON_ALERT_PV_USAGE_THRESHOLD=#{@percentagePVUsageThreshold}\n")
-  file.write("export AZMON_ALERT_JOB_COMPLETION_TIME_THRESHOLD=#{@jobCompletionThresholdMinutes}\n")
-  # Close file after writing all MDM setting environment variables
-  file.close
-  puts "****************End MDM Metrics Config Processing********************"
+  if !file.nil?
+    commands = get_command_windows("AZMON_ALERT_CONTAINER_CPU_THRESHOLD", @percentageCpuUsageThreshold)
+    file.write(commands)
+    commands = get_command_windows("AZMON_ALERT_CONTAINER_MEMORY_WORKING_SET_THRESHOLD", @percentageMemoryWorkingSetThreshold)
+    file.write(commands)
+    # Close file after writing all environment variables
+    file.close
+    puts "****************End MDM Metrics Config Processing********************"
+  else
+    puts "Exception while opening file for writing MDM metric config environment variables"
+    puts "****************End MDM Metrics Config Processing********************"
+  end
 else
-  puts "Exception while opening file for writing MDM metric config environment variables"
-  puts "****************End MDM Metrics Config Processing********************"
+  # Write the settings to file, so that they can be set as environment variables in linux container
+  file = File.open("config_mdm_metrics_env_var", "w")
+
+  if !file.nil?
+    file.write("export AZMON_ALERT_CONTAINER_CPU_THRESHOLD=#{@percentageCpuUsageThreshold}\n")
+    file.write("export AZMON_ALERT_CONTAINER_MEMORY_RSS_THRESHOLD=#{@percentageMemoryRssThreshold}\n")
+    file.write("export AZMON_ALERT_CONTAINER_MEMORY_WORKING_SET_THRESHOLD=\"#{@percentageMemoryWorkingSetThreshold}\"\n")
+    file.write("export AZMON_ALERT_PV_USAGE_THRESHOLD=#{@percentagePVUsageThreshold}\n")
+    file.write("export AZMON_ALERT_JOB_COMPLETION_TIME_THRESHOLD=#{@jobCompletionThresholdMinutes}\n")
+    # Close file after writing all MDM setting environment variables
+    file.close
+    puts "****************End MDM Metrics Config Processing********************"
+  else
+    puts "Exception while opening file for writing MDM metric config environment variables"
+    puts "****************End MDM Metrics Config Processing********************"
+  end
 end

build/common/installer/scripts/tomlparser.rb

@@ -25,8 +25,10 @@
 @enrichContainerLogs = false
 @containerLogSchemaVersion = ""
 @collectAllKubeEvents = false
-@containerLogsRoute = ""
-
+@containerLogsRoute = "v2" # default for linux
+if !@os_type.nil? && !@os_type.empty? && @os_type.strip.casecmp("windows") == 0
+  @containerLogsRoute = "v1" # default is v1 for windows until windows agent integrates windows ama
+end
 # Use parser to parse the configmap toml file to a ruby structure
 def parseConfigMap
   begin
@@ -162,8 +164,12 @@ def populateSettingValuesFromConfigMap(parsedConfig)
     #Get container logs route setting
     begin
       if !parsedConfig[:log_collection_settings][:route_container_logs].nil? && !parsedConfig[:log_collection_settings][:route_container_logs][:version].nil?
-        @containerLogsRoute = parsedConfig[:log_collection_settings][:route_container_logs][:version]
-        puts "config::Using config map setting for container logs route"
+        if !parsedConfig[:log_collection_settings][:route_container_logs][:version].empty?
+           @containerLogsRoute = parsedConfig[:log_collection_settings][:route_container_logs][:version]
+           puts "config::Using config map setting for container logs route: #{@containerLogsRoute}"
+        else 
+           puts "config::Ignoring config map settings and using default value since provided container logs route value is empty"    
+        end         
       end
     rescue => errorStr
       ConfigParseErrorLogger.logError("Exception while reading config map settings for container logs route - #{errorStr}, using defaults, please check config map for errors")
@@ -256,7 +262,7 @@ def get_command_windows(env_variable_name, env_variable_value)
     file.write(commands)
     commands = get_command_windows('AZMON_CLUSTER_COLLECT_ALL_KUBE_EVENTS', @collectAllKubeEvents)
     file.write(commands)
-    commands = get_command_windows('AZMON_CONTAINER_LOGS_EFFECTIVE_ROUTE', @containerLogsRoute)
+    commands = get_command_windows('AZMON_CONTAINER_LOGS_ROUTE', @containerLogsRoute)
     file.write(commands)
     commands = get_command_windows('AZMON_CONTAINER_LOG_SCHEMA_VERSION', @containerLogSchemaVersion)
     file.write(commands)