Skip to content

Rashmi/fluentd docker inventory #160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rashmichandrashekar merged 45 commits into from
Nov 5, 2018
Merged

Rashmi/fluentd docker inventory #160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
45 commits
Select commit Hold shift + click to select a range
d9dbbd8
first stab
Oct 19, 2018
f7f6e3d
changes
Oct 22, 2018
5b215bf
changes
Oct 22, 2018
524f1bb
docker util changes
Oct 22, 2018
9b95f45
working tested util
Oct 23, 2018
b0467cf
input plugin and conf
Oct 23, 2018
34294c1
changes
Oct 23, 2018
7ce3d08
changes
Oct 23, 2018
24d912c
changes
Oct 24, 2018
07de7e5
changes
Oct 24, 2018
73877ff
changes
Oct 24, 2018
ca3ee04
working containerinventory
Oct 25, 2018
1492dab
fixing omi removal from container.conf
Oct 25, 2018
8d4ca54
removing comments
Oct 25, 2018
a8e3498
file write and read
Oct 25, 2018
d0fbccb
deleted containers working
Oct 25, 2018
4efc648
changes
Oct 26, 2018
44b4070
changes
Oct 26, 2018
a5deb48
socket timeout
Oct 26, 2018
cfb0a33
deleting test files
Oct 26, 2018
c2c2a0b
adding log
Oct 26, 2018
fda2cc5
fixing comment
Oct 26, 2018
de76b9e
appinsights changes
Oct 30, 2018
6818c92
changes
Oct 30, 2018
ddd2b88
tel changes
Oct 30, 2018
b15eb56
changes
Oct 30, 2018
9c279b7
changes
Oct 30, 2018
d14324f
changes
Oct 30, 2018
9b5e0e7
changes
Oct 30, 2018
17c308e
lib changes
Oct 31, 2018
6c5d6cf
changes
Oct 31, 2018
9e192ba
changes
Oct 31, 2018
d992e9d
fixes
Oct 31, 2018
17656aa
PR comments
Oct 31, 2018
339bd21
changes
Oct 31, 2018
fcfc437
updating the ownership
Oct 31, 2018
d434d98
changes
Nov 1, 2018
b049ba0
changes
Nov 1, 2018
e9111a9
changes to container data
Nov 1, 2018
0edb539
removing comment
Nov 1, 2018
abf1a67
changes
Nov 1, 2018
2c4ddff
adding collection time
Nov 1, 2018
0d3395b
bug fix
Nov 2, 2018
273af3b
env string truncation
Nov 3, 2018
3b74ca7
changes for acs-engine test
Nov 5, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 9 additions & 37 deletions installer/conf/container.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,22 +9,10 @@

# Container inventory
<source>
type omi
run_interval 60s
tag oms.container.containerinventory
items [
["root/cimv2","Container_ContainerInventory"]
]
</source>

# Image inventory
<source>
type omi
run_interval 60s
tag oms.container.imageinventory
items [
["root/cimv2","Container_ImageInventory"]
]
type containerinventory
tag oms.containerinsights.containerinventory
run_interval 60s
log_level debug
</source>

# Container host inventory
Expand All @@ -45,11 +33,6 @@
log_level debug
</source>

# Filter for correct format to endpoint
<filter oms.container.containerinventory oms.container.imageinventory>
type filter_container
</filter>

<match oms.api.ContainerNodeInventory**>
type out_oms_api
log_level debug
Expand All @@ -63,33 +46,22 @@
max_retry_wait 9m
</match>

<match oms.container.containerinventory**>
<match oms.containerinsights.containerinventory**>
type out_oms
log_level debug
num_threads 5
buffer_chunk_limit 20m
buffer_type file
buffer_path %STATE_DIR_WS%/out_oms_containerinventory*.buffer
buffer_queue_limit 20
buffer_queue_full_action drop_oldest_chunk
flush_interval 20s
retry_limit 10
retry_wait 15s
max_retry_wait 9m
</match>

<match oms.container.imageinventory**>
type out_oms
log_level debug
buffer_chunk_limit 20m
buffer_type file
buffer_path %STATE_DIR_WS%/out_oms_imageinventory*.buffer
buffer_queue_limit 20
flush_interval 20s
retry_limit 10
retry_wait 15s
retry_wait 30s
max_retry_wait 9m
</match>

<match oms.api.cadvisorperf**>
<match oms.api.cadvisorperf**>
type out_oms
log_level debug
num_threads 5
Expand Down
61 changes: 60 additions & 1 deletion installer/datafiles/base_container.data
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,57 @@ MAINTAINER: 'Microsoft Corporation'
/opt/microsoft/omsagent/plugin/in_kube_services.rb; source/code/plugin/in_kube_services.rb; 644; root; root
/opt/microsoft/omsagent/plugin/in_kube_nodes.rb; source/code/plugin/in_kube_nodes.rb; 644; root; root

/opt/microsoft/omsagent/plugin/ApplicationInsightsUtility.rb; source/code/plugin/ApplicationInsightsUtility.rb; 644; root; root
/opt/microsoft/omsagent/plugin/ContainerInventoryState.rb; source/code/plugin/ContainerInventoryState.rb; 644; root; root
/opt/microsoft/omsagent/plugin/DockerApiClient.rb; source/code/plugin/DockerApiClient.rb; 644; root; root
/opt/microsoft/omsagent/plugin/DockerApiRestHelper.rb; source/code/plugin/DockerApiRestHelper.rb; 644; root; root
/opt/microsoft/omsagent/plugin/in_containerinventory.rb; source/code/plugin/in_containerinventory.rb; 644; root; root

/opt/microsoft/omsagent/plugin/lib/application_insights/version.rb; source/code/plugin/lib/application_insights/version.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/rack/track_request.rb; source/code/plugin/lib/application_insights/rack/track_request.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/unhandled_exception.rb; source/code/plugin/lib/application_insights/unhandled_exception.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/telemetry_client.rb; source/code/plugin/lib/application_insights/telemetry_client.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/queue_base.rb; source/code/plugin/lib/application_insights/channel/queue_base.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/asynchronous_queue.rb; source/code/plugin/lib/application_insights/channel/asynchronous_queue.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/synchronous_sender.rb; source/code/plugin/lib/application_insights/channel/synchronous_sender.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/data_point_type.rb; source/code/plugin/lib/application_insights/channel/contracts/data_point_type.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/data_point.rb; source/code/plugin/lib/application_insights/channel/contracts/data_point.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/stack_frame.rb; source/code/plugin/lib/application_insights/channel/contracts/stack_frame.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/request_data.rb; source/code/plugin/lib/application_insights/channel/contracts/request_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/session.rb; source/code/plugin/lib/application_insights/channel/contracts/session.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/page_view_data.rb; source/code/plugin/lib/application_insights/channel/contracts/page_view_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/remote_dependency_data.rb; source/code/plugin/lib/application_insights/channel/contracts/remote_dependency_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/exception_data.rb; source/code/plugin/lib/application_insights/channel/contracts/exception_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/location.rb; source/code/plugin/lib/application_insights/channel/contracts/location.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/operation.rb; source/code/plugin/lib/application_insights/channel/contracts/operation.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/data.rb; source/code/plugin/lib/application_insights/channel/contracts/data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/event_data.rb; source/code/plugin/lib/application_insights/channel/contracts/event_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/metric_data.rb; source/code/plugin/lib/application_insights/channel/contracts/metric_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/device.rb; source/code/plugin/lib/application_insights/channel/contracts/device.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/message_data.rb; source/code/plugin/lib/application_insights/channel/contracts/message_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/dependency_source_type.rb; source/code/plugin/lib/application_insights/channel/contracts/dependency_source_type.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/user.rb; source/code/plugin/lib/application_insights/channel/contracts/user.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/severity_level.rb; source/code/plugin/lib/application_insights/channel/contracts/severity_level.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/application.rb; source/code/plugin/lib/application_insights/channel/contracts/application.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/dependency_kind.rb; source/code/plugin/lib/application_insights/channel/contracts/dependency_kind.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/cloud.rb; source/code/plugin/lib/application_insights/channel/contracts/cloud.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/envelope.rb; source/code/plugin/lib/application_insights/channel/contracts/envelope.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/json_serializable.rb; source/code/plugin/lib/application_insights/channel/contracts/json_serializable.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/domain.rb; source/code/plugin/lib/application_insights/channel/contracts/domain.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/base.rb; source/code/plugin/lib/application_insights/channel/contracts/base.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/reopenings.rb; source/code/plugin/lib/application_insights/channel/contracts/reopenings.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/page_view_perf_data.rb; source/code/plugin/lib/application_insights/channel/contracts/page_view_perf_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/internal.rb; source/code/plugin/lib/application_insights/channel/contracts/internal.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/availability_data.rb; source/code/plugin/lib/application_insights/channel/contracts/availability_data.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts/exception_details.rb; source/code/plugin/lib/application_insights/channel/contracts/exception_details.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/synchronous_queue.rb; source/code/plugin/lib/application_insights/channel/synchronous_queue.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/sender_base.rb; source/code/plugin/lib/application_insights/channel/sender_base.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/telemetry_context.rb; source/code/plugin/lib/application_insights/channel/telemetry_context.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/asynchronous_sender.rb; source/code/plugin/lib/application_insights/channel/asynchronous_sender.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/telemetry_channel.rb; source/code/plugin/lib/application_insights/channel/telemetry_channel.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/event.rb; source/code/plugin/lib/application_insights/channel/event.rb; 644; root; root
/opt/microsoft/omsagent/plugin/lib/application_insights.rb; source/code/plugin/lib/application_insights.rb; 644; root; root

/opt/td-agent-bit/bin/out_oms.so; intermediate/${{BUILD_CONFIGURATION}}/out_oms.so; 755; root; root
/etc/opt/microsoft/docker-cimprov/td-agent-bit.conf; installer/conf/td-agent-bit.conf; 644; root; root
/etc/opt/microsoft/docker-cimprov/out_oms.conf; installer/conf/out_oms.conf; 644; root; root
Expand Down Expand Up @@ -75,12 +126,17 @@ MAINTAINER: 'Microsoft Corporation'
/var/opt/microsoft/docker-cimprov; 755; root; root
/var/opt/microsoft/docker-cimprov/state; 755; root; root
/var/opt/microsoft/docker-cimprov/state/ContainerInventory; 755; root; root
/var/opt/microsoft/docker-cimprov/state/ImageInventory; 755; root; root
/var/opt/microsoft/docker-cimprov/log; 755; root; root

/opt/td-agent-bit; 755; root; root;sysdir
/opt/td-agent-bit/bin; 755; root; root;sysdir

/opt/microsoft/omsagent/plugin/lib; 755; root; root; sysdir
/opt/microsoft/omsagent/plugin/lib/application_insights; 755; root; root; sysdir
/opt/microsoft/omsagent/plugin/lib/application_insights/channel; 755; root; root; sysdir
/opt/microsoft/omsagent/plugin/lib/application_insights/channel/contracts; 755; root; root; sysdir
/opt/microsoft/omsagent/plugin/lib/application_insights/rack; 755; root; root; sysdir

%Dependencies

%Postinstall_10
Expand All @@ -90,6 +146,9 @@ WriteInstallInfo() {
}
WriteInstallInfo

#Make omsagent owner for ContainerInventory directory. This is needed for ruby plugin to have access
chown omsagent:omsagent /var/opt/microsoft/docker-cimprov/state/ContainerInventory

# Get the state file in place with proper permissions
touch /var/opt/microsoft/docker-cimprov/state/LastEventQueryTime.txt
chmod 644 /var/opt/microsoft/docker-cimprov/state/LastEventQueryTime.txt
Expand Down
142 changes: 142 additions & 0 deletions source/code/plugin/ApplicationInsightsUtility.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
#!/usr/local/bin/ruby
# frozen_string_literal: true

class ApplicationInsightsUtility
require_relative 'lib/application_insights'
require_relative 'omslog'
require_relative 'DockerApiClient'
require 'json'
require 'base64'

@@HeartBeat = 'HeartBeatEvent'
@@Exception = 'ExceptionEvent'
@@AcsClusterType = 'ACS'
@@AksClusterType = 'AKS'
@@DaemonsetControllerType = 'DaemonSet'
@OmsAdminFilePath = '/etc/opt/microsoft/omsagent/conf/omsadmin.conf'
@@EnvAcsResourceName = 'ACS_RESOURCE_NAME'
@@EnvAksRegion = 'AKS_REGION'
@@EnvAgentVersion = 'AGENT_VERSION'
@@EnvApplicationInsightsKey = 'APPLICATIONINSIGHTS_AUTH'
@@CustomProperties = {}
@@Tc = nil

def initialize
end

class << self
#Set default properties for telemetry event
def initializeUtility()
begin
resourceInfo = ENV['AKS_RESOURCE_ID']
if resourceInfo.nil? || resourceInfo.empty?
@@CustomProperties["ACSResourceName"] = ENV[@@EnvAcsResourceName]
@@CustomProperties["ClusterType"] = @@AcsClusterType
@@CustomProperties["SubscriptionID"] = ""
@@CustomProperties["ResourceGroupName"] = ""
@@CustomProperties["ClusterName"] = ""
@@CustomProperties["Region"] = ""
else
@@CustomProperties["AKS_RESOURCE_ID"] = resourceInfo
begin
splitStrings = resourceInfo.split('/')
subscriptionId = splitStrings[2]
resourceGroupName = splitStrings[4]
clusterName = splitStrings[8]
rescue => errorStr
$log.warn("Exception in AppInsightsUtility: parsing AKS resourceId: #{resourceInfo}, error: #{errorStr}")
end
@@CustomProperties["ClusterType"] = @@AksClusterType
@@CustomProperties["SubscriptionID"] = subscriptionId
@@CustomProperties["ResourceGroupName"] = resourceGroupName
@@CustomProperties["ClusterName"] = clusterName
@@CustomProperties["Region"] = ENV[@@EnvAksRegion]
end
@@CustomProperties['ControllerType'] = @@DaemonsetControllerType
dockerInfo = DockerApiClient.dockerInfo
@@CustomProperties['DockerVersion'] = dockerInfo['Version']
@@CustomProperties['DockerApiVersion'] = dockerInfo['ApiVersion']
@@CustomProperties['WorkspaceID'] = getWorkspaceId
@@CustomProperties['AgentVersion'] = ENV[@@EnvAgentVersion]
encodedAppInsightsKey = ENV[@@EnvApplicationInsightsKey]
if !encodedAppInsightsKey.nil?
decodedAppInsightsKey = Base64.decode64(encodedAppInsightsKey)
@@Tc = ApplicationInsights::TelemetryClient.new decodedAppInsightsKey
end
rescue => errorStr
$log.warn("Exception in AppInsightsUtility: initilizeUtility - error: #{errorStr}")
end
end

def sendHeartBeatEvent(pluginName)
begin
eventName = pluginName + @@HeartBeat
if !(@@Tc.nil?)
@@Tc.track_event eventName , :properties => @@CustomProperties
@@Tc.flush
$log.info("AppInsights Heartbeat Telemetry sent successfully")
end
rescue =>errorStr
$log.warn("Exception in AppInsightsUtility: sendHeartBeatEvent - error: #{errorStr}")
end
end

def sendCustomEvent(pluginName, properties)
begin
if !(@@Tc.nil?)
@@Tc.track_metric 'LastProcessedContainerInventoryCount', properties['ContainerCount'],
:kind => ApplicationInsights::Channel::Contracts::DataPointType::MEASUREMENT,
:properties => @@CustomProperties
@@Tc.flush
$log.info("AppInsights Container Count Telemetry sent successfully")
end
rescue => errorStr
$log.warn("Exception in AppInsightsUtility: sendCustomEvent - error: #{errorStr}")
end
end

def sendExceptionTelemetry(errorStr)
begin
if @@CustomProperties.empty? || @@CustomProperties.nil?
initializeUtility
end
if !(@@Tc.nil?)
@@Tc.track_exception errorStr , :properties => @@CustomProperties
@@Tc.flush
$log.info("AppInsights Exception Telemetry sent successfully")
end
rescue => errorStr
$log.warn("Exception in AppInsightsUtility: sendExceptionTelemetry - error: #{errorStr}")
end
end

#Method to send heartbeat and container inventory count
def sendTelemetry(pluginName, properties)
begin
if @@CustomProperties.empty? || @@CustomProperties.nil?
initializeUtility
end
@@CustomProperties['Computer'] = properties['Computer']
sendHeartBeatEvent(pluginName)
sendCustomEvent(pluginName, properties)
rescue => errorStr
$log.warn("Exception in AppInsightsUtility: sendTelemetry - error: #{errorStr}")
end
end

def getWorkspaceId()
begin
adminConf = {}
confFile = File.open(@OmsAdminFilePath, "r")
confFile.each_line do |line|
splitStrings = line.split('=')
adminConf[splitStrings[0]] = splitStrings[1]
end
workspaceId = adminConf['WORKSPACE_ID']
return workspaceId
rescue => errorStr
$log.warn("Exception in AppInsightsUtility: getWorkspaceId - error: #{errorStr}")
end
end
end
end
Loading