Skip to content

Container log v2 schema changes #499

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vishiy merged 2 commits into from
Feb 4, 2021
Merged

Container log v2 schema changes #499

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0b5d533
vishiy Jan 23, 2021
1387f19
make pod name in mdsd definition as str for consistency. msgp has no …
vishiy Feb 2, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions build/common/installer/scripts/tomlparser.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
@logExclusionRegexPattern = "(^((?!stdout|stderr).)*$)"
@excludePath = "*.csv2" #some invalid path
@enrichContainerLogs = false
@containerLogSchemaVersion = ""
@collectAllKubeEvents = false
@containerLogsRoute = ""

Expand Down Expand Up @@ -138,6 +139,16 @@ def populateSettingValuesFromConfigMap(parsedConfig)
ConfigParseErrorLogger.logError("Exception while reading config map settings for cluster level container log enrichment - #{errorStr}, using defaults, please check config map for errors")
end

#Get container log schema version setting
begin
if !parsedConfig[:log_collection_settings][:schema].nil? && !parsedConfig[:log_collection_settings][:schema][:containerlog_schema_version].nil?
@containerLogSchemaVersion = parsedConfig[:log_collection_settings][:schema][:containerlog_schema_version]
puts "config::Using config map setting for container log schema version"
end
rescue => errorStr
ConfigParseErrorLogger.logError("Exception while reading config map settings for container log schema version - #{errorStr}, using defaults, please check config map for errors")
end

#Get kube events enrichment setting
begin
if !parsedConfig[:log_collection_settings][:collect_all_kube_events].nil? && !parsedConfig[:log_collection_settings][:collect_all_kube_events][:enabled].nil?
Expand Down Expand Up @@ -200,6 +211,7 @@ def populateSettingValuesFromConfigMap(parsedConfig)
file.write("export AZMON_CLUSTER_CONTAINER_LOG_ENRICH=#{@enrichContainerLogs}\n")
file.write("export AZMON_CLUSTER_COLLECT_ALL_KUBE_EVENTS=#{@collectAllKubeEvents}\n")
file.write("export AZMON_CONTAINER_LOGS_ROUTE=#{@containerLogsRoute}\n")
file.write("export AZMON_CONTAINER_LOG_SCHEMA_VERSION=#{@containerLogSchemaVersion}\n")
# Close file after writing all environment variables
file.close
puts "Both stdout & stderr log collection are turned off for namespaces: '#{@excludePath}' "
Expand Down Expand Up @@ -246,6 +258,8 @@ def get_command_windows(env_variable_name, env_variable_value)
file.write(commands)
commands = get_command_windows('AZMON_CONTAINER_LOGS_EFFECTIVE_ROUTE', @containerLogsRoute)
file.write(commands)
commands = get_command_windows('AZMON_CONTAINER_LOG_SCHEMA_VERSION', @containerLogSchemaVersion)
file.write(commands)

# Close file after writing all environment variables
file.close
Expand Down
67 changes: 47 additions & 20 deletions kubernetes/linux/mdsd.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,20 +48,31 @@
-->
<Schemas>
<Schema name="ContainerLogSchema">
<Column name="Id" type="str" mdstype="mt:wstr" />
<Column name="LogEntry" type="str" mdstype="mt:wstr" />
<Column name="LogEntrySource" type="str" mdstype="mt:wstr" />
<Column name="LogEntryTimeStamp" type="str" mdstype="mt:wstr" />
<Column name="TimeOfCommand" type="str-rfc3339" mdstype="mt:utc" />
<Column name="SourceSystem" type="str" mdstype="mt:wstr" />
<Column name="Computer" type="str" mdstype="mt:wstr" />
<Column name="Image" type="str" mdstype="mt:wstr" />
<Column name="Name" type="str" mdstype="mt:wstr" />
</Schema>
<Column name="Id" type="str" mdstype="mt:wstr" />
<Column name="LogEntry" type="str" mdstype="mt:wstr" />
<Column name="LogEntrySource" type="str" mdstype="mt:wstr" />
<Column name="LogEntryTimeStamp" type="str" mdstype="mt:wstr" />
<Column name="TimeOfCommand" type="str-rfc3339" mdstype="mt:utc" />
<Column name="SourceSystem" type="str" mdstype="mt:wstr" />
<Column name="Computer" type="str" mdstype="mt:wstr" />
<Column name="Image" type="str" mdstype="mt:wstr" />
<Column name="Name" type="str" mdstype="mt:wstr" />
</Schema>
<Schema name="ContainerLogV2Schema">
<Column name="ContainerId" type="str" mdstype="mt:wstr" />
<Column name="LogMessage" type="str" mdstype="mt:wstr" />
<Column name="LogSource" type="str" mdstype="mt:wstr" />
<Column name="TimeGenerated" type="str" mdstype="mt:wstr" />
<Column name="PodName" type="str" mdstype="mt:wstr" />
<Column name="PodNamespace" type="str" mdstype="mt:wstr" />
<Column name="Computer" type="str" mdstype="mt:wstr" />
<Column name="ContainerName" type="str" mdstype="mt:wstr" />
</Schema>
</Schemas>

<Sources>
<Source name="ContainerLogSource" schema="ContainerLogSchema" />
<Source name="ContainerLogSource" schema="ContainerLogSchema" />
<Source name="ContainerLogV2Source" schema="ContainerLogV2Schema" />
</Sources>


Expand Down Expand Up @@ -97,15 +108,22 @@
priority events to be delivered sooner than the next five-minute interval.
-->
<MdsdEvents>
<MdsdEventSource source="ContainerLogSource">
<RouteEvent eventName="ContainerLogEvent" duration="PT10S" priority="High" storeType="Local" disabled="true">
</RouteEvent>
<!--
<RouteEvent eventName="ContainerLogEvent2File" duration="PT1S" priority="High" storeType="File">
</RouteEvent>
-->
<MdsdEventSource source="ContainerLogSource">
<RouteEvent eventName="ContainerLogEvent" duration="PT10S" priority="High" storeType="Local" disabled="true">
</RouteEvent>
<!--
<RouteEvent eventName="ContainerLogEvent2File" duration="PT1S" priority="High" storeType="File">
</RouteEvent>
-->
</MdsdEventSource>
<MdsdEventSource source="ContainerLogV2Source">
<RouteEvent eventName="ContainerLogV2Event" duration="PT10S" priority="High" storeType="Local" disabled="true">
</RouteEvent>
<!--
<RouteEvent eventName="ContainerLogEvent2File" duration="PT1S" priority="High" storeType="File">
</RouteEvent>
-->
</MdsdEventSource>

</MdsdEvents>


Expand All @@ -118,7 +136,16 @@
<Content>
<!-- <![CDATA[<Config workspaceName="CIWORKSPACE" workspaceId="7ab0f4bc-0d3e-4d05-8a19-e2b5c5bf2dc8" logType="CONTAINER_LOG_BLOB" ipName="Containers"/>]]> -->
<!--<![CDATA[<Config workspaceName="CIWORKSPACE" logType="CONTAINER_LOG_BLOB" flushFrequencyInSeconds="20" ipName="Containers"/>]]>-->
<![CDATA[<Config workspaceName="CIWORKSPACE" logType="CONTAINER_LOG_BLOB" ipName="Containers"/>]]>
<![CDATA[<Config workspaceName="CIWORKSPACE" logType="CONTAINER_LOG_BLOB" ipName="ContainerInsights"/>]]>
</Content>
</OMS>
</EventStreamingAnnotation>
<EventStreamingAnnotation name="ContainerLogV2Event">
<OMS>
<Content>
<!-- <![CDATA[<Config workspaceName="CIWORKSPACE" workspaceId="7ab0f4bc-0d3e-4d05-8a19-e2b5c5bf2dc8" logType="CONTAINER_LOG_BLOB" ipName="Containers"/>]]> -->
<!--<![CDATA[<Config workspaceName="CIWORKSPACE" logType="CONTAINER_LOG_BLOB" flushFrequencyInSeconds="20" ipName="Containers"/>]]>-->
<![CDATA[<Config workspaceName="CIWORKSPACE" logType="CONTAINERINSIGHTS_CONTAINERLOGV2" ipName="ContainerInsights"/>]]>
</Content>
</OMS>
</EventStreamingAnnotation>
Expand Down
Loading