Skip to content

Don't crash on incomplete node data #599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
daweim0 wants to merge 7 commits into from
Closed

Don't crash on incomplete node data #599

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
43355a6
linux build speed improvements
daweim0 Jul 2, 2021
fd61e50
small changes
daweim0 Jul 2, 2021
f0f3cf7
added protection against node json with missing data
daweim0 Jul 7, 2021
e54ecf9
Merge branch 'david/faster_builds' into david/malformed-node-fix
daweim0 Jul 8, 2021
e6a5e4b
fixed build speedups
daweim0 Jul 8, 2021
f9fd570
Revert "fixed build speedups"
daweim0 Jul 8, 2021
f83ce83
Revert "Merge branch 'david/faster_builds' into david/malformed-node-…
daweim0 Jul 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 40 additions & 34 deletions source/plugins/ruby/in_kube_nodes.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -280,48 +280,54 @@ def parse_and_emit_records(nodeInventory, batchTime = Time.utc.iso8601)
timeDifference = (DateTime.now.to_time.to_i - @@nodeTelemetryTimeTracker).abs
timeDifferenceInMinutes = timeDifference / 60
if (timeDifferenceInMinutes >= Constants::TELEMETRY_FLUSH_INTERVAL_IN_MINUTES)
properties = getNodeTelemetryProps(item)
properties["KubernetesProviderID"] = nodeInventoryRecord["KubernetesProviderID"]
capacityInfo = item["status"]["capacity"]

ApplicationInsightsUtility.sendMetricTelemetry("NodeMemory", capacityInfo["memory"], properties)
begin
if (!capacityInfo["nvidia.com/gpu"].nil?) && (!capacityInfo["nvidia.com/gpu"].empty?)
properties["nvigpus"] = capacityInfo["nvidia.com/gpu"]
properties = getNodeTelemetryProps(item)
properties["KubernetesProviderID"] = nodeInventoryRecord["KubernetesProviderID"]
capacityInfo = item["status"]["capacity"]

ApplicationInsightsUtility.sendMetricTelemetry("NodeMemory", capacityInfo["memory"], properties)
begin
if (!capacityInfo["nvidia.com/gpu"].nil?) && (!capacityInfo["nvidia.com/gpu"].empty?)
properties["nvigpus"] = capacityInfo["nvidia.com/gpu"]
end

if (!capacityInfo["amd.com/gpu"].nil?) && (!capacityInfo["amd.com/gpu"].empty?)
properties["amdgpus"] = capacityInfo["amd.com/gpu"]
end
rescue => errorStr
$log.warn "Failed in getting GPU telemetry in_kube_nodes : #{errorStr}"
$log.debug_backtrace(errorStr.backtrace)
ApplicationInsightsUtility.sendExceptionTelemetry(errorStr)
end

# Telemetry for data collection config for replicaset
if (File.file?(@@configMapMountPath))
properties["collectAllKubeEvents"] = @@collectAllKubeEvents
end

if (!capacityInfo["amd.com/gpu"].nil?) && (!capacityInfo["amd.com/gpu"].empty?)
properties["amdgpus"] = capacityInfo["amd.com/gpu"]
#telemetry about prometheus metric collections settings for replicaset
if (File.file?(@@promConfigMountPath))
properties["rsPromInt"] = @@rsPromInterval
properties["rsPromFPC"] = @@rsPromFieldPassCount
properties["rsPromFDC"] = @@rsPromFieldDropCount
properties["rsPromServ"] = @@rsPromK8sServiceCount
properties["rsPromUrl"] = @@rsPromUrlCount
properties["rsPromMonPods"] = @@rsPromMonitorPods
properties["rsPromMonPodsNs"] = @@rsPromMonitorPodsNamespaceLength
properties["rsPromMonPodsLabelSelectorLength"] = @@rsPromMonitorPodsLabelSelectorLength
properties["rsPromMonPodsFieldSelectorLength"] = @@rsPromMonitorPodsFieldSelectorLength
end
# telemetry about osm metric settings for replicaset
if (File.file?(@@osmConfigMountPath))
properties["osmNamespaceCount"] = @@osmNamespaceCount
end
ApplicationInsightsUtility.sendMetricTelemetry("NodeCoreCapacity", capacityInfo["cpu"], properties)
telemetrySent = true
rescue => errorStr
Copy link
Member

@vishiy vishiy Jul 9, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

did you test what happens if any code in the block got rescued and rest (partial) of the code (after rescue) runs after the rescue ? Also whats the advantage of this rescue block ? This isn't a crash, correct? There are global rescues in outer methods (enumerate & runperiodic).

$log.warn "Failed in getting GPU telemetry in_kube_nodes : #{errorStr}"
$log.warn "Failed in getting telemetry in_kube_nodes : #{errorStr}"
$log.debug_backtrace(errorStr.backtrace)
ApplicationInsightsUtility.sendExceptionTelemetry(errorStr)
end

# Telemetry for data collection config for replicaset
if (File.file?(@@configMapMountPath))
properties["collectAllKubeEvents"] = @@collectAllKubeEvents
end

#telemetry about prometheus metric collections settings for replicaset
if (File.file?(@@promConfigMountPath))
properties["rsPromInt"] = @@rsPromInterval
properties["rsPromFPC"] = @@rsPromFieldPassCount
properties["rsPromFDC"] = @@rsPromFieldDropCount
properties["rsPromServ"] = @@rsPromK8sServiceCount
properties["rsPromUrl"] = @@rsPromUrlCount
properties["rsPromMonPods"] = @@rsPromMonitorPods
properties["rsPromMonPodsNs"] = @@rsPromMonitorPodsNamespaceLength
properties["rsPromMonPodsLabelSelectorLength"] = @@rsPromMonitorPodsLabelSelectorLength
properties["rsPromMonPodsFieldSelectorLength"] = @@rsPromMonitorPodsFieldSelectorLength
end
# telemetry about osm metric settings for replicaset
if (File.file?(@@osmConfigMountPath))
properties["osmNamespaceCount"] = @@osmNamespaceCount
end
ApplicationInsightsUtility.sendMetricTelemetry("NodeCoreCapacity", capacityInfo["cpu"], properties)
telemetrySent = true
end
end
if telemetrySent == true
Expand Down