The API used by the ServicePulse monitoring page does not work for certain messages

[kbaley]

NOTE: issue edited for release

Symptoms

The ServicePulse monitoring page fails to load, and message similar to the following is logged by ServiceControl:

Visiting /monitored-endpoints/<EndpointName>?history=5 for a problem endpoint locally (i.e. using localhost on the SC server itself) gives the following error (for version 5.0.3):
{
"message": "An error has occurred.",
"exceptionMessage": "Object reference not set to an instance of an object.",
"exceptionType": "System.NullReferenceException",
"stackTrace": " at ServiceControl.Monitoring.Http.Diagrams.DiagramApiController.GetSingleEndpointMetrics(String endpointName) in /_/src/ServiceControl.Monitoring/Http/Diagrams/DiagramApiController.cs:line 158\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()"
}

Who's affected

you are affected if you are using ServiceControl 4.32.3 or later and, when you attempt to open the monitoring page for an endpoint in ServicePulse, it fails to load.

Root cause

Details

### Describe the bug

Description

For certain messages where NServiceBus.ContentType is TaggedLongValueOccurrence, this line in DiagramApiController, reports null and fails. This causes ServicePulse to show an error that it can't connect to ServiceControl on the monitoring page to view metrics for a single endpoint.

The cause appears to be a malformed body for these messages. When the body is deserialized for these malformed messages, it looks like one of the types it is trying to parse is empty. So the current theory is that when the TaggedLongValueOccurrence message is initially created, it's not able to get all the types it needs.

Expected behavior

The GetSingleEndpointMetrics controller method shouldn't report an HTTP exception.

Actual behavior

The method throws a NullReferenceException

Versions

5.0.3-5.0.5 for sure. Probably others close to this version.

Steps to reproduce

For endpoints that exhibit this behaviour, it starts right after it starts processing messages. If the SC monitoring instance is restarted, the monitoring page for the endpoint works until messages start getting reported.

Relevant log output

Visiting /monitored-endpoints/<EndpointName>?history=5 for a problem endpoint locally (i.e. using localhost on the SC server itself) gives the following error (for version 5.0.3):


{
"message": "An error has occurred.",
"exceptionMessage": "Object reference not set to an instance of an object.",
"exceptionType": "System.NullReferenceException",
"stackTrace": " at ServiceControl.Monitoring.Http.Diagrams.DiagramApiController.GetSingleEndpointMetrics(String endpointName) in /_/src/ServiceControl.Monitoring/Http/Diagrams/DiagramApiController.cs:line 158\r\n at lambda_method(Closure , Object , Object[] )\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ActionExecutor.<>c__DisplayClass6_2.<GetExecutor>b__2(Object instance, Object[] methodParameters)\r\n at System.Web.Http.Controllers.ReflectedHttpActionDescriptor.ExecuteAsync(HttpControllerContext controllerContext, IDictionary`2 arguments, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ApiControllerActionInvoker.<InvokeActionAsyncCore>d__1.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__15.MoveNext()"
}

For reference, this is the line in question.

### Additional Information

#### Workarounds

#### Possible solutions

#### Additional information
</details>

[kbaley]

Diagnosing further, this line seems like the likely culprit. We're creating a tag based on the message type but inserting an empty string if it's null. So there could be scenarios where the message type is not known, maybe with native integration or some behaviour that's changing the message type header?

[RemyDuijkeren]

It improved in ServicePulse v1.38.2 and ServiceControl v5.1.0.

I can often now click-through to the details of the endpoint, but now often the page doesnt contain any data. See screenshot below:

I get the feeling this happens after scaling up and then scaling back. The overview still reports 1/2 but at the heartbeats page it shows correctly 1 instance for this endpoint.

[PhilBastian]

release scheduled for 5.6.1

[RemyDuijkeren]

I have rolled this out over the last weekend. So far, I can confirm it fixed the issue. 👍