External integrations fail and cause SC to crash when dispatching an event for a control message that failed processing

[sveinfid]

Symptoms

External integrations would cease to work when encountered a control message that failed processing. Subsequent retries would cause circuit breaker to trigger and cause ServiceControl to shutdown. The problem is persistent and restarting would repeat the same process.

In ServiceControl 1.3.0 the external integration feature would just silently stop working without causing ServiceControl to crash.

Who's affected

• Users of ServiceControl 1.3.0, 1.4.0 and 1.4.1

Error report

We just deployed ServiceControl 1.4.1 to our test environment.
In our test environment we were previously running NServiceBus 2.0 and have just upgraded to NServiceBus 4.7.4, if that is of importance.

After a few minutes running, ServiceControl crashes with the following error message:

2014-11-27 12:58:13.9824|48|Fatal|NServiceBus|Repeated failures when dispatching external integration events.
System.Collections.Generic.KeyNotFoundException: The given key was not present in the dictionary.
   at ServiceControl.ExternalIntegrations.MessageFailedConverter.ToEvent(FailedMessage message) in c:\BuildAgent\work\b7f882efc2cd360b\src\ServiceControl\ExternalIntegrations\MessageFailedConverter.cs:line 16
   at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
   at System.Linq.Enumerable.<SelectManyIterator>d__14`2.MoveNext()
   at ServiceControl.ExternalIntegrations.EventDispatcher.DispatchEventBatch(CancellationToken token) in c:\BuildAgent\work\b7f882efc2cd360b\src\ServiceControl\ExternalIntegrations\EventDispatcher.cs:line 86
   at ServiceControl.ExternalIntegrations.EventDispatcher.DispatchEvents(CancellationToken token) in c:\BuildAgent\work\b7f882efc2cd360b\src\ServiceControl\ExternalIntegrations\EventDispatcher.cs:line 60
   at ServiceControl.ExternalIntegrations.EventDispatcher.<StartDispatcher>b__4() in c:\BuildAgent\work\b7f882efc2cd360b\src\ServiceControl\ExternalIntegrations\EventDispatcher.cs:line 33
   at System.Threading.Tasks.Task.Execute()

Any ideas what the problem is here? Is there a workaround for this issue?

Thanks,
Svein

[SzymonPobiega]

@sveinfid do you have any unprocessed messages in error or Particular.ServiceControl queues? Could this problem be caused by an old 2.0 error message being loaded by ServiceControl?

[sveinfid]

@SzymonPobiega I have no unprocessed messages in the error queue.
The particular.servicecontrol queue has about 40 heartbeat messages but, but nothing else.

I'm not sure if there were any old 2.0 error messages on the error queue before I started ServiceControl for the first time, I didn't check the queue before installing ServiceControl.