BackgroundService handles Exceptions getting thrown differently when tasks are not faulted vs faulted

[PureKrome]

I think how exceptions are handled with BackgroundService class is a bit confusing and may require some discussion.

• If an exception is thrown during startup (e.g. in Main) and before ExecuteAsync is handled, the app will crash.
• If an exception is thrown during ExecuteAsync but before the first await is called, the app will crash.
• If an exception is thrown during ExecuteAsync but during or after the first await is called, the app will ... semi-hang? It stop processing ... but ... Ctrl-C does successfully start the app gracefully terminating.

Now - I hope i have the above steps/scenario's right -- I might have fudged something (async/await/ task contexts, etc are my weak points).

So the problem here is about the inconsistency about what an exception should be doing in a BackgroundService. Should it crash the host (assuming no top level try/catch in Main)? or get swallowed? Maybe the issue is really about documentation (as the code is totally fine).

This issue arose when I started a conversation in the "ASP.NET slack" group when I was trying to handle an exception getting thrown in side the ExecuteAsync method and bubbling that upwards.

To quote some of the summaries of this convo:

@davidfowl [7:06 PM]
great discussion
heh
it could suppress both and if you wanted to get exceptions, then you override StartAsync and put logic there

--

@chrisvanderpennen [7:13 PM]
theres imo three scenarios that need covering:
i don't want any exceptions in this to stop my host
i want an exception during init to stop the host startup, but runtime exceptions are ok
i want a runtime exception to stop my host

@davidfowl [7:30 PM]
somebody file a bug

so CC'ing @poke @gulbanana @chrisvanderpennen and @davidfowl who were in the convo.

• I think the issue subject might need to be refactored.
• happy to edit the main body of this issue, where i got my facts wrong.
• This issue needs the famous tag 'fowler' (fowler has issues). :)

edit: this was a sample gist I tried to make to also test/repo this problem.

[gulbanana]

imo the bug is that BackgroundService deliberately defers exceptions which are stashed in the Task, but can still be brought down by a Task-returning function which throws before or instead of awaiting. it’d be more consistent if it also used try-catch to intercept and store exceptions from that first section of the state machine.

the “semi-hang” might just be the intended behaviour of ConsoleLifetime though.

[gulbanana]

here’s one way that the exceptions could be given a consistent treatment, using a handler function which relies on AsyncTaskMethodBuilder to queue continuations onto the threadpool if they do occur:

https://github.com/gulbanana/local-async-example/blob/master/Program.cs

[muratg]

@davidfowl what's the work in 3.0 here?

cc @Tratcher

[davidfowl]

A few targeted changes to BackgroundService.

• We Task.Run the call to ExecuteAsync because user code cannot be trusted
• Exceptions thrown in the background service are observed in StopAsync but never in StartAsync.

[analogrelay]

@davidfowl are you going to do this?

[davidfowl]

Yes, I have a branch, I've just been lazy finishing it. I'll do it next week.

Curious if anyone has implemented a resilient pattern in their BackgroundService so that it will continue running even if an exception is thrown during ExecutionAsync

[PureKrome]

Sadness -> this is now in Backlog and will not ship with v3?

[analogrelay]

Correct. Unfortunately there was higher priority work that we had to bump this out for.

[chucker]

I would have expected worker services to have a built-in notion of global exception handling, but I guess not (yet)?

Anyway, here's what I do for now:

protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
    while (!stoppingToken.IsCancellationRequested)
    {
        try
        {
            _logger.LogInformation("Worker running at: {time}", DateTimeOffset.Now);

            // actual stuff happens here
        }
        catch (Exception ex)
        {
            // waiting on https://github.com/aspnet/Extensions/issues/813
            _Logger.LogError(ex, "Global exception occurred. Will resume in a moment.");
        }

        await Task.Delay(1000, stoppingToken);
    }
}