[bug]: ChannelArbitrator does not cleanly stop

[Crypt-iQ]

When a ChannelArbitrator is finished with its duties, it attempts to stop itself but can't:

The channelAttendant goroutine calls advanceState which calls stateStep which hits this line if the state is StateFullyResolved:

lnd/contractcourt/channel_arbitrator.go

Lines 1280 to 1283 in e6fbaaf

	if err := c.cfg.MarkChannelResolved(); err != nil {
	log.Errorf("unable to mark channel resolved: %v", err)
	return StateError, closeTx, err
	}

MarkChannelResolved calls ResolveContract:

lnd/contractcourt/chain_arbitrator.go

Lines 399 to 404 in e6fbaaf

	arbCfg.MarkChannelResolved = func() error {
	if c.cfg.NotifyFullyResolvedChannel != nil {
	c.cfg.NotifyFullyResolvedChannel(chanPoint)
	}
	return c.ResolveContract(chanPoint)

ResolveContract calls Stop on the ChannelArbitrator:

lnd/contractcourt/chain_arbitrator.go

Lines 465 to 471 in e6fbaaf

	if chainArb != nil {
	arbLog = chainArb.log
	if err := chainArb.Stop(); err != nil {
	log.Warnf("unable to stop ChannelArbitrator(%v): %v",
	chanPoint, err)
	}

This will close the quit channel and wait for the waitgroup counter to go to zero. However, because this can be called from channelAttendant, the goroutine is waiting for itself to be cleaned up and will just hang. This doesn't cause the node to be unresponsive, but the ChannelArbitrator will sit there and not properly stop.

h/t @bitromortac

[ziggie1984]

The fix is actually in 9253, so reopening.