Hooks abandon their payloads when plugins stopped

[rustyrussell]

Issue and Steps to Reproduce

In particular, this can lead to stuck HTLCs:

def test_hook_chaining_dynamic(node_factory, bitcoind):
    """Test what happens when we remove a plugin while a hook in use"""
    l1, l2 = node_factory.line_graph(2, opts={'may_reconnect': True})

    # We want this to fail, so we can keep retrying.
    routestep = {'msatoshi': 124, 'id': l2.info['id'], 'delay': 5, 'channel': l1.get_channel_scid(l2)}
    l2.rpc.plugin_start(os.path.join(os.path.dirname(__file__), "plugins/hold_htlcs.py"))

    l1.rpc.sendpay([routestep], '00' * 32)
    l2.daemon.wait_for_log('hold_htlcs.py: Holding onto an incoming htlc for 10 seconds')
    l2.rpc.plugin_stop('hold_htlcs.py')

    # Restarting gets things unstuck!  Uncomment to see expected HTLC failure
    # l2.restart()

    # The effect is that we hold onto the HTLC forever.  Nasty, but we should do
    # OK if we have to abandon it.
    bitcoind.generate_block(9)
    l1.daemon.wait_for_log('hit deadline')
    
    bitcoind.generate_block(5, wait_for_mempool=1)
    
    with pytest.raises(RpcError, match=r"WIRE_PERMANENT_CHANNEL_FAILURE"):
        l1.rpc.waitsendpay('00' * 32)

[rustyrussell]

Also commands are stuck, FWIW.

[cdecker]

Well, that sent me down quite a rabbit hole: turns out we weren't detecting plugin stops correctly (stdout being closed gets noticed since we poll for it, but stdin does not since we only poll it when we try to write to it).

I have a patch series that is still a bit flaky but should have it up and running shortly.

[niftynei]

@cdecker do you have an update on this for the 0.8.2 release?

[cdecker]

The hanging HTLCs are resolved correctly now, by failing over to the next plugin or by reverting back to lightningd. RPC calls however currently still hang, I'll add a fix tomorrow.