routing+invoices: allow multi-part payment probes

[joostjager]

Since #3156 it is possible to probe destinations before paying to get an accurate fee estimate upfront.

For single-shot payments can be done by sending a payment with an unknown payment hash and inspecting the returned failure. If it originates from the final destination, the route is deemed to be valid. Following through with the real payment is very likely to succeed.

For multi-part payments, it isn't that simple. Ideally you want to be sure that the selected routes can carry all parts at the same time. Because that is what the receiver will do. They will hold on to every part until the set is complete. However, if you use the single-shot probing method described above, htlcs will already be cancelled back before the set is complete. So the set may complete sequentially, but still fail when the real payment is executed.

The simplest example of this is a 20 sat channel when you need to pay 40 sat. It isn't possible. But if you probe two times 20, it looks as if it is possible.

So what is needed is a way to get the receiver to hold on to the payments until the set is complete and then cancel.

[joostjager]

One way to do it may be to add a custom tlv record that signals probe. In that case, the receiver will cancel the set with a new failure code MPP_PROBE_SUCCESS.

Question is whether it is a problem to probe with the real payment hash. You'd need to trust the receiver that they don't take the probe as the real payment. It may not be the end of the world if they do. Also how to find out whether they support the new probe tlv record? Maybe the probes can use unknown hashes, but inside the probe tlv, there is the real payment hash. The htlc aren't locked to that, so can never be settled, but the invoice is still referenced. Alternatively use the payment address to reference the invoice.

[joostjager]

Latest insight is to use a spontaneous multi-part payment (lightning/bolts#658) to do the probing.

[Roasbeef]

Possible now via AMP, but not yet manifested on the RPC interface. An easy way to do this, would be to create a new first-class probing API that always attempts to send an invalid share as the "last" shard. Alternatively, we can use the other mechanism that instead