sweeper+contractcourt: deadline aware HTLC claims & commitment confirmation

[Roasbeef]

One of the primary goals of anchor outputs is to be able to adjust fee rates to ensure that both the commitment transaction and the possibly contested HTLCs on the transaction get into the block in time. With our inclusion of anchor output enabled commitments, we're now able to do this using CPFP on the main commitment transaction, and either RBF or attaching a new input+output pair for the HTLC transactions. As we're concerned with the critical case of proper multi-hop HTLC resolution, we should expand all sub-systems related to commitment confirmation and HTLC claims to be able to ratchet up the fee rate with a deadline: the expiry of the CTLV delta between HTLC pairs.

Steps to Completion

• Expand the commitment anchor sweeping by the sweeper to allow the caller to specify a new constraint: a confirmation deadline. We visited this route in the past with the hypothetical "super sweeper" sub-system. One open question here still is: do we bump each block we aren't confirmed until we reach a max fee, or do we wait for a period of T blocks, then rachet all the way up to our final fee.

• Expose this new deadline formation in either PendingChannels, or PendingSweeps (walletkit sub-server).

• Dynamically bump the posted transaction fee as we gradually approach the deadline

This is related to #610, as we'll also want the initiator of the channel to be able to set the base commitment fee, and also update it as they wish. In addition, after that issue, we should start to target a higher confirmation goal (in blocks) than we do atm, since we only want to ensure the fee is enough to land in the mempool. Once in the mempool, we should use CPFP to bump accordingly as described above.

[yyforyongyu]

Interesting issue, I'd like to work on it. My thoughts on the main question,

One open question here still is: do we bump each block we aren't confirmed until we reach a max fee, or do we wait for a period of T blocks, then rachet all the way up to our final fee.

Two factors, how much fee to pay and when to broadcast, contribute to the probability of confirmation.

The assumptions are,

• the probability of confirmation increases if a larger fee paid, and,
• the probability of confirmation increases if broadcast earlier.

The most aggressive approach is to broadcast the transaction using the max fee immediately. It has the highest chance of success and costs the most. And the tradeoff is, of course, always between the risk and cost.

Suppose we have N blocks till the deadline is reached

A naive approach is to increase fee by a delta value each block till the deadline, where the delta is (max_fee - initial_fee) / N. Of course, we don't have to use a linear function. The fact is, when the deadline is far away, we are less stressed and thus in less hurry to bump the fee. As a reflection, a polynomial or exponential function may be applied, e.g., delta = ax^2 + b, where x is the time and a, b are some constants.

The actual value of N matters

If N is small, say 1 or 2 blocks, we don't have many choices but to bump it asap.
If N is huge, say hundreds of blocks, we have enough time to wait for confirmation.

As a starting point, according to this PR, we may check the N value against 34. if the deadline is 34 blocks away, we wait. Otherwise, we take action.

While the 34 specifies an upper bound, we want a lower bound. That is, if the deadline is only a few blocks away(say X), we will use up our max fee immediately. Again, according to this PR, we set the X to be equal to S, which is 12.

In summary,

• we apply a min_deadline_delta of 12 and max_deadline_delta of 34.
• If the deadline_delta is below min_deadline_delta, we use up the max fee immediately.
• If the deadline_delta is above max_deadline_delta(rare), we do nothing but wait.
• If it's in between, we apply a time-sensitive algorithm to bump the fee.

We also want to make the parameters configurable, leaving users a choice to CPFP in their own way(or they can use bumpclosefee).

Possible ways to bump fee
Suppose there are 100 units in (max_fee - initial_fee), at block delta 12 we want to spend all, and at block delta 34 we want to spend none. These functions, linear, quadratic, and exponential, are plotted here.

Another factor is the total values of HTLC outputs to be swept. The smaller it gets, the less likely a counterparty will appear and the less money we will lose in case of the worst scenario happens. This can further be reflected in min_deadline_delta if needed.

[joostjager]

Some thought/questions that I have around deadline-aware sweeping:

• how much fee to pay and when to broadcast

Can this be flattened to just "how much to pay at block expiry - x"? We can broadcast every block if we want to. Of course the fee can't be lowered, meaning that some broadcasts may be no-ops.

• Does history matter at all? If we are at block expiry - x, is it relevant what we did at expiry - x - 1? I would think it isn't relevant.

• What is the "max fee"? If we need to claim a 1 BTC htlc, are we willing to pay 1 BTC in fees at block expiry - 1? This can be very expensive if you've been offline for a while. A more sane maximum is probably a fee estimate with a very high probability of confirming in the next block, capped by the value at stake.

• Intuitively I agree with a non-linear function. It would be nice to have more evidence for that. Maybe it is an idea to run a simulation with historical block data and fee estimates to try out different curves. Observe the success rate (swept before deadline) and fee paid.

• I would avoid using highly specific constants like 12 and 34. It should be possible to solve this problem in a more generic way.

[yyforyongyu]

Can this be flattened to just "how much to pay at block expiry - x"?

Indeed, "when to broadcast" is not accurate. What matters is we do it early. My assumption is that, if transaction A
is broadcast at block expiry - x, transaction B, which is the same as A, is broadcast at expiry - x - 1, and we check the result at expiry, then transaction B should have a higher chance to be confirmed than A does.

Does history matter at all?

Nope, I don't think it matters. Moreover, I think the fee rate behaves pretty randomly...

What is the "max fee"?

I think this value is set by the user? There is this MaxFeeAllocation, default to 0.5.
My understanding is that, the precondition here is we have a max fee defined already, and we want to know how to spend it. We can only offer best-effort delivery, increase the chance to succeed while keeping the cost low. Once we've reached the max fee, we've done the job of spending it.

[joostjager]

we check the result at expiry, then transaction B should have a higher chance to be confirmed than A does.

I don't think you need to wait until expiry. Assuming both are published to the same mempool, B will have either replaced A or B is rejected (BIP125). Both can't exist at the same time.

My understanding is that, the precondition here is we have a max fee defined already, and we want to know how to spend it.

Yes, definition of max fee can be kept a precondition. It was more a broader question about how to define that. The MaxFeeAllocation isn't ideal, but we can work with it.

[yyforyongyu]

More discussion on this issue,

I don't think you need to wait until expiry. Assuming both are published to the same mempool, B will have either replaced A or B is rejected (BIP125). Both can't exist at the same time.

Sorry, maybe I wasn’t clear about my statements. I was referring to the occurrence of an event in terms of its probability, in particular, I was making the assumption that when a transaction is broadcast earlier, it has a higher chance to be confirmed before the deadline.

If it helps, the statement is similar to, if I go to work at 8 am I’d have a higher chance of not being late versus going to work at 9 am.

If this assumption is correct, then it means we want to broadcast early even the fee paid stays unchanged. We’d prefer expiry - x - 1 over expiry - x. Otherwise, the only factor under this context is the fee and the complexity would be greatly decreased.

I would avoid using highly specific constants like 12 and 34. It should be possible to solve this problem in a more generic way.

I think it’s up to the user to decide these values while we provide some defaults?

Intuitively I agree with a non-linear function. It would be nice to have more evidence for that. Maybe it is an idea to run a simulation with historical block data and fee estimates to try out different curves. Observe the success rate (swept before deadline) and fee paid.

In short term this may help, essentially I think a machine learning model is being developed here...It’s an interesting approach, while history data only has limited efficacy, especially in the long run. I will try to derive some statistics and see what I get;)