Raiden gets stuck in Update NonClosing balance proof transaction

[BCsRock]

Hi guys,

tl;dr:
this could be a bug in case an account joins a payment channel with zero Ethereum or having not enough funds to pay the update / settlement transactions. It seems to run into a stucked state with a repeatedly failing transaction.

Find debug.log file further below.

Problem Definition

Following scenario was tested :

Acc1 had some ERC20 token and some Ethereum
Acc2 had no ERC20 token and no Ethereum

Acc1 registered ERC20 token as Raiden payment channel network, opened a channel with Acc2, deployed a deposit, sent some ERC20 token off-chain to Acc2.
All of this was successful.

Then, Acc1 called a settlement which caused a transaction to be mined -> was successful.

Then Acc2 should have called a transaction but this failed due to low gas error.

Then Acc1 sent on-chain some ethereum to Acc2.

Restarting Raiden client with Acc2 tries to make Update Nonclosing balance proof transaction transaction, but fails with error message below:

Update NonClosing balance proof transaction threw. Receipt=AttributeDict({'blockHash': HexBytes('0x79c90b22f969b6d7b0067a9ba068519c8559ae453e980e311d21861436f28d09'), 'blockNumber': 3801406, 'contractAddress': None, 'cumulativeGasUsed': 1156498, 'gasUsed': 96119, 'logs': [], 'logsBloom': HexBytes('0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000'), 'root': None, 'status': 0, 'transactionHash': HexBytes('0x07416563ae15729a7c7b76048f5e69428866accd495231ee4bbf3135a4de525a'), 'transactionIndex': 15})

The raiden-deub.log can be found via gist record.

The ever failing transactions from restarting Acc2 can be found for example here or here

Now the problem is, the channel is officially in the state closed but I can neither send any Tokens between Acc1 and Acc2. Acc1 has received the correct balance from the off-chain transfers, but Acc2 did not receive a balance. I can also not re-open the channel, because it already exists. If I see correctly, the ERC20 tokens for Acc2 are frozen in the smart contract which I cannot settle / update.

System Description

Ubuntu 16.04 LTC
Raiden-0.4.2
Parity/v1.10.6-unstable-bc0d134-20180605/x86_64-linux-gnu/rustc1.26.2

[BCsRock]

P.S. I am not sure, but it might correlate to issue 3 of #2086

[LefterisJP]

Hello @BCsRock. Thank you for your issue report.

Then, Acc1 called a settlement which caused a transaction to be mined -> was successful.

I guess you mean that it called close? This is the only thing acc1 could have done at that point.

Restarting Raiden client with Acc2 tries to make Update Nonclosing balance proof transaction transaction, but fails with error message below:

How can Acc2 send a transaction at this point? Did you refill it with some Ether?

If yes then the position at which your node is in, it would indeed be one of being stuck due to sending repeatedly the updateNonClosingBalanceproof() transaction.

With #2086 this should have been addressed.

@hackaugusto or @rakanalh can you check and make sure this is properly handled?

[hackaugusto]

Hi @BCsRock , thanks for the report.

this could be a bug in case an account joins a payment channel with zero Ethereum or having not enough funds to pay the update / settlement transactions. It seems to run into a stucked state with a repeatedly failing transaction.

In the latest release (v0.5.0 - Tanacetum) we added two strategies to mitigate this problem:

• The user won't be able to open a channel if he doesn't have enough ether to complete the channel lifecycle. This, however, doesn't fix the case where a node acc1 which has ether opens a channel with a node acc2 which doesn't have ether.
• We added a warning to inform the user when the account's ether is insufficient, (issue Simple lookahead strategy for gas and raiden transactions #1998 , implemented by Gas lookahead strategy #2031), so the node acc2 would have a notification printed to the console when acc1 opens a channel with him.

This is not a perfect strategy and will be improved, for more information refer to the issue #1861

Restarting Raiden client with Acc2 tries to make Update Nonclosing balance proof transaction transaction, but fails with error message below:

Prior to v0.5.0 we had to retry all transactions on every restart (not necessarily all where re-sent, but that was the case for update transfer), as @LefterisJP mentioned this was fixed (issue #1950) which is integrated since v0.5.0. So, on the latest version transaction known to fail would not be sent.

Details about the error:

• the channel was opened with a settlement window of 501 blocks.
• the channel was closed at block 3791424
• The failing transaction linked was at the block height 3801071, which is 9146 blocks after the settlement window, that is why the resending the transaction failed, even thought acc2 had ether.

With the above said we have to do two things in order to fix this (as of v0.5.0):

• The state machine assumes the partner will always successfully call update transfer, and includes both balance proofs in the settlement call, if the update transaction fails then the settle transaction will also fail. To fix this the settle state change should not include the balance proof, but the balance hash, and the balance proof should be recovered from the WAL.
• The transaction queue is cleared exclusively based on the available events, in this case the settle event is necessary to clear the update transaction from the queue, but the settle event would never happen because of the above bug, so we should add the block number and clear time dependent transactions which have lost the time frame to successfully execute.

[BCsRock]

Hey @LefterisJP and @hackaugusto , thanks a lot guys. I was exciting to see this and other issues addressed in 0.5.0. Absolutely excited to test it out.

This, however, doesn't fix the case where a node acc1 which has ether opens a channel with a node acc2 which doesn't have ether.

I guess then we leave this issue open until this is addressed as well?
I am thinking if this could be a spam attack vector? -> opening a lot of dead payment channels with empty addresses could make the pathfinding logic more painful?

A solution could be: instead of opening directly a channel on-chain, first the 2 clients exchange secret proofs that they both will enable a payment channel with balance x and then one party of both may be able to settle the transaction on the blockchain.
On the other hand, the counterparty is always at right to move Ethereum away from the account (only the ERC20 tokens are in smart contract custody), making it dead afterwards. However, it would make spamming the network with dead channels more expensive as Ethereum has to 1. move to the account and 2. go away from the account.

Alternative: do not only lock ERC20 tokens in the contract but a minimum amount of Ethereum too.

[LefterisJP]

I guess then we leave this issue open until this is addressed as well?

This can't be addressed and is not really a problem. If you open a channel with someone who does not have enough ether to go through the channel lifecycle that's fine. If that someone does not update his side of the channel within the appropriate amount of time allotted to him by the settlement period the channel can still close.

We could have a check before opening a channel with someone to see if he has enough ether but as you said he can also just move his ether away at any point.

[hackaugusto]

We introduced a lookahead strategy a while ago, #2031, a node without enough funds will be warned, this is the best we can do.