Node crashes with new_netting_channel failed

[Dominik1999]

Intro

I tried to join the WETH network after a previous crashed. When I clicked on the "join network" button my node crashed with the following error message.

Problem Definition

018-10-29 13:30:42 [debug    ] State change                   [raiden.raiden_service] node=2a7d58e6 state_change={"block_number": 6605554, "gas_limit": 8000029, "block_hash": "0xc3c1a322c93d4dc7d119dc448f0d282bc3a0c35958f0710cf7196145540d0b08", "_type": "raiden.transfer.state_change.Block", "_version": 0}
2018-10-29 13:30:43 [debug    ] Getting token list             [raiden.api.rest] node=2a7d58e6 registry_address=0xB9633dd9a9a71F22C933bF121d7a22008f66B908
2018-10-29 13:30:43 [info     ] 127.0.0.1 - - [2018-10-29 14:30:43] "GET /api/1/tokens HTTP/1.1" 200 182 0.002705 [raiden.api.rest.pywsgi] 
2018-10-29 13:30:43 [debug    ] Getting connection managers info [raiden.api.rest] node=2a7d58e6 registry_address=0xB9633dd9a9a71F22C933bF121d7a22008f66B908
2018-10-29 13:30:43 [info     ] 127.0.0.1 - - [2018-10-29 14:30:43] "GET /api/1/connections HTTP/1.1" 200 110 0.003362 [raiden.api.rest.pywsgi] 
2018-10-29 13:30:44 [debug    ] Calculated gas price for transaction [raiden.network.rpc.client] calculated_gas_price=6842397831 node=2a7d58e6 node_gas_price=10000000000
2018-10-29 13:30:44 [debug    ] send_raw_transaction called    [raiden.network.rpc.client] gasLimit=6400000 gasPrice=6842397831 node=2a7d58e6 nonce=1
2018-10-29 13:30:44 [critical ] new_netting_channel failed     [raiden.network.proxies.token_network] peer1=2a7d58e6 peer2=10f4bc8c
2018-10-29 13:30:44 [critical ] Unhandled exception when processing endpoint request [raiden.api.rest] node=2a7d58e6
Traceback (most recent call last):
  File "site-packages/raiden/network/rpc/smartcontract_proxy.py", line 77, in transact
  File "site-packages/raiden/network/rpc/client.py", line 414, in send_transaction
  File "site-packages/web3/eth.py", line 274, in sendRawTransaction
  File "site-packages/web3/manager.py", line 112, in request_blocking
ValueError: {'code': -32010, 'message': 'Insufficient funds. The account you tried to send transaction from does not have enough funds. Required 43791346118400000 and got: 9219772000000000.'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "site-packages/flask/app.py", line 1813, in full_dispatch_request
  File "site-packages/flask/app.py", line 1799, in dispatch_request
  File "site-packages/flask_restful/__init__.py", line 480, in wrapper
  File "site-packages/flask/views.py", line 88, in view
  File "site-packages/flask_restful/__init__.py", line 595, in dispatch_request
  File "site-packages/webargs/core.py", line 478, in wrapper
  File "site-packages/raiden/api/v1/resources.py", line 194, in put
  File "site-packages/raiden/api/rest.py", line 648, in connect
  File "site-packages/raiden/api/python.py", line 207, in token_network_connect
  File "site-packages/raiden/connection_manager.py", line 167, in connect
  File "site-packages/raiden/connection_manager.py", line 431, in _open_channels
  File "src/gevent/greenlet.py", line 849, in gevent._greenlet.joinall
  File "src/gevent/greenlet.py", line 865, in gevent._greenlet.joinall
  File "src/gevent/greenlet.py", line 312, in gevent._greenlet.Greenlet._raise_exception
  File "site-packages/gevent/_compat.py", line 47, in reraise
  File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
  File "site-packages/raiden/connection_manager.py", line 342, in _join_partner
  File "site-packages/raiden/api/python.py", line 302, in channel_open
  File "site-packages/raiden/network/proxies/token_network.py", line 170, in new_netting_channel
  File "site-packages/raiden/network/proxies/token_network.py", line 202, in _new_netting_channel
  File "site-packages/raiden/network/rpc/smartcontract_proxy.py", line 85, in transact
raiden.exceptions.ReplacementTransactionUnderpriced: Transaction was rejected. This is potentially caused by the reuse of the previous transaction nonce as well as paying an amount of gas less than or equal to the previous transaction's gas amount
2018-10-29 13:30:44 [debug    ] REST API stopped               [raiden.api.rest] node=2a7d58e6

more logs here - https://gist.github.com/Dominik1999/4c12e0daa360fc1da50a1afb86de12c3

Issue checklist

Read through Requirements for safe usage mentioned in our documentation and make sure your usage follows the requirements.

1. Is your Ethereum node syncing and up to date with the blockchain?
2. Are you using the same Ethereum account with anything other than the Raiden node? Or are you running multiple Raiden nodes with the same Ethereum account?
3. Does your Ethereum account used by Raiden have sufficient ETH?
4. Did you alter or delete the local DB that exists by default at ~/.raiden?
5. Did you stop / restart your raiden node while having open channels and/or pending transfers? if so, please provide further information.
6. Did you stop / restart your Ethereum node while Raiden was running?
7. Did you switch from one Ethereum node to another? example: switching from a local Ethereum node to an Infura one.
8. Is Raiden REST API protected from public access?

System Description

Here add a detailed description of your system, e.g. output of the following script:

Linux dominik-ThinkPad-T450s 4.19.0-041900-generic #201810221809 SMP Mon Oct 22 22:11:45 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

/usr/bin/solc
solc, the solidity compiler commandline interface
Version: 0.4.25+commit.59dbf8f1.Linux.g++

Geth (I use the brainbot node via VPN Access)

raiden-v0.15.1.dev6+g14cba904-linux

[LefterisJP]

Duplicate of #2924 -- you don't have enough funds

[Dominik1999]

That's not me to judge whether you consider this as a bug or not, but

• as a user using the UI I would expect that there pops up a message saying "You don't have enough funds", here the node crashed
• in addition, for me as a user using the UI, I can not see how much funds are needed. I just see that my node crashes

[LefterisJP]

@Dominik1999 you are actually right. It should. Let me check this.

[LefterisJP]

Hmmm there is an additional exception in there. So this seems like the other issue which is also seen by some people with ReplacementTransactionUnderpriced.

raiden.exceptions.ReplacementTransactionUnderpriced: Transaction was rejected. This is potentially caused by the reuse of the previous transaction nonce

[LefterisJP]

I think this should be solved also by #2943

[hackaugusto]

I don't think #2943 fixed this, the original exception is complaining about lack of funds

[LefterisJP]

The reason I closed it was that I thought the lack of funds would have been caught but due to the connection manager spawning multiple open channels we hit the nonce issue before the funds error having enough time to propagate. But I checked and that's not possible.

The lack of funds is not caught by the gas reserve check.

That is probably due to the codepath being in the connection manager spawning multiple _join_partner greenlets all of which are trying to create a new channel so there is a potential race there or the has_enough_gas_reserve() function is not operating as we expect.

If it's the race issue then perhaps a lock can help?