Error if there's dupe connections

[BrandonWeng]

Describe your changes and provide context

The peers fall into an unrecoverable state if they already have a connection to another peer and they try to connect again.

It seems like Peer A lost its connection to Peer B, but Peer B's mapping was not updated to be aware of this. So when Peer A tries to connect to peer B again, it fails. Therefore if we get a duplicate request where the peer is already connected, we should Error out the existing connection AND return error, this way if it retries it can successfully connect

Dec 12 15:02:02 ip-172-31-33-167 seid[16502]: 3:02PM ERR failed to accept connection err="peer \"4b336938f10433839baf727999e78965dcbae4d5\" is already connected" module=p2p op=incoming/accepted peer=4b336938f10433839baf727999e78965dcbae4d5
Dec 12 15:02:02 ip-172-31-33-167 seid[16502]: 3:02PM INF stopping service module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58322,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF starting service impl=MConnection module=p2p peer={"Hostname":"3.143.253.100","NodeID":"53dd9f3b8edf7e37ddd6ee152d0eec205181752b","Path":"","Port":26656,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF peer connected endpoint={} module=p2p peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF Connection is closed @ recvRoutine (likely by the other side) conn={} module=p2p peer={"Hostname":"3.143.253.100","NodeID":"53dd9f3b8edf7e37ddd6ee152d0eec205181752b","Path":"","Port":26656,"Protocol":"mconn"}
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF stopping service module=p2p peer={"Hostname":"3.143.253.100","NodeID":"53dd9f3b8edf7e37ddd6ee152d0eec205181752b","Path":"","Port":26656,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF received peer update module=statesync peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b status=up
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF peer disconnected endpoint={} module=p2p peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF received peer update module=statesync peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b status=down
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF starting service impl=MConnection module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58324,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM ERR failed to accept connection err="peer \"4b336938f10433839baf727999e78965dcbae4d5\" is already connected" module=p2p op=incoming/accepted peer=4b336938f10433839baf727999e78965dcbae4d5
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF stopping service module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58324,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF starting service impl=MConnection module=p2p peer={"Hostname":"3.143.253.100","NodeID":"53dd9f3b8edf7e37ddd6ee152d0eec205181752b","Path":"","Port":26656,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF peer connected endpoint={} module=p2p peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF Connection is closed @ recvRoutine (likely by the other side) conn={} module=p2p peer={"Hostname":"3.143.253.100","NodeID":"53dd9f3b8edf7e37ddd6ee152d0eec205181752b","Path":"","Port":26656,"Protocol":"mconn"}
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF stopping service module=p2p peer={"Hostname":"3.143.253.100","NodeID":"53dd9f3b8edf7e37ddd6ee152d0eec205181752b","Path":"","Port":26656,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF received peer update module=statesync peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b status=up
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF peer disconnected endpoint={} module=p2p peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b
Dec 12 15:02:03 ip-172-31-33-167 seid[16502]: 3:02PM INF received peer update module=statesync peer=53dd9f3b8edf7e37ddd6ee152d0eec205181752b status=down
Dec 12 15:02:04 ip-172-31-33-167 seid[16502]: 3:02PM INF starting service impl=MConnection module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58326,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:04 ip-172-31-33-167 seid[16502]: 3:02PM ERR failed to accept connection err="peer \"4b336938f10433839baf727999e78965dcbae4d5\" is already connected" module=p2p op=incoming/accepted peer=4b336938f10433839baf727999e78965dcbae4d5
Dec 12 15:02:04 ip-172-31-33-167 seid[16502]: 3:02PM INF stopping service module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58326,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:05 ip-172-31-33-167 seid[16502]: 3:02PM INF starting service impl=MConnection module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58328,"Protocol":"mconn"} service=MConnection
Dec 12 15:02:05 ip-172-31-33-167 seid[16502]: 3:02PM ERR failed to accept connection err="peer \"4b336938f10433839baf727999e78965dcbae4d5\" is already connected" module=p2p op=incoming/accepted peer=4b336938f10433839baf727999e78965dcbae4d5
Dec 12 15:02:05 ip-172-31-33-167 seid[16502]: 3:02PM INF stopping service module=p2p peer={"Hostname":"3.22.225.111","NodeID":"4b336938f10433839baf727999e78965dcbae4d5","Path":"","Port":58328,"Protocol":"mconn"} service=MConnection

Testing performed to validate your change

Deployed this in a test cluster and deployed to Peer B, and saw that the Peer A was then able to connect to Peer B again

[philipsu522]

This looks good. I think eviction should be ok, since we will retry anyways. Additionally, this logic seems to be used pretty liberally if at max capacity here https://github.com/sei-protocol/sei-tendermint/blob/main/internal/p2p/router.go#L406