Fallback chains

[thewhaleking]

Adds subclasses of SubstrateInterface and AsyncSubstrateInterface that allow for handling chain failures by using backup chains. If a sustained network failure is encountered on a chain endpoint, the object will initialize a new connection on the next chain in the fallback_chains list. If the retry_forever flag is set, upon reaching the last chain in fallback_chains, the connection will attempt to iterate over the list (starting with url) again.

E.g.

substrate = RetrySyncSubstrate(
    "wss://entrypoint-finney.opentensor.ai:443",
    fallback_chains=["ws://127.0.0.1:9946"]
)

In this case, if there is a failure on entrypoint-finney, the connection will next attempt to hit localhost. If this
also fails, a MaxRetriesExceeded exception will be raised.

substrate = RetrySyncSubstrate(
    "wss://entrypoint-finney.opentensor.ai:443",
    fallback_chains=["ws://127.0.0.1:9946"],
    retry_forever=True
)

In this case, rather than a MaxRetriesExceeded exception being raised upon failure of the second chain (localhost), the object will again being to initialize a new connection on entrypoint-finney, and then localhost, and so on and so forth.

[basfroman]

we shouldn't release it without minimum tests coverage of retry behavior

[thewhaleking]

we shouldn't release it without minimum tests coverage of retry behavior

Yeah. Testing it (in an automated, repeatable way) is difficult. Will work on it.

[basfroman]

we shouldn't release it without minimum tests coverage of retry behavior

Yeah. Testing it (in an automated, repeatable way) is difficult. Will work on it.

I think a simple e2e test will be enough for a start:

1. Start a localnet (fast-blocks enough)
2. Create a subnet (using regular Substrate class instance), publish some data (for example, a classic commitment)
3. Recreate the connection with the list ["ws://127.0.0.111:9944", "ws://127.0.0.1:9944"]. The first will give an error, the second will connect successfully
4. Get a commitment from the subnet and assert it