Ensure raft ID uniqueness

[aaronlehmann]

Currently, raft IDs are randomly generated on the client side before joining the cluster. If the ID it generates collides with an ID that's already in use, this is not handled gracefully.

There are two ways I can see to handle this properly:

• Generate raft IDs on the server side: Have the Join RPC call to respond with a valid ID. This either requires that Join requests are forwarded to the leader, or that Join retries proposing a configuration change until one gets committed with an ID that doesn't conflict with anything else in the log. The logic around this could be a bit complex.
• Continue to generate the ID client-side, but have the Join RPC call return an error if the ID is not unique (after the ProposeConfChange raft round). Have the client retry with a different ID.

cc @abronan

[abronan]

Just sharing a few thoughts here. As discussed with @aaronlehmann, the second option is not as easy because the network can behave erratically and a client could submit multiple Join requests with different IDs which will require additional checks server side anyways.

Also, I don't think a new Manager should be trusted at the Join phase but only when the ConfChange gets submitted, the connection is initiated and we make sure the node is not malicious (meaning at least not storming with Raft Join/ConfChange requests). Potentially even prevent a node to be processed on Join if we suspect a malicious behavior.

Keeping track of this adds much more complexity though but we should make sure that we don't end up with a hundred registered raft node while we have only four. DDoS will always be a problem but it's much more sensitive on the Managers which are replicating the state.

The most important scenario we should avoid is:

• A client could send multiple Join requests on multiple raft servers and possibly with different IDs, removing its cluster state every time (byzantine failure, ie. crash amnesia).

Potential solution: we can generate IDs on Join but check only after the ConfChange is sent to every member. Operation should be idempotent on every existing member. This means that we have a way to uniquely identify a node with different attributes (certs, IP, etc.) and make sure all the Managers have the same signature for that node. A ConfChange is processed only if the node is not already registered in the memberlist. Following ConfChange requests for that node signature are thus rejected on every member.

/cc @diogomonica

[diogomonica]

My thoughts:

• Let's put the client ID inside of the TLS certificate of the node
• The agent, by virtue of using the TLS cert, has a certain ID that the manager can verify
• The CA issues certificates with a a big enough ID such that random collisions are impossible (at least in the lifetime of our universe)
• We can think about not allowing the same ID to be registered twice simultaneously (if it is already registered in the cluster, we can reject a Join). This would mitigate people copying certificates around by accident and having collisions between nodes.

[aaronlehmann]

@abronan: Is this blocked on #504? Not sure I see a better way than having the leader propose the configuration changes.