Better node identification

[chanwit]

From the last July Swarm2K experiment, there were a lot of leave / join operations (more than 3,000 times I guessed).

Some nodes joined unsuccessfully and left ~800 DOWN or UNKNOWN entries inside the log.
Is it possible to have a better mechanism to identify the nodes? So when they rejoin, the Swarm will know that they are the old nodes trying to join, not the whole new set of nodes.

[aluzzardi]

@aaronlehmann @diogomonica I think this is related to re-using the existing CSR or something like that which we discussed earlier today.

Is there an issue for that already or is this the master issue? If so, we should probably make it a 1.13 P2 IMO.

[aaronlehmann]

I was thinking of #756, but we closed that one becaused we stopped doing manual node acceptance. I think this is a bit different. Let's use this as the master issue.

[aluzzardi]

Sounds good. Feel free to rephrase the issue title

[diogomonica]

I don't think we will ever do this: So when they rejoin, the Swarm will know that they are the old nodes trying to join, not the whole new set of nodes.

If a node leaves the cluster, and removes the credentials, it will never be assigned the same ID again, since it can't prove that it owned it before (IDs are assigned by the managers).

Now, if this is a problem with join where a node tries to join, doesn't succeed but creates a raft node entry and when it tries again the system ends up with 2 node entries, then yes, we should fix it.

Also, auto-removal of node for a legitimate swarm leave will probably help with this.

[aluzzardi]

@diogomonica I think the title is misleading, but this is the issue I mentioned where duplicate nodes were popping up. My theory is somehow join failed and they retried but didn't reuse the existing key material.

[aaronlehmann]

@aluzzardi: The node ID is assigned by the manager, so I don't think reusing key material would help.

[diogomonica]

The only think that helps is to store the NodeID that is returned from the IssueCertificate call, and if there is a failure between that andCertificateStatus where we actually retrieve the certificate, we could recover by jumping to CertificateStatus, instead of regenerating the CSR, and applying for a new certificate.

Note that there is nothing that protects us against a request that creates a node in Raft, but fails to return the NodeID back to the caller. The remote node has no recourse but to ask for a totally new certificate.

[aluzzardi]

ping @diogomonica - do we want to do this for 1.13?