coordinator: add notion of staleness

[burgerdev]

The Coordinator has an internal in-memory state and an external state managed by a Store implementation. There's currently a two-way synchronization between the two:

• SetManifest requests result in store updates.
• Out-of-band store updates result in state refreshes (implemented in syncState).

The latter mechanism is not suitable for distributed Coordinators as specified in RFC010, because it derives new mesh certificates on persistency changes, although the Coordinator that wrote the update already derived its own mesh cert. Thus, we need to replace the store-to-state update happening in syncState with a peer recovery attempt. It's important to note that syncState was misguided to begin with, even for a single coordinator, because an out-of-band update could not really happen to a running Coordinator.

As a first step towards peer recovery, I'm replacing the syncState function with the concept of state staleness, which implements the recovery mode outlined in the RFC. A Coordinator starts out uninitialized, becomes ready through a first manifest set, a user recovery or a peer recovery, and eventually becomes stale because other coordinators receive manifest updates.

stateDiagram-v2
    state "out-of-band manifest update" as oob
    [*] --> uninit
    uninit --> userapi.Recover
    userapi.Recover --> ready
    uninit --> meshapi.Recover
    meshapi.Recover -->  ready
    ready --> oob
    oob --> stale
    stale --> userapi.Recover
    stale --> meshapi.Recover

Loading

The coordinator would be in recovery mode (i.e., should attempt to do peer recovery and should accept user recovery) when it is either uninitialized or stale. Staleness is a property of the state itself, and when a state becomes stale it never becomes fresh again (a new state object could be fresh, though). This is why it's safe to track staleness as a boolean field in State that is only ever flipped in one direction.

All API methods verify that their state is fresh before responding. However, state can become stale after the Coordinator started responding to a request. This is perfectly fine for the meshapi, where the client is just unlucky to initialize during a manifest update, but still gets certificates for the existing deployment. In the userapi, this means that there are concurrent requests. GetManifests returns a state, but not necessarily the latest committed to storage, which is acceptable. Concurrent calls to SetManifest will fail eventually due to the CompareAndSwap logic in the store.

Staleness is not checked for meshapi functions: they should work on the state they are invoked with, regardless of that state becoming stale.

[davidweisse]

Overall LGTM

stale