Detailed documentation for ServingState semantics

[mattmoor]

This covers documenting the detailed semantics of ServingState.

In particular, I am interested in:

1. ServingState is desired state, how do I check the observed state?
2. How does Active -> {Reserve,Retired} affect Revision conditions (e.g. Ready)? Are new conditions needed?
3. How do we ensure that the activator is wired up before resources are torn down?

To be clear, I expect that for some of the above additional development work will be necessary, and I'd like to reach a point where we're updating docs either before or alongside the code.

Related (pre-req): #644

[mattmoor]

Lifecycle of a Revision (today)

Before proposing an alternative, I wanted to sketch my understanding of the current thinking today, and where I believe there are weaknesses we need to address. This may be wholly inaccurate, so please LMK where my understanding is flawed.

Event: First deploy

The Revision is created in an Active state. Deployment and Service are created. When the Service’s Endpoint contains an entry, the Revision is declared “Ready”.

Event: Scaled to Zero

After a period of inactivity, the autoscaler puts the Revision into the “ServingState” of “Reserve”.

The Route controller observes this state change, and redirects the ingress to an “Activator” component. Keying off of the same signal, the Revision controller deletes the underlying K8s resources. There is a race here.

Despite the endpoint having no entries the Revision won’t leave a “Ready” state because of this logic (IIUC). No information is conveyed to indicate that the Revision has been actually scaled to zero and is unprepared to receive traffic.

Event: Scaled from Zero

Traffic arrives for our Revision, but hits the activator. The activator looks up the Revision that should be receiving the traffic and sets its “ServingState” to “Active”.

However, because “Active” is indistinguishable from “Reserve”, the activator has no meaningful way of knowing when the Revision has recreated its elements (and they are ready).

---

Again, this may be a completely flawed representation of the plan, but it reflects my understanding of the plan, and at a minimum highlights our need to better document the semantics of these fields!

I will follow up with a proposal, that I believe addresses some of these problems next.

[mattmoor]

Lifecycle of a Revision (proposed)

This proposal is intended to address some of the weaknesses above, including the race and the surfacing of “observed” serving state.

Event: First deploy

(same)

Event: Scaled to Zero

After a period of inactivity, the autoscaler puts the Revision into the “ServingState” of “ActiveReserve” (a new state).

The Route controller watches for Revisions entering the “ActiveReserve” state and updates their Ingress to go through the activator instead of referencing them directly. Once done, the Route controller puts the Revision into a “Reserve” state.

The Revision controller will take no action on entering the “ActiveReserve” state; from its perspective the Revision is still active and may receive traffic at any time. Upon entering the “Reserve” state, the Revision Controller will remove some portion of the underlying k8s resources.

At this point, we need to convey that the Revision is no longer “Ready”, so we will surface the following conditions via the Revision’s status:

status:
  conditions:
  - type: Ready
    status: False
  - type: Reserve
    status: True
    reason: NoTraffic
    message: This revision was put into a reserve state due to inactivity.

Event: Scaled from Zero

Traffic arrives for our Revision, but hits the activator. The activator looks up the Revision that should be receiving the traffic and sets its “ServingState” to “Active”.

The activator then watches the Revision, waiting for it to re-enter a Ready state, at which point the queued traffic is forwarded.

Event: Removed from Route

When the Revision is removed from the Route’s Spec, the Route will first remove the Revision’s service from its ingress rules, so that it stops receiving traffic. Then the Route puts the Revision into a “Retired” state.

At this point, we need to convey that the Revision is no longer “Ready”, so we will surface the following conditions via the Revision’s status:

status:
  conditions:
  - type: Ready
    status: False
  - type: Retired
    status: True
    reason: NotRoutable
    message: This revision is no longer reachable via a Route.

Event: Added to Route

When the Revision is added (back) to the Route’s Spec, the Route must first ensure that the Revision is “Ready”. If the Revision is not in an “Active” state, the Route will change it and wait for it to report itself “Ready”.

---

If this informal sketch makes sense, I will write it up as more official documentation including updates to the existing specification where appropriate.

cc @josephburnett @vaikas-google @evankanderson @grantr @cooperneil

[grantr]

The Route controller observes this state change, and redirects the ingress to an “Activator” component. Keying off of the same signal, the Revision controller deletes the underlying K8s resources. There is a race here.

To make sure I understand: the race is between the parallel actions of redirecting the ingress and deleting the resources. If resources are deleted before the ingress is changed, requests will be lost.

After a period of inactivity, the autoscaler puts the Revision into the “ServingState” of “ActiveReserve” (a new state).

This feels like a status to me. "ActiveReserve" is temporary, not a desired end state. To avoid hijacking this issue, I've created #669 describing an alternative approach that IMO creates a better separation between status and spec. In the meantime, I'm 👍 on the proposal here to remove the race condition.

---

🤔 What should happen for a route with a zero-percent traffic rule? For example:

spec:
  traffic:
  - revisionName: foo
    percent: 100
  - revisionName: bar
    percent: 0

Is the second traffic rule removed from the spec? Is bar considered to be unrouted? Does it go into the Retired or Reserved state?

[mattmoor]

This feels like a status to me. "ActiveReserve" is temporary, not a desired end state.

Like replicas: N on a ReplicaSet it's often both. The auto-scaler observes an idle Active revision, and configures it's desired state to be ActiveReserve (similar to lame duck). Once the activator switch is made, the Route controller sets it's desired state to Reserve, which may take time to realize.

When the Revision is ultimately removed from routing, the transition to Retired is also an expression of desired state, and the Revision controller must clean up.

[akyyy]

@mattmoor, @josephburnett

In Lifecycle of a Revision (Today),
Event: Scaled to Zero part:
"Despite the endpoint having no entries the Revision won’t leave a “Ready” state because of this logic (IIUC). No information is conveyed to indicate that the Revision has been actually scaled to zero and is unprepared to receive traffic."

This is not what we're doing now. When the k8s resources are deleted, the revision is marked with this RevisionCondition, so it's not ready.
reason: Inactive
status: "False"
type: Ready

This is the code:
https://github.com/elafros/elafros/blob/0be70191e5021bfeb8e29593490e5d0a2c24d8cc/pkg/controller/revision/revision.go#L635

[mattmoor]

thanks @akyyy.

I hadn't looked to see that we were manually updating the status. I think we still need a Ready: True => False path when our Service's Endpoint becomes empty, but that isn't necessary for auto-scaling.

[mattmoor]

I think we're punting this to get solved by some of the work @grantr and @glyn are driving, which is certainly going to extend into M5.

/assign @grantr