Replace single tenant autoscaler with multitenant autoscaler

[glyn]

Before this can happen the following tasks need to be completed:

• Check that the multitenant autoscaler behaves correctly
• Implement dynamic reconfiguration for the multitenant autoscaler: Multitenant autoscaler dynamic reconfiguration #1658
• Draft a design for removing scalability restrictions from the multitenant autoscaler
• Verify multitenant autoscaler metrics show up in dashboard
• Enable the multitenant autoscaler by default
• Delete all code supporting the single tenant autoscaler, except that which supports the multitenant autoscaler too
• Update documentation

Scalability requirements for the multitenant autoscaler include:

1. The number of revisions must be able to exceed the available number of ports of a single instance of the multitenant autoscaler (e.g. for websocket connections used to push metrics from queue proxies). Solution may include pulling metrics, e.g. from Envoy, using short-lived connections, and using multiple instances of the multitenant autoscaler to increase the number of available ports.
2. The multitenant autoscaler must be able to consume revision metrics sufficiently fast that it does not "get behind". Solutions include increasing CPU and maybe other resources for the multitenant autoscaler and using multiple instances.
3. The multitenant autoscaler must not "get behind" in taking scaling decisions. The time taken increases linearly with the number of revisions. Solutions include increasing CPU and maybe other resources for the multitenant autoscaler and using multiple instances.
4. What other requirements need to be considered?

Expected Behavior

The multitenant autoscaler is enabled by default and the single tenant autoscaler is deleted and no longer available for use.

Actual Behavior

Both types of autoscaler are available for use and the single tenant autoscaler is enabled by default.

Steps to Reproduce the Problem

1. Deploy and invoke a configuration.
2. Observe an instance of the single tenant autoscaler running.

Additional Info

/area autoscale
/kind dev
/assign @josephburnett
/assign @glyn

[glyn]

Hmmm. Not sure why prow is ignoring the /assign commands?

[mattmoor]

That is weird
/assign @josephburnett
/assign @glyn

[glyn]

@josephburnett I'd appreciate your thoughts on this issue.

[jchesterpivotal]

The multitenant autoscaler must be able to consume revision metrics sufficiently fast that it does not "get behind".

Dropping metrics should be fine, I think. We're usually most interested in the latest condition and if the multiscaler is overwhelmed then I expect any integral term is being drowned out by the proportional and derivate terms.

Though this suggests a problem of quis custodiet ipsos custodes: what scales the multiscaler?

[glyn]

Hmmm. Dropping metrics might not be fine if we are unfortunate enough not to retain any metrics for a particular revision. We would then be scaling that revision "in the dark".

In terms of what scales the multiscaler, we might be able to get away with a single process for that.

[jchesterpivotal]

The "no metrics" case suggests to me that we will still need a metric to indicate 0 activity. Otherwise we won't be able to tell if we're seeing nothing because there's nothing to do or because something is broken. In the latter case I'd expect folks would have alerts raised from their observation tools of choice.

[glyn]

I feel the scalability of the multitenant autoscaler has been "socialised" sufficiently here and on slack. We might not have all the answers, but at least we have aired the questions. Checking/ticking off that item above.

[glyn]

@josephburnett Now that #1809 is merged, I'm deferring to you (as the owner of autoscaling) to switch to the shared autoscaler and rip out the single tenant only code. Ok?

[josephburnett]

@glyn, sounds good.

[markusthoemmes]

@josephburnett is this done per #1854?