Generic kubelet config interface

[jackfrancis]

What this PR does / why we need it: exposes a generic key/val interface for configuring kubelet, using the configuration keys documented here:

https://kubernetes.io/docs/reference/generated/kubelet/

• Deprecated appropriate kubernetesConfig properties in favor of using the generic kubelet interface (i.e., if a kubernetesConfig property is basically a thin wrapper around kubelet runtime config, we deprecated; if the kubernetesConfig property is a higher order configuration declaration that includes kubelet runtime config changes as a side-effect, we retained the kubernetesConfig property and made the relevant kubelet config property read-only.
  • deprecated nodeStatusUpdateFrequency
  • deprecated hardEvictionThreshold
  • deprecated nonMasqueradeCidr

Release note:

generic kubelet config interface

[jackfrancis]

@jchauncey and @CecileRobertMichon please take a look at this approach and weigh in on whether or not this is an improvement. I've made a guinea pig of the DNSServiceIP property of KubernetesConfig: it's been removed in favor of a reserved "--cluster-dns" key in the new, proposed KubernetesConfig.KubeletConfig generic configuration object.

The real proposal here is that we settle on a common, thin interface to expose all Kubernetes component runtime config, and then move all existing one-off config vectors into an organized place.

For example:

"kubernetesConfig": {
    "kubeletConfig": {
        "--cluster-dns": "172.17.1.11",
        "--max-pods": "30",
        <etc...>
    },
    "apiserverConfig": {
        "--storage-backend": "etcd3",
        <etc...>
    }
}

[jchauncey]

i like it. 👍

[CecileRobertMichon]

👍👍 I'm definitely in favor of decluttering KubernetesConfig

[CecileRobertMichon]

Out of curiosity, what's the reasoning behind naming this "--cluster-dns" vs "cluster-dns" since in this context it's not used as an arg but as a json string?

[jackfrancis]

Good question. The thinking is to transparently use the exact string that we'll be passing to the component at runtime, so that we may eventually just pass it on as-is. Also, -- as a prefix might not be generally valid going forward, it's possible some options might have a - prefix (or some other prefix).

[CecileRobertMichon]

Is KubernetesConfig part of both OrchestratorProfile and Master+Agent Profile?

[jackfrancis]

For the purposes of this PR, this is a change that enables master-specific and agent-specific kubelet configs (as opposed to using a common one).

In a longer-term sense, adding KubernetesConfig to the master/agents separately means that we can do k8s configuration generally on a per-master and per-agent node basis.

I couldn't find a more elegant way to do this from a kubelet config perspective, but I'm open to ideas.

[CecileRobertMichon]

Ok that makes sense. What about removing KubernetesConfig from OrchestratorProfile? In that case KubernetesConfig would act like vmSize where masters and agents have their own and in the case of a common one it is duplicated in the master and agent profile.

[jackfrancis]

Eventually we may do that, but that's pretty destructive (everyone's api models out there in the world refer to it). My thoughts for this are:

• OrchestratorProfile.KubernetesConfig = common (all Azure resourses) Kubernetes-related config
• MasterProfile.KubernetesConfig = Kubernetes-related config for master nodes
• AgentPoolProfile.KubernetesConfig = Kubernetes-related config for agent nodes in an agent pool

[CecileRobertMichon]

So what happens if I use the following api model?

{
"apiVersion": "vlabs",
"properties": {
"orchestratorProfile": {
"orchestratorType": "Kubernetes",
"kubernetesConfig": {
"maxPods": "110"
}
},
"masterProfile": {
"count": 1,
"dnsPrefix": "",
"vmSize": "Standard_D2_v2",
"kubernetesConfig": {
"maxPods": "220"
}
},
"agentPoolProfiles": [
{
"name": "agentpool1",
"count": 3,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet",
"kubernetesConfig": {
"maxPods": "220"
}
}
],
...
}

[jackfrancis]

At present, nothing. The operable changes are here:

https://github.com/Azure/acs-engine/pull/1854/files#diff-8edb2653431a5da9ad1f4286097702a7R479

That copies over the currently supported KubernetesConfig (from OrchestratorProfile) so that it can be consumed thusly:

by the agents:

• https://github.com/Azure/acs-engine/pull/1854/files#diff-7d5f50f227edfc447ff04539035aa614R106
  by the masters:
• https://github.com/Azure/acs-engine/pull/1854/files#diff-91eb315fd90ae6b7177cd6290a360600R235

[croomes]

Just want to jump in with a 👍

The use case we have is to allow setting feature gates, where an administrator can add (in our case) --feature-gates MountPropagation=true to both the kubelet and api-server. This is currently a requirement for storage platforms running within Kubernetes, to allow volumes to be exposed to other containers.

Was happy to see separate configs for kubelet and api-server in the example, not just the kubelet as the issue title suggests.

[jackfrancis]

@croomes Yes, the example is a sort of high-level expression of why we're doing this with kubelet.

To be clear, this PR is for kubelet only, but I'm aiming to refer to it as a model for api server and controller manager generic config implementations as well, hopefully in the next few weeks.

[CecileRobertMichon]

lgtm

[CecileRobertMichon]

This can move up to configurable

[jackfrancis]

good catch!

[jackfrancis]

Oh, actually: "either because a higher order configuration vector is available that enforces kubelet configuration". In this case we're calling it static because it is overridable by useCloudControllerManager. So I think we want to keep it here in terms of documentation (in other words, we don't want to encourage folks to modify --cloud-provider, we want them instead to set useCloudControllerManager to true to bypass azure cloud-provider implementation.