Run kubelet outside container

[CecileRobertMichon]

What this PR does / why we need it: In current (v0.14.6) acs-engine implementation, kubelet starts in a Docker container.
There are a lot of regressions of containerized kubelet reported in Kubernetes v1.10 (kubernetes/kubernetes#61456) . Since there are no full e2e tests for this, it may also happen in future releases (thanks @feiskyer for reporting). This PR moves kubelet to run outside the container, which is expected to be more stable.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

If applicable:

• documentation
• unit tests
• tested backward compatibility (ie. deploy with previous version, upgrade with this branch)

Release note:

[CecileRobertMichon]

TODO: adapt kubelet start for coreos filepath

[jackfrancis]

We can shorten this to /bin/chmod a+x /usr/local/bin/kubelet /usr/local/bin/kubectl (the build of chmod on ubuntu hosts accepts a list of file targets to apply the mode changes against). Verified:

azureuser@k8s-master-18440544-0:/tmp$ ls -la test*
-rw-rw-r-- 1 azureuser azureuser 0 Apr  3 23:08 test1
-rw-rw-r-- 1 azureuser azureuser 0 Apr  3 23:08 test2
azureuser@k8s-master-18440544-0:/tmp$ /bin/chmod a+x test1 test2
azureuser@k8s-master-18440544-0:/tmp$ ls -la test*
-rwxrwxr-x 1 azureuser azureuser 0 Apr  3 23:08 test1
-rwxrwxr-x 1 azureuser azureuser 0 Apr  3 23:08 test2

[CecileRobertMichon]

Yes good catch, thanks

[jackfrancis]

See below. Didn't verify on a coreos host, but I'm confident this is a standard pattern.

[jackfrancis]

ditto above about chmod command

[jackfrancis]

ditto

[jackfrancis]

In the systemd extract definition, I think we want a ConditionPathExists=!/usr/local/bin/kubelet as well. (and the CoreOS flavor also)

[jackfrancis]

Let's change the directory name kubectldir (under /tmp on the host OS and /opt inside the container) to something more general, like hyperkubedir or something.

[jackfrancis]

Let's do this in one line to make it a little clearer that we are copying the same file to two different places. One example:

for dest in /usr/local/bin/kubelet /usr/local/bin/kubectl ; do cp /tmp/kubectldir/hyperkube $dest; done

And then just add a cleanup line after, like:

rm -Rf /tmp/kubectldir

(but replace kubectldir with the new, general name in both instances :) )

[CecileRobertMichon]

I don't really find that

ExecStartPre=for dest in /usr/local/bin/kubelet /usr/local/bin/kubectl ; do /bin/cp /tmp/hyperkube/hyperkube $dest; done
ExecStartPre=rm -Rf /tmp/kubectldir

is much more readable than

ExecStartPre=/bin/cp /tmp/hyperkubedir/hyperkube /usr/local/bin/kubelet
ExecStartPre=/bin/mv /tmp/hyperkubedir/hyperkube /usr/local/bin/kubectl

How about:

ExecStartPre=/bin/mv /tmp/hyperkubedir/hyperkube /usr/local/bin/kubelet
ExecStartPre=/bin/cp /usr/local/bin/kubelet /usr/local/bin/kubectl

?

[CecileRobertMichon]

I mean I see where you're coming from in terms of logic, I just find that it makes the syntax more complex to have a for loop for only two files. Please feel free to disagree with me :)

[CecileRobertMichon]

@jackfrancis re: In the systemd extract definition, I think we want a ConditionPathExists=!/usr/local/bin/kubelet as well. (and the CoreOS flavor also) : I thought about it when I was writing the code but I didn't add it because we want the condition to be: run if either file is missing. A ConditionPathExists will prevent the extract service from running if any of the two files is present (&&, not ||) since both conditions will need to be satisfied. But what happens if the first file gets copied but the second one doesn't (unlikely, but software is unpredictable)? we still want to run the extraction again (read retry) in that case, so I left the condition as !/usr/local/bin/kubectl since /usr/local/bin/kubectl gets created last. Thoughts?

[feiskyer]

Besides the extraction, dependencies should also be installed. e.g. on ubuntu, kubelet depends on iptables (>= 1.4.21), kubernetes-cni (= 0.6.0), iproute2, socat, util-linux, mount, ebtables, ethtool, init-system-helpers (>= 1.18~)

[CecileRobertMichon]

Thanks @feiskyer. Just checked and all of these are already being installed except for socat. I will add a step to install them if they are missing so we are safe. I can't find a package kubernetes-cni however, do you have more info on that one?

[CecileRobertMichon]

For the cni plugin, I think we are already installing it at

acs-engine/parts/k8s/kubernetesmastercustomscript.sh

Line 217 in b463cd0

retrycmd_get_tarball 60 1 $CONTAINERNETWORKING_CNI_TGZ_TMP ${CNI_PLUGINS_URL}

[CecileRobertMichon]

Removed the 1.5 version since k8s 1.5 was deprecated in #2394 and it's not in use anymore

[CecileRobertMichon]

Ensure dependencies are installed on agents

[CecileRobertMichon]

Ensure dependencies are installed on masters

[jackfrancis]

Are we confident that all of these packages will reliably install in under 2 mins? (speaking to the 120 second timeout)

[CecileRobertMichon]

Yes most of them should already be there so it's very quick. In my tests it took under 10 seconds. I can increase the timeout if we want to be sure.

[jackfrancis]

#2596 :)

[CecileRobertMichon]

Oh yeah I just moved it out because I moved out apt_get_update so I thought might as well but we should generalize it in another PR

[CecileRobertMichon]

I increased it to 5 minutes... Should be plenty enough

[jackfrancis]

lgtm if we feel good about the 2 minute timeout for apt-get install for kubelet deps

[jackfrancis]

lgtm