Sysbox kubernetes install fails behind HTTPS Proxy

[FFock]

When installing sysbox via kubernetes (in a Rancher 2.6 downstream cluster with k8s 1.21.10) behind a Internet HTTPs proxy following the instructions on https://github.com/nestybox/sysbox/blob/master/docs/user-guide/install-k8s.md using Ubuntu 20.04 (latest) as node OS all pods on the node(s) where sysbox should be installed report the following issue during "container creation" in Rancher 2.6:

Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_cattle-node-agent-5mdcl_cattle-system_534ba7cd-b43f-4911-a1a6-4346e0d75d06_0": Error initializing source docker://k8s.gcr.io/pause:3.5: error pinging docker registry k8s.gcr.io: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 127.0.0.53:53: server misbehaving
Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_coredns-685d6d555d-wbm64_kube-system_8b8b549e-52e1-44ff-b825-f7c6eee340a4_0": Error initializing source docker://k8s.gcr.io/pause:3.5: error pinging docker registry k8s.gcr.io: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 127.0.0.53:53: server misbehaving

Is there a workaround to specify proxy information (HTTP_PROXY, HTTPS_PROXY, NO_PROXY) to the installer (install.yml) or is this setup not supported at all if located behind an Internet proxy?

(Searched the Web but did not find a single hint so far)

[rodnymolina]

@FFock, thanks for filing this issue. I would need to reproduce this one locally to identify the fix, but at first glance, I see room for the behavior that you're commenting on. Basically, this would require an extension to our sysbox-deploy daemonset. Will look at it when have a chance.

[FFock]

Meanwhile I tried to find a workaround, but failed so far.

Our Rancher uses RKEv1 to deploy the downstream kubernetes cluster, which should get sysbox enabled. I tried to define the proxy settings in /etc/default/docker but this seems to be no longer supported by docker 20.10.7 or it newer supported docker API calls which are used by Rancher RKE.

So far, it seems that it would be sufficient to apply proxy environment variables (HTTPS_PROXY and NO_PROXY) to the kubelet container. However, this seems to be more difficult in this combination as expected.

Please note that the docker daemon is well configured to use our Internet proxy to download docker images. That is working fine. The sysbox configuration is trying to download something from the Internet (k8s.gcr.io) directly from the kubelet container which finally affects all containers on the sysbox enabled node(s).

[rodnymolina]

By looking at your initial logs, it looks like kubelet is struggling to find the pause image, which in cri-o's setups is typically configured within its default config file: /etc/crio/crio.conf.

If that's the case, then cri-o may be unaware of your proxy settings, so as a potential workaround I would try to adjust your cri-o service file as indicated below:

/etc/systemd/system/crio.service

[Service]
Environment=”HTTP_PROXY=http://proxy_ip:proxy_port"
Environment=”HTTPS_PROXY=http://proxy_ip:proxy_port"

Digest systemd changes and restart cri-o:

$ sudo systemctl daemon-reload
$ sudo systemctl restart crio

Let me know if that works.

[FFock]

I created the /etc/systemd/system/crio.service file with the config above (adapted of course), but the error is still the same:

Failed to create pod sandbox: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_pushprox-kube-controller-manager-client-cprjg_cattle-monitoring-system_048d41c7-d208-43d2-969b-b060103b25df_0": Error initializing source docker://k8s.gcr.io/pause:3.5: error pinging docker registry k8s.gcr.io: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 127.0.0.53:53: server misbehaving

[FFock]

Pulling the k8s.gcr.io/pause:3.5 image on the node using docker pull does not fix the issue too. Is this initializing code downloading the image in a docker-in-docker environment?

[FFock]

The /etc/crio/crion.conf file is:

[crio]
  storage_driver = "overlay"
  storage_option = ["overlay.mountopt=metacopy=on"]

  [crio.network]
    plugin_dirs = ["/opt/cni/bin", "/home/kubernetes/bin"]

  [crio.runtime]
    cgroup_manager = "cgroupfs"
    conmon_cgroup = "pod"
    default_capabilities = ["CHOWN", "DAC_OVERRIDE", "FSETID", "FOWNER", "SETUID", "SETGID", "SETPCAP", "SETFCAP", "NET_BIND_SERVICE", "KILL", "AUDIT_WRITE", "NET_RAW", "SYS_CHROOT", "MKNOD"]
    pids_limit = 16384

    [crio.runtime.runtimes]

      [crio.runtime.runtimes.sysbox-runc]
        allowed_annotations = ["io.kubernetes.cri-o.userns-mode"]
        runtime_path = "/usr/bin/sysbox-runc"
        runtime_type = "oci"

[rodnymolina]

Pulling the k8s.gcr.io/pause:3.5 image on the node using docker pull does not fix the issue too. Is this initializing code downloading the image in a docker-in-docker environment?

During installation, sysbox-k8s-deploy daemonset replaces docker-shim as the CRI utilized by the rke's kubelet component. That's to say that all the k8s control-plane components are re-instantiated through cri-o, with the exception of kubelet itself which continues to operate within a regular docker container.

Now, notice that the cri-o daemon being installed runs at the host level, so it could very well be that kubelet is fully aware of your http-proxy settings (as defined through dockerd config), while cri-o process is not. That's why i believe that we need to extend our sysbox-k8s-deploy daemonset to parse/extract the proper http-proxy setting from the original kubelet (or dockerd) config, and generate a crio.conf file (or containers.conf) that reflects the proper http-proxy configuration.

Coming back to your point/question above. It makes sense to me that problem is not fixed when you fetch the pause image through a docker pull instruction, as all you're doing there is to store this image in docker's image-layer caches. This won't help those pods that cri-o is attempting to launch. Instead, you can rely on the crictl command to do something equivalent from any of your affected k8s/rke nodes:

$ sudo crictl pull -n <workspace> k8s.gcr.io/pause:3.5

I'll have more details once that I have cycles to recreate the issue.

[FFock]

Ok, thank you for explaining the background. Things are getting clearer now. The problem can be indeed reproduced quite simple:

sudo crictl pull k8s.gcr.io/pause:3.5
FATA[0000] pulling image: rpc error: code = Unknown desc = error pinging docker registry k8s.gcr.io: Get "https://k8s.gcr.io/v2/": dial tcp: lookup k8s.gcr.io on 127.0.0.53:53: server misbehaving

Thus, I will now try to get the image pulled by crictl using some proxy config for it (hopefully this can be done somehow...).

[rodnymolina]

Haven't looked in details but i believe the proper config file is containers.conf. You will need to add something like this within the "engine" section and then restart crio:

...
[engine]
env = [
     "http_proxy=http://1.2.3.4:5678",
     "https_proxy=http://1.2.3.4:5678",
     "no_proxy=localhost,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16",
   ]
...

[FFock]

Problem solved (with workaround):
Adding these two lines to the [Service] section of /etc/systemd/system/crio.service fixes the issue*:
*NO_PROXY might be improved with CIDR notation - I do not know yet, if crio supports it.

Environment="HTTP_PROXY=http://\<proxy\>:\<port>" "NO_PROXY=0,1,2,3,4,5,6,7,8,9,.svc,.cluster.local,localhost"
Environment="HTTPS_PROXY=http://\<proxy\>:\<port>" "NO_PROXY=0,1,2,3,4,5,6,7,8,9,.svc,.cluster.local,localhost"

To activate these settings do:

$ sudo systemctl daemon-reload
$ sudo systemctl restart crio

To verify that pulling now works:

$ sudo crictl pull k8s.gcr.io/pause:3.5
Image is up to date for k8s.gcr.io/pause@sha256:1ff6c18fbef2045af6b9c16bf034cc421a29027b800e4f9b68ae9b1cb3e9ae07

[rodnymolina]

Excellent @FFock. Let's leave this issue open till we implement a proper solution in our sysbox-k8s-deploy daemonset.

[fhaefemeier]

There is a similar issue in a non-k8 (simple docker) envionment. The workaround is the same. Adding the proxy parameter to the daemon or in the config file (~/.docker/config.json) in my home directory (docker manual). But it is a workaround, because it enables the proxy globally for each docker container, which is not the intention.