[1.6.1] LoadBalancer does not work with Calico or Cilium

[alexzimmer96]

I've installed a kubernetes-cluster on Hetzner-Cloud using Kubespray. I've installed the hcloud-controller-manager and the LoadBalancer resources gets deployed when i try to create one, but it seems like the traffic can't be forwarded to the applications.

I'm using:

• Kubernetes v1.18.4 installed via Kubespray
• hcloud-controller-manager v1.6.1
• Calico 3.14.1

I guess the traffic should be routed using a underlying Nodeport?
If thats true, then i assume that Calico may not be properly configured to allow this behaviour (even i've not enabled any security constraints atm)

Maybe someone here can tell if that's a bug or a configuration issue?

[acim]

Have you configured network properly?

After upgrade to 1.6.1, Cillium still works for me.

[alexzimmer96]

Okay, i have an update on this after i tried with Cilium. I've tested this with deploying a simple nginx-ingress-controller using Helm. Cloud-Controller-Manager is properly set up.

I can see that the LoadBalancer is created and can connect to the service if a use the NodePorts mentioned by the LoadBalancer.

Here is the LoadBalancer-Service Manifest (I've removed some unnecessary fields):

apiVersion: v1
kind: Service
metadata:
  annotations:
    load-balancer.hetzner.cloud/location: nbg1
    load-balancer.hetzner.cloud/name: nginx-ingress-lb
    load-balancer.hetzner.cloud/use-private-ip: "true"
  name: ingress-nginx-ingress-controller
  namespace: kube-system
spec:
  clusterIP: 10.233.56.32
  externalTrafficPolicy: Local
  healthCheckNodePort: 31134
  ports:
  - name: http
    nodePort: 31337
    port: 80
    protocol: TCP
    targetPort: http
  - name: https
    nodePort: 31719
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    app: nginx-ingress
    app.kubernetes.io/component: controller
    release: ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 167....
    - ip: 192.168.0.5

Here are some logs

I0626 22:09:25.753157       1 serving.go:313] Generated self-signed cert in-memory
W0626 22:09:26.203013       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0626 22:09:26.207971       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
W0626 22:09:26.620198       1 controllermanager.go:132] detected a cluster without a ClusterID.  A ClusterID will be required in the future.  Please tag your cluster to avoid any future issues
Hetzner Cloud k8s cloud controller v1.6.1 started
I0626 22:09:26.621026       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.621068       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0626 22:09:26.621108       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.621113       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0626 22:09:26.621270       1 secure_serving.go:178] Serving securely on [::]:10258
I0626 22:09:26.622040       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0626 22:09:26.622247       1 controllermanager.go:247] Started "service"
I0626 22:09:26.622345       1 controller.go:208] Starting service controller
I0626 22:09:26.622358       1 shared_informer.go:223] Waiting for caches to sync for service
I0626 22:09:26.707120       1 controllermanager.go:247] Started "route"
I0626 22:09:26.708629       1 route_controller.go:100] Starting route controller
I0626 22:09:26.708651       1 shared_informer.go:223] Waiting for caches to sync for route
I0626 22:09:26.710168       1 node_controller.go:110] Sending events to api server.
I0626 22:09:26.710250       1 controllermanager.go:247] Started "cloud-node"
I0626 22:09:26.711945       1 node_lifecycle_controller.go:78] Sending events to api server
I0626 22:09:26.712092       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0626 22:09:26.723993       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file 
I0626 22:09:26.724076       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file 
I0626 22:09:26.808991       1 shared_informer.go:230] Caches are synced for route 
I0626 22:09:26.822637       1 shared_informer.go:230] Caches are synced for service 
I0626 22:09:44.591416       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9147", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0626 22:09:44.605421       1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-ingress-controller" nodes=[k8s-worker-2 k8s-worker-1]
I0626 22:09:46.312257       1 load_balancer.go:360] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="ingress-nginx-ingress-controller" targetName="k8s-worker-2"
I0626 22:09:47.430045       1 load_balancer.go:360] "add target" op="hcops/LoadBalancerOps.ReconcileHCLBTargets" service="ingress-nginx-ingress-controller" targetName="k8s-worker-1"
I0626 22:09:48.593661       1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35777
I0626 22:09:49.449492       1 load_balancer.go:420] "add service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=35777
I0626 22:09:50.394766       1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35777
I0626 22:09:50.562668       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9147", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer
I0626 22:09:50.601969       1 load_balancers.go:81] "ensure Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" service="ingress-nginx-ingress-controller" nodes=[k8s-worker-2 k8s-worker-1]
I0626 22:09:50.602126       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9195", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
I0626 22:09:50.727823       1 load_balancer.go:409] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=80 loadBalancerID=35777
I0626 22:09:51.687194       1 load_balancer.go:409] "update service" op="hcops/LoadBalancerOps.ReconcileHCLBServices" port=443 loadBalancerID=35777
I0626 22:09:52.624144       1 load_balancers.go:117] "reload HC Load Balancer" op="hcloud/loadBalancers.EnsureLoadBalancer" loadBalancerID=35777
I0626 22:09:52.768408       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"ingress-nginx-ingress-controller", UID:"87539d5a-e9ae-420b-bc95-56927466dab6", APIVersion:"v1", ResourceVersion:"9195", FieldPath:""}): type: 'Normal' reason: 'EnsuredLoadBalancer' Ensured load balancer

[alexzimmer96]

Have you configured network properly?

After upgrade to 1.6.1, Cillium still works for me.

Hi @acim
It seems properly configured. The routes have been created. Its cut out of the log above because i "restarted" the hcloud-controller-manager for a clear log output.

[LKaemmerling]

@alexzimmer96 does everything work for you now? Actually i can't see a problem based on the logs. Does the health check work (visible within the cloud console)?

[alexzimmer96]

@LKaemmerling unfortunately i deleted the original cluster to try other installation methods and network plugins beside Calico und Cillium. But indeed, the health checks did not succeed. Although i was able to create a connection using the Ports that were specified as the LoadBalancers target, if used the public ip of one any node.

[MatthiasLohr]

I think I'm hitting a similar issue and did take a deeper look into it. Actually, the issue seems to be "quite known" in the Kubernetes community (metallb/metallb#153, kubernetes/kubernetes#79976, kubernetes/kubernetes#66607, kubernetes/kubernetes#92312, kubernetes/enhancements#1392, kubernetes/kubernetes#79783, kubernetes/kubernetes#59976).

TLDR: I think the problem is the following:

Using hcloud-cloud-controller-manager, LoadBalancer services get to know their external IPs. This IP gets added to the ipvs0 interface to allow cluster-internal access to the LoadBalancer. Also, a route will be created pointing to this IP on all nodes (ip route show table local). If now the (Hetzner) Load Balancer tries to send a health check packet, the cluster's reply will stay within the cluster, since the route is pointing to the ipvs0 interface instead of the internal network's network card.

There are a lot solutions in discussion, but as far as I know, nothing helpful so far. The only workaround seems to be to use iptables instead of ipvs as kube_proxy mode (didn't try yet with Hetzner Load Balancer). However, this will come with a drawback regarding performance (https://www.projectcalico.org/comparing-kube-proxy-modes-iptables-or-ipvs/).
As a very dirty hack and experiment, I temporarily removed the local route (ip route del local $internal_loadbalancer_ip dev kube-ipvs0 table local) and health checks started to lighten in green immediatly. However, this ugly workaround will not survive a reboot.

Currently reading about stuff that it might be possbile to replace kube_proxy/ipvs with cilium, but just started with trying to understand things there...For now, I guess, only iptables will "work". But I'm happy to discuss and work with you and Hetzner staff to find a solution.

[MatthiasLohr]

Just found this: https://docs.cilium.io/en/v1.8/gettingstarted/kubeproxy-free/

[acim]

I forgot to mention that I use Cilium without kubeproxy.

helm install cilium cilium/cilium --namespace kube-system \
  --set global.kubeProxyReplacement=strict \
  --set global.k8sServiceHost=192.168.x.y \
  --set global.k8sServicePort=6443

192.168.x.y = private IP of master

[LKaemmerling]

You might need to set the hostname annotation https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/internal/annotation/load_balancer.go#L43 on the LoadBalancer service to make the k8s proxy routing the data correctly. That is a known bug within k8s (but the k8s networking guys don't think it is a bug...).

[github-actions]

This issue has been marked as stale because it has not had recent activity. The bot will close the issue if no further action occurs.