Skip to content

Unstable node IP in a multi NIC setup #573

Closed
Closed
Unstable node IP in a multi NIC setup#573
Labels
kind/bugCategorizes issue or PR as related to a bug.
@brtkwr

Description

@brtkwr
brtkwr
opened on Mar 29, 2019

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug

/kind feature

What happened:

When there is more than one NIC attached to an instance, openstack cloud provider returns a random InternalIP back to the host resulting in instability with API server which only talks to a default interface. Additionally, when a --node-ip arg is provided to kubelet with the intended node IP, this is not respected.

What you expected to happen:

Consistently return the default interface IP.

How to reproduce it (as minimally and precisely as possible):

Create a cluster with multiple NICs with --cloud-provider=openstack arg or --cloud-provider=external with OpenStack cloud provider pod running on kube-system.

Anything else we need to know?:

Environment:

  • openstack-cloud-controller-manager version: latest
  • OS (e.g. from /etc/os-release): Fedora Atomic 27
  • Kernel (e.g. uname -a): Linux k8s-demo-n7mwfmzoeg2q-minion-0.novalocal 4.16.12-200.fc27.x86_64
  • Install tools: OpenStack Magnum
  • Others: NA

Here's what it looks like when I watch my nodes:

(venv-openstack) ➜  ~ kubectl get nodes -o wide --watch
NAME                             STATUS   ROLES    AGE   VERSION   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                       KERNEL-VERSION            CONTAINER-RUNTIME
k8s-demo-n7mwfmzoeg2q-minion-0   Ready    <none>   10d   v1.13.4   10.60.253.61   <none>        Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.1.0.16   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.60.253.61   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1
k8s-demo-n7mwfmzoeg2q-minion-0   Ready   <none>   10d   v1.13.4   10.3.0.22   <none>   Debian GNU/Linux 9 (stretch)   4.16.12-200.fc27.x86_64   docker://18.3.1

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions