lo interface wasn't set up properly when using initrd

[liwei]

Description of problem

After changing from image to initrd, lo interface wasn't set up properly.

edit /etc/kata-containers/configuration.toml

#image = "/usr/share/kata-containers/kata-containers.img"
initrd = "/usr/share/kata-containers/kata-containers-initrd.img"

docker run --runtime=kata --rm -ti busybox

/ # ip link
1: lo: <LOOPBACK> mtu 65536 qdisc noop qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

/ # ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq qlen 1000
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:acff:fe11:2/64 scope link
       valid_lft forever preferred_lft forever
3: sit0@NONE: <NOARP> mtu 1480 qdisc noop qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0

Expected result

lo brought up and have IP address

Actual result

lo is down and without IP address

[jodh-intel]

Thanks for raising @liwei.

@GabyCT - could you create a test for this?

[jodh-intel]

I've take a brief look at this and the agent is segfaulting when run as init:

time="2018-06-07T11:29:23.436662457+01:00" level=info msg="time=\"2018-06-07T10:29:23.417013954Z\" level=info msg=\"Getting interface from MAC address\" interface-name=eth0 mac-address=\"02:42:ac:11:00:02\" name=kata-agent pid=1 source=agent\n" name=kata-proxy pid=13000 source=agent
time="2018-06-07T11:29:23.438540332+01:00" level=info msg="[    0.550702] init[65]: segfault at 0 ip 0000000000945b6a sp 000000c42015a9a8 error 4 in init[400000+acf000]\n" name=kata-proxy pid=13000 source=agent
time="2018-06-07T11:29:23.438619968+01:00" level=info msg="time=\"2018-06-07T10:29:23.418666341Z\" level=info msg=\"Link found\" interface-name=eth0 link=\"&{LinkAttrs:{Index:2 MTU:1500 TxQLen:1000 Name:eth0 HardwareAddr:02:42:ac:11:00:02 Flags:broadcast|multicast RawFlags:4098 ParentIndex:0 MasterIndex:0 Namespace:<nil> Alias: Statistics:0xc42019d5d4 Promisc:0 Xdp:0xc42006d5e0 EncapType:ether Protinfo:<nil> OperState:down NetNsID:0 NumTxQueues:0 NumRxQueues:0}}\" mac-address=\"02:42:ac:11:00:02\" name=kata-agent pid=1 source=agent\n" name=kata-proxy pid=13000 source=agent

I don't think the logs either side of the segfault are related as eth0 does come up. After unpacking the initrd:

$ addr2line -e ./init 0000000000945b6a
/home/jcvenega/go/src/github.com/kata-containers/agent/vendor/github.com/vishvananda/netlink/protinfo.go:22

That points at https://github.com/vishvananda/netlink/blob/master/protinfo.go#L22 which suggests
a nil Protinfo object is responsible for the crash.

I've added in a bunch of debug to the code, but have been scratching my head for a while trying to work out why my custom debug agent wasn't working. The proxy log showed the kernel reported

Failed to execute /init (error -2)

... and yet the binary was there. After chroot-ing into the rootfs I see the problem:

# ldd /init
        /lib64/ld-linux-x86-64.so.2 (0x7f410397e000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f410397e000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f410397e000)
        libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f410397e000)
Error relocating /init: __snprintf_chk: symbol not found
Error relocating /init: __vfprintf_chk: symbol not found
Error relocating /init: __vsnprintf_chk: symbol not found
Error relocating /init: __longjmp_chk: symbol not found
Error relocating /init: __fprintf_chk: symbol not found

I've tried to build the agent as a static binary, but it no luck so far:

$ go build -o kata-agent -a --ldflags '-linkmode external -extldflags "-static"'
$ CGO_ENABLED=0 go build -o kata-agent -a --ldflags '-linkmode external -extldflags "-static"'
agent.go:32:2: build constraints exclude all Go files in /home/james/go/src/github.com/kata-containers/agent/vendor/github.com/opencontainers/runc/libcontainer/nsenter

@bergwolf - any suggestions here?

[bergwolf]

@jodh-intel the kata-agent binary for alpine image is using libc.musl-x86_64.so.1 instead of libc.so. You need to build it in an alpine environment (most importantly using the alpine version of go). That is why your custom kata agent binary cannot start.

[grahamwhaley]

and I think you can get a golang alpine env from a docker image here: https://hub.docker.com/_/golang/ @jodh-intel (I was looking around to see how we could build without the snprintf chk types etc.)

[jodh-intel]

Ta. I was able to build my version of the agent using osbuilder with a minor tweak (PR to follow...).

I now get:

runtime: goroutine stack exceeds 1000000000-byte limit
fatal error: stack overflow

That would look pretty good on a T-shirt I reckon 😄

[jodh-intel]

Paring back the debug a bit...

[jodh-intel]

This bug is a royal pita. Adding debug to the netlink library is guaranteed to blow the stack for me, regardless of how big I make it it seems.

I've re-vendored netlink but that didn't help. I can see what I think is the problem: ProtInfo for lo is nil, I just don't know why as I can't debug the library itself.

What I have learnt is that the problem is specific to the agent running as PID 1 - that is, we can rule out the (alpine) rootfs as: I built the agent-as-init in a centos rootfs (required some hacking) and, like we see in alpine with agent-as-init, lo is not "up".

[bergwolf]

Fixed by kata-containers/agent/pull/266