Sandbox can't be launched when memory exceed 120Gi in centos

[wenlxie]

Description of problem

I am now integrating kata runtime with k8s.
In the test environment, I found that my pod can't become running because sandbox can't be launched.
When the container's request memory is 120Gi +, then we can see this issue.
But when the memory is lower than 120Gi, then it works fine.

Expected result

The sandbox can launched

Actual result

It fails.

---

(replace this text with the output of the kata-collect-data.sh script, after
you have reviewed its content to ensure it does not contain any private
information).

Meta details

Running kata-collect-data.sh version 1.5.0 (commit 5f7fcd773089a615b776862f92217e987f06df0a) at 2019-02-15.19:36:43.520404111-0700.

---

Runtime is /usr/local/bin/kata-runtime.

kata-env

Output of "/usr/local/bin/kata-runtime kata-env":

[Meta]
  Version = "1.0.20"

[Runtime]
  Debug = false
  Trace = false
  DisableGuestSeccomp = false
  DisableNewNetNs = false
  Path = "/var/usrlocal/bin/kata-runtime"
  [Runtime.Version]
    Semver = "1.5.0"
    Commit = "5f7fcd773089a615b776862f92217e987f06df0a"
    OCI = "1.0.1-dev"
  [Runtime.Config]
    Path = "/etc/kata-containers/configuration.toml"

[Hypervisor]
  MachineType = "pc"
  Version = "QEMU emulator version 2.11.2(kata-static)\nCopyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers"
  Path = "/var/usrlocal/kata/opt/kata/bin/qemu-system-x86_64"
  BlockDeviceDriver = "virtio-scsi"
  EntropySource = "/dev/urandom"
  Msize9p = 8192
  MemorySlots = 10
  Debug = false
  UseVSock = false

[Image]
  Path = "/var/usrlocal/kata/opt/kata/share/kata-containers/kata-containers-centos.img"

[Kernel]
  Path = "/var/usrlocal/kata/opt/kata/share/kata-containers/vmlinuz-4.14.67-22"
  Parameters = "init=/usr/lib/systemd/systemd systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket"

[Initrd]
  Path = ""

[Proxy]
  Type = "kataProxy"
  Version = "kata-proxy version 1.5.0-9e77a0b1925d1ab3afda7d46ec256be8a9cff222"
  Path = "/usr/local/kata/opt/kata/libexec/kata-containers/kata-proxy"
  Debug = false

[Shim]
  Type = "kataShim"
  Version = "kata-shim version 1.5.0-efbf3bb25065ce89099630b753d218cdc678e758"
  Path = "/var/usrlocal/kata/opt/kata/libexec/kata-containers/kata-shim"
  Debug = false

[Agent]
  Type = "kata"

[Host]
  Kernel = "4.15.0-43.generic.x86_64"
  Architecture = "amd64"
  VMContainerCapable = true
  SupportVSocks = true
  [Host.Distro]
    Name = "CentOS Linux"
    Version = "7"
  [Host.CPU]
    Vendor = "GenuineIntel"
    Model = "Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz"

[Netmon]
  Version = "kata-netmon version 1.5.0"
  Path = "/usr/local/kata/opt/kata/libexec/kata-containers/kata-netmon"
  Debug = false
  Enable = false

---

Runtime config files

Runtime default config files

/etc/kata-containers/configuration.toml
/usr/share/defaults/kata-containers/configuration.toml

Runtime config file contents

Output of "cat "/etc/kata-containers/configuration.toml"":

[hypervisor.qemu]
path = "/usr/local/kata/opt/kata/bin/qemu-system-x86_64"
kernel = "/usr/local/kata/opt/kata/share/kata-containers/vmlinuz.container"
image = "/usr/local/kata/opt/kata/share/kata-containers/kata-containers.img"
machine_type = "pc"
kernel_params = ""
firmware = ""
machine_accelerators=""
default_vcpus = 1
default_maxvcpus = 0
default_bridges = 1
#memory_slots = 10
disable_block_device_use = false

block_device_driver = "virtio-scsi"

enable_iothreads = false

#enable_mem_prealloc = true

#enable_hugepages = true

#enable_swap = true

#enable_debug = true

#disable_nesting_checks = true

#msize_9p = 8192

#use_vsock = true

#hotplug_vfio_on_root_bus = true

#disable_vhost_net = true

#entropy_source= "/dev/urandom"

#guest_hook_path = "/usr/share/oci/hooks"

#enable_template = true

[proxy.kata]
path = "/usr/local/kata/opt/kata/libexec/kata-containers/kata-proxy"

#enable_debug = true

[shim.kata]
path = "/usr/local/kata/opt/kata/libexec/kata-containers/kata-shim"


#enable_debug = true

[agent.kata]

[netmon]
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/usr/local/kata/opt/kata/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
internetworking_model="macvtap"
#disable_new_netns = true

Output of "cat "/opt/kata/share/defaults/kata-containers/configuration.toml"":

# Copyright (c) 2017-2019 Intel Corporation
#
# SPDX-License-Identifier: Apache-2.0
#

# XXX: WARNING: this file is auto-generated.
# XXX:
# XXX: Source file: "cli/config/configuration-qemu.toml.in"
# XXX: Project:
# XXX:   Name: Kata Containers
# XXX:   Type: kata

[hypervisor.qemu]
path = "/opt/kata/bin/qemu-system-x86_64"
kernel = "/opt/kata/share/kata-containers/vmlinuz.container"
image = "/opt/kata/share/kata-containers/kata-containers.img"
machine_type = "pc"

# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc.
#
# WARNING: - any parameter specified here will take priority over the default
# parameter value of the same name used to start the virtual machine.
# Do not set values here unless you understand the impact of doing so as you
# may stop the virtual machine from booting.
# To see the list of default parameters, enable hypervisor debug, create a
# container and look for 'default-kernel-parameters' log entries.
kernel_params = ""

# Path to the firmware.
# If you want that qemu uses the default firmware leave this option empty
firmware = ""

# Machine accelerators
# comma-separated list of machine accelerators to pass to the hypervisor.
# For example, `machine_accelerators = "nosmm,nosmbus,nosata,nopit,static-prt,nofw"`
machine_accelerators=""

# Default number of vCPUs per SB/VM:
# unspecified or 0                --> will be set to 1
# < 0                             --> will be set to the actual number of physical cores
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores
default_vcpus = 1

# Default maximum number of vCPUs per SB/VM:
# unspecified or == 0             --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# > 0 <= number of physical cores --> will be set to the specified number
# > number of physical cores      --> will be set to the actual number of physical cores or to the maximum number
#                                     of vCPUs supported by KVM if that number is exceeded
# WARNING: Depending of the architecture, the maximum number of vCPUs supported by KVM is used when
# the actual number of physical cores is greater than it.
# WARNING: Be aware that this value impacts the virtual machine's memory footprint and CPU
# the hotplug functionality. For example, `default_maxvcpus = 240` specifies that until 240 vCPUs
# can be added to a SB/VM, but the memory footprint will be big. Another example, with
# `default_maxvcpus = 8` the memory footprint will be small, but 8 will be the maximum number of
# vCPUs supported by the SB/VM. In general, we recommend that you do not edit this variable,
# unless you know what are you doing.
default_maxvcpus = 0

# Bridges can be used to hot plug devices.
# Limitations:
# * Currently only pci bridges are supported
# * Until 30 devices per bridge can be hot plugged.
# * Until 5 PCI bridges can be cold plugged per VM.
#   This limitation could be a bug in qemu or in the kernel
# Default number of bridges per SB/VM:
# unspecified or 0   --> will be set to 1
# > 1 <= 5           --> will be set to the specified number
# > 5                --> will be set to 5
default_bridges = 1

# Default memory size in MiB for SB/VM.
# If unspecified then it will be set 2048 MiB.
default_memory = 2048
#
# Default memory slots per SB/VM.
# If unspecified then it will be set 10.
# This is will determine the times that memory will be hotadded to sandbox/VM.
#memory_slots = 10

# The size in MiB will be plused to max memory of hypervisor.
# It is the memory address space for the NVDIMM devie.
# If set block storage driver (block_device_driver) to "nvdimm",
# should set memory_offset to the size of block device.
# Default 0
#memory_offset = 0

# Disable block device from being used for a container's rootfs.
# In case of a storage driver like devicemapper where a container's
# root file system is backed by a block device, the block device is passed
# directly to the hypervisor for performance reasons.
# This flag prevents the block device from being passed to the hypervisor,
# 9pfs is used instead to pass the rootfs.
disable_block_device_use = false

# Block storage driver to be used for the hypervisor in case the container
# rootfs is backed by a block device. This is virtio-scsi, virtio-blk
# or nvdimm.
block_device_driver = "virtio-scsi"

# Specifies cache-related options will be set to block devices or not.
# Default false
#block_device_cache_set = true

# Specifies cache-related options for block devices.
# Denotes whether use of O_DIRECT (bypass the host page cache) is enabled.
# Default false
#block_device_cache_direct = true

# Specifies cache-related options for block devices.
# Denotes whether flush requests for the device are ignored.
# Default false
#block_device_cache_noflush = true

# Enable iothreads (data-plane) to be used. This causes IO to be
# handled in a separate IO thread. This is currently only implemented
# for SCSI.
#
enable_iothreads = false

# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true

# Enable huge pages for VM RAM, default false
# Enabling this will result in the VM memory
# being allocated using huge pages.
# This is useful when you want to use vhost-user network
# stacks within the container. This will automatically
# result in memory pre allocation
#enable_hugepages = true

# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true

# This option changes the default hypervisor and kernel parameters
# to enable debug output where available. This extra output is added
# to the proxy logs, but only when proxy debug is also enabled.
#
# Default false
#enable_debug = true

# Disable the customizations done in the runtime when it detects
# that it is running on top a VMM. This will result in the runtime
# behaving as it would when running on bare metal.
#
#disable_nesting_checks = true

# This is the msize used for 9p shares. It is the number of bytes
# used for 9p packet payload.
#msize_9p = 8192

# If true and vsocks are supported, use vsocks to communicate directly
# with the agent and no proxy is started, otherwise use unix
# sockets and start a proxy to communicate with the agent.
# Default false
#use_vsock = true

# VFIO devices are hotplugged on a bridge by default.
# Enable hotplugging on root bus. This may be required for devices with
# a large PCI bar, as this is a current limitation with hotplugging on
# a bridge. This value is valid for "pc" machine type.
# Default false
#hotplug_vfio_on_root_bus = true

# If host doesn't support vhost_net, set to true. Thus we won't create vhost fds for nics.
# Default false
#disable_vhost_net = true
#
# Default entropy source.
# The path to a host source of entropy (including a real hardware RNG)
# /dev/urandom and /dev/random are two main options.
# Be aware that /dev/random is a blocking source of entropy.  If the host
# runs out of entropy, the VMs boot time will increase leading to get startup
# timeouts.
# The source of entropy /dev/urandom is non-blocking and provides a
# generally acceptable source of entropy. It should work well for pretty much
# all practical purposes.
#entropy_source= "/dev/urandom"

# Path to OCI hook binaries in the *guest rootfs*.
# This does not affect host-side hooks which must instead be added to
# the OCI spec passed to the runtime.
#
# You can create a rootfs with hooks by customizing the osbuilder scripts:
# https://github.com/kata-containers/osbuilder
#
# Hooks must be stored in a subdirectory of guest_hook_path according to their
# hook type, i.e. "guest_hook_path/{prestart,postart,poststop}".
# The agent will scan these directories for executable files and add them, in
# lexicographical order, to the lifecycle of the guest container.
# Hooks are executed in the runtime namespace of the guest. See the official documentation:
# https://github.com/opencontainers/runtime-spec/blob/v1.0.1/config.md#posix-platform-hooks
# Warnings will be logged if any error is encountered will scanning for hooks,
# but it will not abort container execution.
#guest_hook_path = "/usr/share/oci/hooks"

[factory]
# VM templating support. Once enabled, new VMs are created from template
# using vm cloning. They will share the same initial kernel, initramfs and
# agent memory by mapping it readonly. It helps speeding up new container
# creation and saves a lot of memory if there are many kata containers running
# on the same host.
#
# When disabled, new VMs are created from scratch.
#
# Note: Requires "initrd=" to be set ("image=" is not supported).
#
# Default false
#enable_template = true

[proxy.kata]
path = "/opt/kata/libexec/kata-containers/kata-proxy"

# If enabled, proxy messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[shim.kata]
path = "/opt/kata/libexec/kata-containers/kata-shim"

# If enabled, shim messages will be sent to the system log
# (default: disabled)
#enable_debug = true

# If enabled, the shim will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
#
# Note: By default, the shim runs in a separate network namespace. Therefore,
# to allow it to send trace details to the Jaeger agent running on the host,
# it is necessary to set 'disable_new_netns=true' so that it runs in the host
# network namespace.
#
# (default: disabled)
#enable_tracing = true

[agent.kata]
# There is no field for this section. The goal is only to be able to
# specify which type of agent the user wants to use.

[netmon]
# If enabled, the network monitoring process gets started when the
# sandbox is created. This allows for the detection of some additional
# network being added to the existing network namespace, after the
# sandbox has been created.
# (default: disabled)
#enable_netmon = true

# Specify the path to the netmon binary.
path = "/opt/kata/libexec/kata-containers/kata-netmon"

# If enabled, netmon messages will be sent to the system log
# (default: disabled)
#enable_debug = true

[runtime]
# If enabled, the runtime will log additional debug messages to the
# system log
# (default: disabled)
#enable_debug = true
#
# Internetworking model
# Determines how the VM should be connected to the
# the container network interface
# Options:
#
#   - bridged
#     Uses a linux bridge to interconnect the container interface to
#     the VM. Works for most cases except macvlan and ipvlan.
#
#   - macvtap
#     Used when the Container network interface can be bridged using
#     macvtap.
#
#   - none
#     Used when customize network. Only creates a tap device. No veth pair.
#
#   - tcfilter
#     Uses tc filter rules to redirect traffic from the network interface
#     provided by plugin to a tap interface connected to the VM.
#
internetworking_model="macvtap"

# disable guest seccomp
# Determines whether container seccomp profiles are passed to the virtual
# machine and applied by the kata agent. If set to true, seccomp is not applied
# within the guest
# (default: true)
disable_guest_seccomp=true

# If enabled, the runtime will create opentracing.io traces and spans.
# (See https://www.jaegertracing.io/docs/getting-started).
# (default: disabled)
#enable_tracing = true

# If enabled, the runtime will not create a network namespace for shim and hypervisor processes.
# This option may have some potential impacts to your host. It should only be used when you know what you're doing.
# `disable_new_netns` conflicts with `enable_netmon`
# `disable_new_netns` conflicts with `internetworking_model=bridged` and `internetworking_model=macvtap`. It works only
# with `internetworking_model=none`. The tap device will be in the host network namespace and can connect to a bridge
# (like OVS) directly.
# If you are using docker, `disable_new_netns` only works with `docker run --net=none`
# (default: false)
#disable_new_netns = true

Config file /usr/share/defaults/kata-containers/configuration.toml not found

---

KSM throttler

version

Output of " --version":

kata-collect-data.sh version 1.5.0 (commit 5f7fcd773089a615b776862f92217e987f06df0a)

systemd service

Image details

---
osbuilder:
  url: "https://github.com/kata-containers/osbuilder"
  version: "unknown"
rootfs-creation-time: "2019-02-11T14:55:18.794194275+0000Z"
description: "osbuilder rootfs"
file-format-version: "0.0.2"
architecture: "x86_64"
base-distro:
  name: "Centos"
  version: "7"
  packages:
    default:
      - "bridge-utils"
      - "iproute"
      - "iptables"
      - "net-tools"
      - "nfs-utils"
      - "systemd"
    extra:
      - "bash"
agent:
  url: "https://github.com/kata-containers/agent"
  name: "kata-agent"
  version: "1.5.0-rc2-91a87bc7176750eee2f032aa564e8b28f4ad0e73"
  agent-is-init-daemon: "no"

---

Initrd details

No initrd

---

Logfiles

Runtime logs

Recent runtime problems found in system journal:

time="2019-02-08T18:26:20.060071324-07:00" level=error msg="Container ID (72adba7d365c782b16a6fcf66e4becbaaf2606586537e8c6fa38e584003131fe) does not exist" arch=amd64 command=state container=72adba7d365c782b16a6fcf66e4becbaaf2606586537e8c6fa38e584003131fe name=kata-runtime pid=5107 source=runtime
time="2019-02-08T18:26:20.092439463-07:00" level=error msg="Container ID (72adba7d365c782b16a6fcf66e4becbaaf2606586537e8c6fa38e584003131fe) does not exist" arch=amd64 command=state container=72adba7d365c782b16a6fcf66e4becbaaf2606586537e8c6fa38e584003131fe name=kata-runtime pid=5114 source=runtime
time="2019-02-09T21:10:12.95855186-07:00" level=error msg="Container ID (8a6b705405d9b4053799057e51901fa17eb650c128fb22a743f150f6c36dfdea) does not exist" arch=amd64 command=state container=8a6b705405d9b4053799057e51901fa17eb650c128fb22a743f150f6c36dfdea name=kata-runtime pid=13928 source=runtime
time="2019-02-10T22:41:21.514906376-07:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df error="open /run/vc/sbs/e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df/devices.json: no such file or directory" name=kata-runtime pid=47233 sandbox=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df sandboxid=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df source=virtcontainers subsystem=sandbox
time="2019-02-10T22:41:22.51165651-07:00" level=warning msg="unsupported address" address="fe80::64b8:adff:febb:5465/64" arch=amd64 command=create container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df name=kata-runtime pid=47233 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-10T22:41:22.511801094-07:00" level=warning msg="unsupported route" arch=amd64 command=create container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df destination="fe80::/64" name=kata-runtime pid=47233 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-14T22:29:40.673292897-07:00" level=error msg="rpc error: code = Unavailable desc = transport is closing" arch=amd64 command=create container=8aeaa7c3818326e047e5a53ae94f58ac680b9f955c0c5dcd0c42d5e3a7cee52a name=kata-runtime pid=47389 source=runtime
time="2019-02-14T22:30:25.679861365-07:00" level=error msg="context deadline exceeded" arch=amd64 command=state container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df name=kata-runtime pid=47741 source=runtime
time="2019-02-14T22:31:10.728727876-07:00" level=error msg="context deadline exceeded" arch=amd64 command=state container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df name=kata-runtime pid=52828 source=runtime
time="2019-02-15T02:43:34.36673938-07:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c error="open /run/vc/sbs/adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c/devices.json: no such file or directory" name=kata-runtime pid=62757 sandbox=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c sandboxid=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c source=virtcontainers subsystem=sandbox
time="2019-02-15T02:43:35.424885934-07:00" level=warning msg="unsupported address" address="fe80::eca0:aaff:fe41:bf8/64" arch=amd64 command=create container=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c name=kata-runtime pid=62757 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-15T02:43:35.425026329-07:00" level=warning msg="unsupported route" arch=amd64 command=create container=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c destination="fe80::/64" name=kata-runtime pid=62757 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-15T02:54:57.145504841-07:00" level=error msg="context deadline exceeded" arch=amd64 command=state container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df name=kata-runtime pid=10914 source=runtime
time="2019-02-15T02:58:00.741052274-07:00" level=error msg="rpc error: code = Unavailable desc = transport is closing" arch=amd64 command=create container=b1b653828793790fa4a5f4de5b75e0cefc3b344f8eb04e2756a64b1471de25ae name=kata-runtime pid=62881 source=runtime
time="2019-02-15T03:03:32.993013847-07:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=0e4ca8780e1157d7be2cd7a72b2b4f3ddfbb7f05aeb6f946352c695d3345d10e error="open /run/vc/sbs/0e4ca8780e1157d7be2cd7a72b2b4f3ddfbb7f05aeb6f946352c695d3345d10e/devices.json: no such file or directory" name=kata-runtime pid=24568 sandbox=0e4ca8780e1157d7be2cd7a72b2b4f3ddfbb7f05aeb6f946352c695d3345d10e sandboxid=0e4ca8780e1157d7be2cd7a72b2b4f3ddfbb7f05aeb6f946352c695d3345d10e source=virtcontainers subsystem=sandbox
time="2019-02-15T03:03:34.097068278-07:00" level=warning msg="unsupported address" address="fe80::90a5:e3ff:fe30:50c1/64" arch=amd64 command=create container=0e4ca8780e1157d7be2cd7a72b2b4f3ddfbb7f05aeb6f946352c695d3345d10e name=kata-runtime pid=24568 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-15T03:03:34.097187524-07:00" level=warning msg="unsupported route" arch=amd64 command=create container=0e4ca8780e1157d7be2cd7a72b2b4f3ddfbb7f05aeb6f946352c695d3345d10e destination="fe80::/64" name=kata-runtime pid=24568 source=virtcontainers subsystem=network unsupported-route-type=ipv6
time="2019-02-15T03:04:26.081288663-07:00" level=error msg="context deadline exceeded" arch=amd64 command=state container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df name=kata-runtime pid=25182 source=runtime
time="2019-02-15T03:13:11.129955149-07:00" level=warning msg="fetch sandbox device failed" arch=amd64 command=create container=fa74c4c8a35f15d8b95dc8475e98dbb2241ebfb10114a258f96bf25fcde17cea error="open /run/vc/sbs/fa74c4c8a35f15d8b95dc8475e98dbb2241ebfb10114a258f96bf25fcde17cea/devices.json: no such file or directory" name=kata-runtime pid=31443 sandbox=fa74c4c8a35f15d8b95dc8475e98dbb2241ebfb10114a258f96bf25fcde17cea sandboxid=fa74c4c8a35f15d8b95dc8475e98dbb2241ebfb10114a258f96bf25fcde17cea source=virtcontainers subsystem=sandbox
time="2019-02-15T03:13:12.38998092-07:00" level=warning msg="unsupported address" address="fe80::784f:86ff:febd:e623/64" arch=amd64 command=create container=fa74c4c8a35f15d8b95dc8475e98dbb2241ebfb10114a258f96bf25fcde17cea name=kata-runtime pid=31443 source=virtcontainers subsystem=network unsupported-address-type=ipv6
time="2019-02-15T03:13:12.390084088-07:00" level=warning msg="unsupported route" arch=amd64 command=create container=fa74c4c8a35f15d8b95dc8475e98dbb2241ebfb10114a258f96bf25fcde17cea destination="fe80::/64" name=kata-runtime pid=31443 source=virtcontainers subsystem=network unsupported-route-type=ipv6

Proxy logs

Recent proxy problems found in system journal:

time="2019-02-10T22:19:39.335534648-07:00" level=fatal msg="failed to handle exit signal" error="close unix @->/run/vc/vm/320650fc9dbdf75465b74bf53f01a1e58191c7f0fabefd81f636d2c39b1bb92d/kata.sock: use of closed network connection" name=kata-proxy pid=14990 sandbox=320650fc9dbdf75465b74bf53f01a1e58191c7f0fabefd81f636d2c39b1bb92d source=proxy
time="2019-02-14T22:29:40.33853425-07:00" level=warning msg="2019/02/14 22:29:40 [ERR] yamux: Failed to write header: write unix @->/run/vc/vm/e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df/kata.sock: write: broken pipe\n" component=yamux name=kata-proxy pid=47253 sandbox=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df source=proxy
time="2019-02-14T22:29:40.338731988-07:00" level=fatal msg="channel error" error="session shutdown" name=kata-proxy pid=47253 sandbox=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df source=proxy
time="2019-02-15T02:58:00.394448389-07:00" level=fatal msg="channel error" error="session shutdown" name=kata-proxy pid=62776 sandbox=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c source=proxy

Shim logs

Recent shim problems found in system journal:

time="2019-02-14T22:29:40.341281038-07:00" level=error msg="failed waiting for process" container=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df error="rpc error: code = Unavailable desc = transport is closing" exec-id=e5f1ce65e7af56facf1289971d0fee1cd5b83d8b6e35015515e21d0052e583df name=kata-shim pid=1 source=shim
time="2019-02-15T03:00:20.396223073-07:00" level=error msg="failed waiting for process" container=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c error="rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = \"transport: Error while dialing rpc error: code = DeadlineExceeded desc = timed out connecting to unix socket ////run/vc/sbs/adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c/proxy.sock\"" exec-id=adc9b60a7ce858a6b5b570b947021465e9161167a905c70a8be942e660ef9b6c name=kata-shim pid=1 source=shim

Throttler logs

No recent throttler problems found in system journal.

---

Container manager details

Have docker

Docker

Output of "docker version":

Client:
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.6
 Git commit:        6247962
 Built:             Sun Feb 10 04:13:27 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 03:47:25 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of "docker info":

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 18.09.2
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 09c8266bf2fcf9519a651b04ae54c967b9ab86ec
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.15.0-43.generic.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 64
Total Memory: 377.5GiB
Name: xxxxxxxx
ID: XIQY:V6PV:P76V:GYXC:EVBX:O727:PX4H:BFTR:ILE6:J7XW:QXIF:UKA4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: true
Product License: Community Engine

WARNING: No swap limit support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled

Output of "systemctl show docker":

Type=notify
Restart=always
NotifyAccess=main
RestartUSec=2s
TimeoutStartUSec=0
TimeoutStopUSec=0
WatchdogUSec=0
WatchdogTimestamp=Fri 2019-02-15 03:10:24 -07
WatchdogTimestampMonotonic=19318162
StartLimitInterval=60000000
StartLimitBurst=3
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=13697
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Fri 2019-02-15 03:10:19 -07
ExecMainStartTimestampMonotonic=14344567
ExecMainExitTimestampMonotonic=0
ExecMainPID=13697
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecStartPost={ path=/usr/sbin/iptables ; argv[]=/usr/sbin/iptables -P FORWARD ACCEPT ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
ExecReload={ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/docker.service
MemoryCurrent=159477760
TasksCurrent=147
Delegate=yes
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
Environment=PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=18446744073709551615
LimitAS=18446744073709551615
LimitNPROC=18446744073709551615
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=1546157
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=docker.service
Names=docker.service
Requires=basic.target docker.socket
Wants=network-online.target system.slice
WantedBy=multi-user.target
ConsistsOf=docker.socket
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=firewalld.service docker.socket system.slice network-online.target systemd-journald.socket basic.target
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
Description=Docker Application Container Engine
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/usr/lib/systemd/system/docker.service
UnitFileState=enabled
UnitFilePreset=disabled
InactiveExitTimestamp=Fri 2019-02-15 03:10:19 -07
InactiveExitTimestampMonotonic=14344610
ActiveEnterTimestamp=Fri 2019-02-15 03:10:24 -07
ActiveEnterTimestampMonotonic=19321276
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=yes
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Fri 2019-02-15 03:10:19 -07
ConditionTimestampMonotonic=14342957
AssertTimestamp=Fri 2019-02-15 03:10:19 -07
AssertTimestampMonotonic=14342958
Transient=no

Have kubectl

Kubernetes

Output of "kubectl version":

Client Version: version.Info{Major:"1", Minor:"12+", GitVersion:"v1.12.3-439+fabd6e72745b91", GitCommit:"fabd6e72745b91409cd03e44d783d26a101e9198", GitTreeState:"clean", BuildDate:"2019-01-29T05:50:07Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

Output of "kubectl config view":

apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

Output of "systemctl show kubelet":

Type=simple
Restart=always
NotifyAccess=none
RestartUSec=2s
TimeoutStartUSec=1min 30s
TimeoutStopUSec=1min 30s
WatchdogUSec=0
WatchdogTimestamp=Fri 2019-02-15 03:10:33 -07
WatchdogTimestampMonotonic=27557835
StartLimitInterval=0
StartLimitBurst=5
StartLimitAction=none
FailureAction=none
PermissionsStartOnly=no
RootDirectoryStartOnly=no
RemainAfterExit=no
GuessMainPID=yes
MainPID=21040
ControlPID=0
FileDescriptorStoreMax=0
StatusErrno=0
Result=success
ExecMainStartTimestamp=Fri 2019-02-15 03:10:33 -07
ExecMainStartTimestampMonotonic=27557802
ExecMainExitTimestampMonotonic=0
ExecMainPID=21040
ExecMainCode=0
ExecMainStatus=0
ExecStart={ path=/usr/local/bin/kubelet ; argv[]=/usr/local/bin/kubelet $DAEMON_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }
Slice=system.slice
ControlGroup=/system.slice/kubelet.service
MemoryCurrent=48115712
TasksCurrent=0
Delegate=no
CPUAccounting=no
CPUShares=18446744073709551615
StartupCPUShares=18446744073709551615
CPUQuotaPerSecUSec=infinity
BlockIOAccounting=no
BlockIOWeight=18446744073709551615
StartupBlockIOWeight=18446744073709551615
MemoryAccounting=no
MemoryLimit=18446744073709551615
DevicePolicy=auto
TasksAccounting=no
TasksMax=18446744073709551615
EnvironmentFile=/etc/sysconfig/kubelet (ignore_errors=no)
UMask=0022
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=4096
LimitAS=18446744073709551615
LimitNPROC=1546157
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=1546157
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615
OOMScoreAdjust=0
Nice=0
IOScheduling=0
CPUSchedulingPolicy=0
CPUSchedulingPriority=0
TimerSlackNSec=50000
CPUSchedulingResetOnFork=no
NonBlocking=no
StandardInput=null
StandardOutput=journal
StandardError=inherit
TTYReset=no
TTYVHangup=no
TTYVTDisallocate=no
SyslogPriority=30
SyslogLevelPrefix=yes
SecureBits=0
CapabilityBoundingSet=18446744073709551615
AmbientCapabilities=0
MountFlags=0
PrivateTmp=no
PrivateNetwork=no
PrivateDevices=no
ProtectHome=no
ProtectSystem=no
SameProcessGroup=no
IgnoreSIGPIPE=yes
NoNewPrivileges=no
SystemCallErrorNumber=0
RuntimeDirectoryMode=0755
KillMode=process
KillSignal=15
SendSIGKILL=yes
SendSIGHUP=no
Id=kubelet.service
Names=kubelet.service
Requires=criproxy.service basic.target
Wants=criproxy.service system.slice
WantedBy=multi-user.target
Conflicts=shutdown.target
Before=shutdown.target multi-user.target
After=criproxy.service systemd-journald.socket basic.target system.slice
Documentation=https://github.com/kubernetes/kubernetes
Description=Kubernetes Kubelet Server
LoadState=loaded
ActiveState=active
SubState=running
FragmentPath=/etc/systemd/system/kubelet.service
UnitFileState=enabled
UnitFilePreset=disabled
InactiveExitTimestamp=Fri 2019-02-15 03:10:33 -07
InactiveExitTimestampMonotonic=27557850
ActiveEnterTimestamp=Fri 2019-02-15 03:10:33 -07
ActiveEnterTimestampMonotonic=27557850
ActiveExitTimestampMonotonic=0
InactiveEnterTimestampMonotonic=0
CanStart=yes
CanStop=yes
CanReload=no
CanIsolate=no
StopWhenUnneeded=no
RefuseManualStart=no
RefuseManualStop=no
AllowIsolate=no
DefaultDependencies=yes
OnFailureJobMode=replace
IgnoreOnIsolate=no
IgnoreOnSnapshot=no
NeedDaemonReload=no
JobTimeoutUSec=0
JobTimeoutAction=none
ConditionResult=yes
AssertResult=yes
ConditionTimestamp=Fri 2019-02-15 03:10:33 -07
ConditionTimestampMonotonic=27555739
AssertTimestamp=Fri 2019-02-15 03:10:33 -07
AssertTimestampMonotonic=27555739
Transient=no

No crio

---

Packages

No dpkg
Have rpm
Output of "rpm -qa|egrep "(cc-oci-runtimecc-runtimerunv|kata-proxy|kata-runtime|kata-shim|kata-ksm-throttler|kata-containers-image|linux-container|qemu-)"":

---

[wenlxie]

We install the packages via get the 1.5.0 release tar and then install it via binary.

[wenlxie]

If I enable the console log, then from the console, I can see the error:

[   61.832516] acpi PNP0C80:01: add_memory failed
[   61.834407] acpi PNP0C80:01: acpi_memory_enable_device() error
[   61.838092] Out of memory: Kill process 133 (kata-agent) score 4 or sacrifice child
[   61.841639] Killed process 204 (pause) total-vm:3824kB, anon-rss:1000kB, file-rss:4kB, shmem-rss:0kB
time="2019-02-15T05:42:36.692647012Z" level=info msg="ignoring unexpected signal" name=kata-agent pid=133 sandbox=66168c006afb63ba156a51f496607dff06e0560ade0d1d1e30f78bff92486b45 signal="child exited" source=agent
[   61.887771] Out of memory: Kill process 133 (kata-agent) score 4 or sacrifice child
[   61.890555] Killed process 133 (kata-agent) total-vm:463460kB, anon-rss:8760kB, file-rss:4kB, shmem-rss:0kB
[   61.937938] systemd-udevd[92]: worker [1189] did not accept message -111 (Connection refused), kill it
[   61.941428] systemd-udevd[92]: worker [1194] did not accept message -111 (Connection refused), kill it
[   61.944994] systemd-udevd[92]: worker [1196] did not accept message -111 (Connection refused), kill it
[   61.948558] systemd-udevd[92]: worker [1197] did not accept message -111 (Connection refused), kill it
[   61.994515] reboot: Power down

[wenlxie]

The image [Image] Path = "/var/usrlocal/kata/opt/kata/share/kata-containers/kata-containers-centos.img" is the image we build via osbuilder with debug log enabled. If we change to the default image, the issue still exists.

The console log:

SeaBIOS (version rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org)
Machine UUID 5487246d-5c9a-4cb7-a1e7-7e685743d05a
Booting from ROM..
bash-4.2# time="2019-02-15T08:00:33.765152406Z" level=info msg=announce device-handlers="mmioblk,blk,scsi,nvdimm" name=kata-agent pid=132 source=agent storage-handlers="9p,blk,mmioblk,sc3
time="2019-02-15T08:00:33.771298882Z" level=info msg="Started listening for uevents" name=kata-agent pid=132 source=agent subsystem=udevlistener
time="2019-02-15T08:00:33.775000008Z" level=info msg="agent grpc server starts" name=kata-agent pid=132 source=agent
time="2019-02-15T08:00:33.781045952Z" level=info msg="Getting interface from MAC address" interface-name=eth0 mac-address="ba:9a:da:d5:35:2b" name=kata-agent pci-address= pid=132 source=t
time="2019-02-15T08:00:33.78471215Z" level=info msg="Link found" interface-name=eth0 link="&{LinkAttrs:{Index:2 MTU:1500 TxQLen:1000 Name:eth0 HardwareAddr:ba:9a:da:d5:35:2b Flags:broadct
time="2019-02-15T08:00:33.799826234Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/di
time="2019-02-15T08:00:33.874638155Z" level=info msg="ignoring unexpected signal" name=kata-agent pid=132 sandbox=bd351b19f7fb770d9cdc6fafce913842e461259a10f9728089052326a14d4044 signal=t
time="2019-02-15T08:00:33.877933761Z" level=info msg="ignoring unexpected signal" name=kata-agent pid=132 sandbox=bd351b19f7fb770d9cdc6fafce913842e461259a10f9728089052326a14d4044 signal=t
time="2019-02-15T08:00:34.166432874Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.212933357Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.245957149Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.257942384Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.271811607Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.283766778Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.296702003Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.308916502Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.32091496Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/deu
time="2019-02-15T08:00:34.332937527Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.34592971Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/deu
time="2019-02-15T08:00:34.357923155Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.369918853Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.382504483Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.394920612Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.40691389Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/deu
time="2019-02-15T08:00:34.418923452Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.43182574Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/deu
time="2019-02-15T08:00:34.443916412Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:34.456540331Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/du
time="2019-02-15T08:00:36.662756147Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname=cpu/1/msr uevent-dr
time="2019-02-15T08:00:36.670462073Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname=cpu/1/cpuid ueventd
time="2019-02-15T08:00:36.670545353Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/dl
time="2019-02-15T08:00:36.670901527Z" level=info msg="Received add uevent" name=kata-agent pid=132 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/dy
[  184.754738] Out of memory: Kill process 132 (kata-agent) score 4 or sacrifice child
[  184.758297] Killed process 197 (pause) total-vm:3824kB, anon-rss:996kB, file-rss:4kB, shmem-rss:0kB
[  184.771424] Out of memory: Kill process 132 (kata-agent) score 4 or sacrifice child
[  184.775237] Killed process 132 (kata-agent) total-vm:529060kB, anon-rss:8732kB, file-rss:4kB, shmem-rss:0kB
[  184.869191] Out of memory: Kill process 1431 (systemd-udevd) score 1 or sacrifice child
[  184.872790] Killed process 1431 (systemd-udevd) total-vm:43516kB, anon-rss:2292kB, file-rss:0kB, shmem-rss:0kB
[  194.877789] reboot: Power down

[wenlxie]

1. The qemu start parameter -m 2048M,slots=20,maxmem=387630M 2. From the kata runtime log, both object-add and device-add has no errors during memory hotplug

[wenlxie]

Is there any memory hotplug limitation per dimm?
All of the memory will be add in the same dimm no matter how big the memory size is.

[wenlxie]

@grahamwhaley @egernst @jcvenegas @gnawux @caoruidong Please help to have a check.

[wenlxie]

Sometimes at the boot log, there are some trace, but not exist everytime:

time="2019-02-18T05:36:28.552089315Z" level=info msg="Received add uevent" name=kata-agent pid=138 source=agent subsystem=udevlistener uevent-action=add uevent-devname= uevent-devpath=/devices/system/memory/memory36 uevent-seqnum=1593 uev
ent-subsystem=memory
[    7.172271] kworker/u128:2: page allocation failure: order:9, mode:0x14084c0(GFP_KERNEL|__GFP_RETRY_MAYFAIL|__GFP_ZERO), nodemask=(null)
[    7.177327] kworker/u128:2 cpuset=/ mems_allowed=0
[    7.179504] CPU: 0 PID: 95 Comm: kworker/u128:2 Not tainted 4.14.67 #1
[    7.182229] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[    7.184455] Call Trace:
[    7.185465]  dump_stack+0x5c/0x78
[    7.186821]  warn_alloc.cold.32+0x93/0x125
[    7.188785]  ? compaction_zonelist_suitable+0xc2/0x130
[    7.190984]  __alloc_pages_nodemask+0xcc5/0xda0
[    7.192983]  vmemmap_alloc_block+0x35/0x73
[    7.194810]  __vmemmap_alloc_block_buf+0xfc/0x108
[    7.196758]  vmemmap_populate+0xee/0x227
[    7.198453]  sparse_mem_map_populate+0x22/0x32
[    7.200538]  sparse_add_one_section+0x4a/0x157
[    7.202417]  __add_pages+0x82/0x1b0
[    7.203842]  add_pages+0x10/0x60
[    7.205262]  add_memory_resource+0xe5/0x150
[    7.206981]  add_memory+0x7b/0xc0
[    7.208471]  ? acpi_memory_get_resource+0xe0/0xe0
[    7.210582]  acpi_memory_device_add+0x120/0x290
[    7.212459]  acpi_bus_attach+0xc5/0x1c0
[    7.214084]  acpi_bus_scan+0x2e/0x60
[    7.215469]  acpi_device_hotplug+0x248/0x420
[    7.217354]  acpi_hotplug_work_fn+0x15/0x20
[    7.219175]  process_one_work+0x1a1/0x300
[    7.220838]  worker_thread+0x26/0x3d0
[    7.222366]  ? process_one_work+0x300/0x300
[    7.224266]  kthread+0x10f/0x130
[    7.225734]  ? kthread_create_on_node+0x40/0x40
[    7.227670]  ret_from_fork+0x35/0x40
[    7.229298] Mem-Info:
[    7.230355] active_anon:4983 inactive_anon:2102 isolated_anon:0
[    7.230355]  active_file:23 inactive_file:22 isolated_file:0
[    7.230355]  unevictable:0 dirty:18 writeback:0 unstable:0
[    7.230355]  slab_reclaimable:589 slab_unreclaimable:3504
[    7.230355]  mapped:677 shmem:2110 pagetables:381 bounce:0
[    7.230355]  free:4288 free_pcp:232 free_cma:0
[    7.244002] Node 0 active_anon:19932kB inactive_anon:8408kB active_file:92kB inactive_file:88kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2708kB dirty:72kB writeback:0kB shmem:8440kB writeback_tmp:0kB unstable:0kB al
l_unreclaimable? no
[    7.253727] DMA free:9420kB min:44kB low:56kB high:68kB active_anon:8kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bo
unce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[    7.263882] lowmem_reserve[]: 0 1964 1964 1964
[    7.265738] DMA32 free:7628kB min:5648kB low:7656kB high:9664kB active_anon:19932kB inactive_anon:8408kB active_file:92kB inactive_file:88kB unevictable:0kB writepending:0kB present:2080636kB managed:2016036kB mlocked:0kB kernel_stack:
2192kB pagetables:1524kB bounce:0kB free_pcp:928kB local_pcp:732kB free_cma:0kB

then there will be errors:

fatal error: runtime: cannot allocate memory

[mrlhansen]

I experienced the exact same issue on our system today. Has any progress been made since this issue was opened?

[devimc]

@mrlhansen looks like this bug was fixed here kata-containers/packaging#335
what version of kata containers are you using?

[mrlhansen]

@devimc I am using 1.11.0-rc0, specifically the packages from this repository. I checked that our kernel (v5.2.20) is compiled with the flag mentioned in the other issue.

[root@host ~]# grep CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE /boot/config-$(uname -r)
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y

[devimc]

I checked that our kernel (v5.2.20) is compiled with the flag mentioned in the other issue.
[root@host ~]# grep CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE /boot/config-$(uname -r)
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y

v5.2.20? this is the host kernel, right? CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE is required in the guest, not in the host, if you are using a custom guest kernel, make sure it includes this feature.

[mrlhansen]

@devimc This is the host kernel yes. I did not realize the flag should be in the guest kernel - but that actually makes a lot more sense. Thanks a lot, I will try it out.

[devimc]

closing issue, since it was fixed here kata-containers/packaging#335, feel free to reopen if needed