Evaluate, and implement, any applicable `update` features

[grahamwhaley]

The runtime does not implement the update command (and thus implicitly probably does not support some of the docker update functionality.
It may be possible to implement some of the functionality either by affecting the VM or injecting the requests into the container itself. Investigation and architecture needs researching to define what is feasible.

[wcwxyz]

Now runtime doesn't support per container cpu/memory configuration. Perhaps with an update command, runtime can do cpu/memory hot plugging. So people can create a container and update cpu/memory.

[grahamwhaley]

Hi @wcwxyz
@mcastelino recently added an Issue for cpu resources: #341
but I cannot find one for memory resource like docker run -m MEMORY - so I will add one in a moment.
For me, I think support of docker run -m MEMORY is probably more important and a first step before we support update.
Yes, for update to work I would expect either we would use hotplug, or we would assign lots of resource to the VM in the first place and then use cgroups to constrain what is available (and that cgroup manipulation could be done either inside or ouside the VM, which then have different pro/con effects I think).
Before we dive into any implementation then, first I think we would have to have a thorough design discussion with @mcastelino and others ( @egernst @jodh-intel @sboeuf et. al.).

Let me open an Issue for docker run -m MEMORY...

[grahamwhaley]

I meant to add...
supporting memory size changes via hotplug will be difficult due to granularity (the hotplug is generally done in DIMM sized chunks afaik for instance, and there is a limit to the number of slots), and complexity in the VM (there will have to be a bunch of migration etc. if we are unplugging and shrinking memory that is in use for instance). And, if we set up the VMs to handle the 'maximum possible memory' for each VM, then each VM will carry a large overhead to support the functionality, whether that functionality is used or not - so, that may not be an ideal overhead to impose on all containers and all users etc.
For CPU hotplug, the situation might be somewhat easier I think, but I've not looked into QEMU cpu hotplug much.

[wcwxyz]

Yes, you're right about memory hotplug.

But I'm curious how much overhead would VM has if it's configured 'maximum possible memory'? I imagine it'd only cost actual memory, the rest would be just overcommitted.

[grahamwhaley]

We did some measurements on this a little while back. @jcvenegas do you have any info you can share? Basically there is overhead in the VM for setting up the page management for the larger space iirc.
On the overcommit front, that might also be an issue. The kernel by default iirc won't let you 'wildly' over-commit for the VMs by default (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/vm/overcommit-accounting). For instance, at present if we have <2G of memory 'free' then the kernel will not allow us to launch any more VMs.
If we allocate the 'maximum available' to all VMs, then I suspect by default the kernel will let us launch very few.
If we disable the kernel overcommit check, and then multiple containers try to utilise all of their memory, then I think we are likely to enter a swap or OOM situation, both of which we'd probably like to avoid.

I think this is not a new problem - it is similar to the existing world of VM allocation and solutions such as ballooning.

I was thinking, maybe we should start a wiki page listing all the possibilities, their pros and cons, and then we can do some research and annotate that page to help us make some decisions. Maybe we need to offer multiple optional solutions to suit particular situations.

[wcwxyz]

I did some memory testing comparing -m 2G,slots=2,maxmem=3G and -m 2G,slots=2,maxmem=32G(hacked runtime).

I didn't see any memory footprint difference between these two configuration. I was able to launch as many as VMs until host available memory drop under 2G. I got 280~ VM running in hoth situation on a 32G machine (there's other apps running on this host, so 280 isn't the best it would run).

Inside the VM, I got:

MemTotal:        2052760 kB
MemFree:         2020036 kB
MemAvailable:    2008360 kB

and

MemTotal:        2052760 kB
MemFree:         2019872 kB
MemAvailable:    2008316 kB

It seems like maxmem is trivial. I think maybe we should set maxmem to host memory size. Then we got a chance to get memory hot-add working, up to the most available memory in host. Well memory hot-remove is kind of tricky and gonna be another story.

Currently we set maxmem to mem*1.5, which is not enough for running a small VM (eg. 128M) considering nvdimm device consumes memory address space.

[mcastelino]

@wcwxyz yes. Setting maxmem should not change memory footprint. This only defines the upper bounds for memory hotplug. Changing maxmem to host memory size makes sense.

[wcwxyz]

I'll send a patch to virtcontainers to address maxmem issue.

[sboeuf]

Sounds good to me :)

[grahamwhaley]

Indeed, bumping the maxmem won't cost you space in the VM/container unless you then map (plug in) that memory. The experiments we had previously done were mapping all of host memory to the VM.

I'll note that currently the -m 2G,slots=2,maxmem=3G is not 1.5* the mem size, it is n+1G, where the 1G is the extra space to map the nvdimm DAX into. Thus, we may always want/need to add the +1G, both for very small and very large setups.

If we are extending the maxmem to work towards enabling hotplug, then I suspect we will also want to modify the slots number as well on that QEMU command line. If we left it at '2', then I don't think we'd be able to hotplug anything else in:
https://github.com/clearcontainers/qemu/blob/master/docs/memory-hotplug.txt

The number of slots we set I think will be a balance between fine granularity and potential management overhead cost in the container kernel (something else we can measure I think @wcwxyz ?). I had a feeling that the maximum number of slots supported was relatively small, but having a check I think we will be bounded buy what is supported in the kernel KVM config, and that would appear to be quite large:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/include/asm/kvm_host.h#n41

[wcwxyz]

I'll note that currently the -m 2G,slots=2,maxmem=3G is not 1.5* the mem size, it is n+1G, where the 1G is the extra space to map the nvdimm DAX into. Thus, we may always want/need to add the +1G, both for very small and very large setups.

Although the code is written as maxmem=mem*1.5, I think maxmem=mem+1G makes sense.

If we are extending the maxmem to work towards enabling hotplug, then I suspect we will also want to modify the slots number as well on that QEMU command line. If we left it at '2', then I don't think we'd be able to hotplug anything else in:
https://github.com/clearcontainers/qemu/blob/master/docs/memory-hotplug.txt

AFAIK, regular ram doesn't consumes slot. But I'm not sure about nvdimm device (it looks like it should).

The number of slots we set I think will be a balance between fine granularity and potential management overhead cost in the container kernel (something else we can measure I think @wcwxyz ?). I had a feeling that the maximum number of slots supported was relatively small, but having a check I think we will be bounded buy what is supported in the kernel KVM config, and that would appear to be quite large:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/x86/include/asm/kvm_host.h#n41

I can give it a try. I'll look into memory footprint for different number of slots configured.

[grahamwhaley]

Ah @wcwxyz , I see, at here:
https://github.com/containers/virtcontainers/blob/master/qemu.go#L473
there is the hard coded *1.5. I think that is probably incorrect and has been a mis-translation of the original fixed line from: https://github.com/01org/cc-oci-runtime/blob/master/data/hypervisor.args.in#L11

let me go open an Issue on virtcontainers for that so we can clear that up.

On the slots - my reading of the QEMU docs is that you plug/unplug virtual dimm devices into slots to add/remove memory. It would actually be nice if that is not the case, but I think it all ties into how the hotplug events are injected into and handled by the client OS I expect.