kernel: enable config CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE by default by wenlxie · Pull Request #335 · kata-containers/packaging

wenlxie · 2019-02-19T06:40:42Z

In my test env, I assigned 120Gi memory to the container, but found that Sandbox can't be launched. For details: kata-containers/runtime#1244
kata-agent and udev process was killed because of OOM triggerd.
https://lwn.net/Articles/668944/

Signed-off-by: Wenli Xie wenlxie@ebay.com

jodh-intel

lgtm

We might want input from @grahamwhaley in case this has an impact on the fragements work (#314).

grahamwhaley · 2019-02-19T09:48:52Z

:-) I think more importantly is how this does or does not fit in with the memory hotplug flow of kata - @jcvenegas @devimc for input from the 'how is kata hotplug meant to work' side.
But, this is also very good evidence for the 120Gb hotplug failure case as well - nice find @wenlxie

devimc · 2019-02-19T14:23:19Z

@wenlxie great job! do you know why kata-agent receives OOM?
Currently we connect the memory (and CPUs) manually through gRPC/kata-agent, with this patch this implementation should change. I'm not sure but I think this patch can break compatibility with ARM (cc @Pennyzct @Weichen81 )

jcvenegas · 2019-02-19T17:25:53Z

@wenlxie intesting what is the maximal amount of memory you can hotplug currently with out get the agent killed ?

sboeuf · 2019-02-19T18:17:30Z

@wenlxie the way memory hotplug works for ARM has been implemented here by @Pennyzct.

@Pennyzct could you take a look at this PR and provide some feedback since I'm worried having the config CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE enabled might not be the right fix here.

wenlxie · 2019-02-20T01:01:10Z

@jcvenegas

@wenlxie intesting what is the maximal amount of memory you can hotplug currently with out get the agent killed ?

In my env, the limitation is 120Gi. I tested 119Gi memory, it is ok.

wenlxie · 2019-02-20T01:23:51Z

@devimc
I think this is the reason. https://lwn.net/Articles/668944/

Currently, all newly added memory blocks remain in 'offline' state unless
someone onlines them, some linux distributions carry special udev rules
like:

SUBSYSTEM=="memory", ACTION=="add", ATTR{state}=="offline", ATTR{state}="online"

to make this happen automatically. This is not a great solution for virtual
machines where memory hotplug is being used to address high memory pressure
situations as such onlining is slow and a userspace process doing this
(udev) has a chance of being killed by the OOM killer as it will probably
require to allocate some memory.

Pennyzct · 2019-02-20T03:43:12Z

Hi~@wenlxie @devimc @sboeuf
For ARM, we also rely on listenToUdevEvents and onlineCPUMem to online hot-plugged memory blocks.
And I have tried this CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE, it works well, which means all above manual dependency also could be eliminated on arm platform. ;)

jcvenegas · 2019-02-20T13:42:39Z

@chavafg I am afraid we dont have a full kata testing in this repository we need to add one to fully test.
@wenlxie meanwhile.

Seems that CI SoB-check is failing due to missing commit body, please add something similar that you put in this PR. Also please open an github issue for this. And also add to the commit body a line like:

Fixes: #ISSUE_NUMBER

While we have a job here to fully test kata we need to open a dummy in runtime and add a depends on I can do that for you.

jcvenegas · 2019-02-20T13:44:37Z

@wenlxie @sboeuf if this works well we will need to sync to also remove onlineCPUMem the memory part.

sboeuf · 2019-02-20T15:27:24Z

@jcvenegas
Ok so if I summarize here, by having this flag, the kernel will take care of hotplugging any new memory added on the fly while the system is already booted, right?

@Pennyzct confirmed this feature works well for ARM and we don't need the manual logic of doing OnlineCpuMem from the agent, is that right?

We need to try this out with x86_64 and s390, to validate that the manual online cpu and memory logic can be totally removed.
@jcvenegas can you check this for x86?
@alicefr can you check this for s390?

amshinde · 2019-02-20T23:44:24Z

@jcvenegas @chavafg Can we add full CI testing to this PR?
Would be good to test changes to be made in the agent (to remove manual online memory) with this change.

Pennyzct · 2019-02-21T03:03:42Z

@sboeuf yep. that's what I mean. I deleted two parts to test this config, one in listenToUdevEvents, and the whole onlineCPUMem, and it works well. all memory blocks automatically online. ;)

wenlxie · 2019-02-21T05:56:54Z

@chavafg I am afraid we dont have a full kata testing in this repository we need to add one to fully test.
@wenlxie meanwhile.

Seems that CI SoB-check is failing due to missing commit body, please add something similar that you put in this PR. Also please open an github issue for this. And also add to the commit body a line like:
Fixes: #ISSUE_NUMBER
While we have a job here to fully test kata we need to open a dummy in runtime and add a depends on I can do that for you.

@jcvenegas Thanks, now the SoB check had passed.

alicefr · 2019-02-21T09:12:45Z

@wenlxie you can remove the option for s390x. The memory hotplug is not supported by qemu KVM yet

wenlxie · 2019-02-21T09:49:22Z

@wenlxie you can remove the option for s390x. The memory hotplug is not supported by qemu KVM yet

@alicefr removed, thanks

sboeuf · 2019-02-21T14:45:47Z

/test

wenlxie · 2019-02-22T03:06:16Z

/test

wenlxie · 2019-02-22T03:06:50Z

kata_config_version is also need to be changed. Or the Ci job will fail.

sboeuf · 2019-02-22T17:57:52Z

/test

wenlxie · 2019-02-27T01:40:22Z

@jodh-intel thanks.
I had bump the kata_config_version

wenlxie · 2019-02-27T08:49:54Z

Need to run CI job again. Can someone help to trigger it?

jodh-intel · 2019-02-27T08:52:25Z

Sure....

/retest

jodh-intel · 2019-02-27T08:52:34Z

/test

grahamwhaley · 2019-03-04T17:19:07Z

@@ -1 +1 @@
-25
+26


Hi @wenlxie - sorry about this, but I think the kernel config version has moved ahead in the master branch again - can you rebase and repush... sorry for the hassle - let's see if we can get this one moving again...

wenlxie · 2019-03-06T09:27:36Z

@grahamwhaley I rebase it and need someone help to trigger the test, thank.

nitkon · 2019-03-06T10:05:51Z

/test

wenlxie · 2019-03-07T01:25:18Z

The kernel config version bumps again. :(

jcvenegas · 2019-03-07T03:05:35Z

/test

jcvenegas · 2019-03-07T03:06:24Z

@jodh-intel could you merge to another bump race condition :P ?

jodh-intel · 2019-03-07T09:27:11Z

Restarted CI which failed due to a timeout.

jcvenegas · 2019-03-07T16:32:40Z

time out again copy here to check if is a not stable test

• [SLOW TEST:5.021 seconds]
docker exit code
/tmp/jenkins/workspace/kata-containers-packaging-ubuntu-18-04-PR/go/src/github.com/kata-containers/tests/integration/docker/exitcode_test.go:20
  check exit codes
  /tmp/jenkins/workspace/kata-containers-packaging-ubuntu-18-04-PR/go/src/github.com/kata-containers/tests/vendor/github.com/onsi/ginkgo/extensions/table/table.go:92
    with exit code '256' when interactive mode is: 'false', it should exit '0'
    /tmp/jenkins/workspace/kata-containers-packaging-ubuntu-18-04-PR/go/src/github.com/kata-containers/tests/vendor/github.com/onsi/ginkgo/extensions/table/table_entry.go:46
------------------------------
Build timed out (after 50 minutes). Marking the build as aborted.

jcvenegas · 2019-03-07T16:32:50Z

/test

jodh-intel · 2019-03-11T14:39:37Z

18.04 timed out. Prodded it with a sharper stick this time...

amshinde · 2019-03-12T17:38:14Z

@wenlxie This needs another update to kata_config_version! Can you please do that?
Would like to see this merged soon.

Fixes: kata-containers#343 Signed-off-by: Wenli Xie <wenlxie@ebay.com>

wenlxie · 2019-03-13T05:04:15Z

@amshinde I rebased it. But the ci job always fails with timeout, no more infos to debug the issue.

grahamwhaley · 2019-03-13T09:37:45Z

/test
let's see how the CI is now - and keep an eye open to determine if this is a CI fail specific to this PR, or a more generic instability...
thx for your patience @wenlxie

jodh-intel · 2019-03-13T11:40:52Z

/me pounces...

wenlxie · 2019-03-14T01:31:55Z

/test
let's see how the CI is now - and keep an eye open to determine if this is a CI fail specific to this PR, or a more generic instability...
thx for your patience @wenlxie

@grahamwhaley @jodh-intel Thanks

wenlxie force-pushed the fixmemory branch from 4cb58d1 to e85773c Compare February 19, 2019 07:18

jodh-intel approved these changes Feb 19, 2019

View reviewed changes

wenlxie force-pushed the fixmemory branch from e85773c to 7eeec1c Compare February 20, 2019 01:04

wenlxie force-pushed the fixmemory branch 2 times, most recently from 2533a47 to 313e8b9 Compare February 21, 2019 02:11

wenlxie force-pushed the fixmemory branch from 313e8b9 to bd060df Compare February 21, 2019 05:00

wenlxie force-pushed the fixmemory branch from bd060df to c51aee1 Compare February 21, 2019 09:47

wenlxie force-pushed the fixmemory branch 3 times, most recently from 28abbef to 78cb669 Compare February 22, 2019 02:25

grahamwhaley mentioned this pull request Feb 22, 2019

kernel: add CFS for CPU hotplug on s390x #340

Closed

wenlxie force-pushed the fixmemory branch from 78cb669 to 5fe65f7 Compare February 27, 2019 01:03

jodh-intel approved these changes Feb 27, 2019

View reviewed changes

grahamwhaley reviewed Mar 4, 2019

View reviewed changes

grahamwhaley mentioned this pull request Mar 4, 2019

kernel: config: add config fragment support #314

Closed

wenlxie force-pushed the fixmemory branch 2 times, most recently from e05719c to 12d3cd6 Compare March 6, 2019 09:12

wenlxie force-pushed the fixmemory branch 2 times, most recently from 42263e0 to ee28a84 Compare March 7, 2019 01:23

enable config CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE by default

47a8725

Fixes: kata-containers#343 Signed-off-by: Wenli Xie <wenlxie@ebay.com>

wenlxie force-pushed the fixmemory branch from ee28a84 to 47a8725 Compare March 13, 2019 05:02

jodh-intel merged commit 594ce2f into kata-containers:master Mar 13, 2019

devimc mentioned this pull request Jun 16, 2020

Sandbox can't be launched when memory exceed 120Gi in centos kata-containers/runtime#1244

Closed

Conversation

wenlxie commented Feb 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jodh-intel left a comment

Choose a reason for hiding this comment

Uh oh!

grahamwhaley commented Feb 19, 2019

Uh oh!

devimc commented Feb 19, 2019

Uh oh!

jcvenegas commented Feb 19, 2019

Uh oh!

sboeuf commented Feb 19, 2019

Uh oh!

wenlxie commented Feb 20, 2019

Uh oh!

wenlxie commented Feb 20, 2019

Uh oh!

Pennyzct commented Feb 20, 2019

Uh oh!

jcvenegas commented Feb 20, 2019

Uh oh!

jcvenegas commented Feb 20, 2019

Uh oh!

sboeuf commented Feb 20, 2019

Uh oh!

amshinde commented Feb 20, 2019

Uh oh!

Pennyzct commented Feb 21, 2019

Uh oh!

wenlxie commented Feb 21, 2019

Uh oh!

alicefr commented Feb 21, 2019

Uh oh!

wenlxie commented Feb 21, 2019

Uh oh!

sboeuf commented Feb 21, 2019

Uh oh!

wenlxie commented Feb 22, 2019

Uh oh!

wenlxie commented Feb 22, 2019

Uh oh!

sboeuf commented Feb 22, 2019

Uh oh!

wenlxie commented Feb 27, 2019

Uh oh!

wenlxie commented Feb 27, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jodh-intel commented Feb 27, 2019

Uh oh!

jodh-intel commented Feb 27, 2019

Uh oh!

grahamwhaley Mar 4, 2019

Choose a reason for hiding this comment

Uh oh!

wenlxie commented Mar 6, 2019

Uh oh!

nitkon commented Mar 6, 2019

Uh oh!

wenlxie commented Mar 7, 2019

Uh oh!

jcvenegas commented Mar 7, 2019

Uh oh!

jcvenegas commented Mar 7, 2019

Uh oh!

jodh-intel commented Mar 7, 2019

Uh oh!

jcvenegas commented Mar 7, 2019

Uh oh!

jcvenegas commented Mar 7, 2019

Uh oh!

jodh-intel commented Mar 11, 2019

Uh oh!

amshinde commented Mar 12, 2019

Uh oh!

wenlxie commented Mar 13, 2019

Uh oh!

grahamwhaley commented Mar 13, 2019

wenlxie commented Feb 19, 2019 •

edited

Loading

wenlxie commented Feb 27, 2019 •

edited

Loading