fix container fail to start by Ace-Tang · Pull Request #331 · kata-containers/runtime

Ace-Tang · 2018-05-22T05:00:51Z

copyParentCPUSet copies the cpuset.cpus and cpuset.mems
from the parent, but if parent directory's file content
is also zero, it will fail. Recursive copy data from
parent directory until cpuset cgroup root path.

Modify variable cgroupsDirPath to cgroupsRootPath(cli/oci.go),
make it more reasonable and not conflict with cgroupsDirPath(cli/create.go).

fixes: #330

Signed-off-by: Ace-Tang aceapril@126.com

jshachm · 2018-05-22T05:04:49Z

@Ace-Tang Plz use modules: summary to generate commit message~~~

sboeuf · 2018-05-22T05:10:54Z

/cc @devimc

Ace-Tang · 2018-05-22T05:45:21Z

@jshachm , thanks for reminding, I have update commit message.

devimc · 2018-05-22T12:15:37Z

CI is not happy

docker: Error response from daemon: OCI runtime create failed: Fail to copy cpuset from parent: open /sys/fs/cgroup/cpu/docker/cpuset.cpus: no such file or directory: unknown.

and this patch makes me wonder if we need integration tests for pouch

Ace-Tang · 2018-05-22T13:24:13Z

docker: Error response from daemon: OCI runtime create failed: Fail to copy cpuset from parent: open /sys/fs/cgroup/cpu/docker/cpuset.cpus: no such file or directory: unknown.

the data should copy from /sys/fs/cgroup/cpuset/docker/cpuset.cpus, that make the error happen, I check my machine, the cpu cgroup is link to cpuset,cpu,cpuacct, so I am not get this error.

lrwxrwxrwx  1 root root 18 May  8 03:52 cpu -> cpuset,cpu,cpuacct

copyParentCPUSet copies the cpuset.cpus and cpuset.mems from the parent, but if parent directory's file content is also zero, it will make pid write into cpuset cgroup error. So recursive copy data from parent directory until cpuset cgroup root path. Modify variable cgroupsDirPath to cgroupsRootPath(cli/oci.go), make it more reasonable and not conflict with cgroupsDirPath(cli/create.go). Fixes #330 Signed-off-by: Ace-Tang <aceapril@126.com>

katabuilder · 2018-05-23T05:46:43Z

PSS Measurement:
Qemu: 143850 KB
Proxy: 6765 KB
Shim: 8931 KB

Memory inside container:
Total Memory: 2045972 KB
Free Memory: 2013040 KB

codecov · 2018-05-23T14:00:36Z

Codecov Report

Merging #331 into master will decrease coverage by 0.11%.
The diff coverage is 24%.

@@            Coverage Diff             @@
##           master     #331      +/-   ##
==========================================
- Coverage   63.89%   63.78%   -0.12%     
==========================================
  Files          87       87              
  Lines        8623     8640      +17     
==========================================
+ Hits         5510     5511       +1     
- Misses       2529     2545      +16     
  Partials      584      584

Impacted Files	Coverage Δ
cli/create.go	`72.37% <0%> (-7.02%)`	⬇️
cli/oci.go	`90.81% <85.71%> (ø)`	⬆️
cli/config.go	`89.09% <0%> (+0.04%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 086d197...f68f894. Read the comment docs.

katabuilder · 2018-05-23T14:40:49Z

PSS Measurement:
Qemu: 141854 KB
Proxy: 4722 KB
Shim: 10939 KB

Memory inside container:
Total Memory: 2045972 KB
Free Memory: 2011676 KB

devimc · 2018-05-23T15:36:49Z

@Ace-Tang please try to fix code coverage

lgtm

thanks @Ace-Tang

sboeuf · 2018-05-24T00:55:56Z

@devimc I don't follow why you approved this PR while your comment on the corresponding issue #330 (comment) was saying that you could not reproduce ?

devimc · 2018-05-24T15:22:40Z

@sboeuf read the description of the problem

I test container with pouch, a container engine like moby.
Start a kata container with a clean cgroup path which has no child directory, it will fail

I couldn't produce it because of I'm using docker not pouch #330 (comment), this patch doesn't impact the compatibility with docker but fixes the compatibility with pouch, right @Ace-Tang ?

devimc

waiting for @Ace-Tang response

Ace-Tang · 2018-05-25T01:41:18Z

No, @devimc , the error has not related whether you use pouch or mody, I also re-produce it with mody, as I comment in #330 .

Maybe it is linux distro related, could you try again on a centos mechine, @devimc , I make a simple test on ubuntu machine, seems like ubuntu will inherit cpuset.cpus and cpuset.mems but centos does not.

root@www:/sys/fs/cgroup/cpuset# mkdir newtest
root@www:/sys/fs/cgroup/cpuset# cd newtest/
root@www:/sys/fs/cgroup/cpuset/newtest# cat cpuset.cpus
0-3

here is my test with moby

#docker run --runtime=kata-runtime -d  719f2024853e top
5de544c966b7e3e7ca3e3b0af96c9c4c1f884c1cd30b296aa713a55844c98f01

#docker run --runtime=kata-runtime -d --cgroup-parent=cleanparent 719f2024853e top
c00494cb9a7fa516d56972a04507df6653a9cf0dcc55ebc48e55c24d5c473c14
docker: Error response from daemon: OCI runtime create failed: write /sys/fs/cgroup/cpu/cleanparent/c00494cb9a7fa516d56972a04507df6653a9cf0dcc55ebc48e55c24d5c473c14/tasks: no space left on device: unknown.

#cat /etc/redhat-release 
Alibaba Group Enterprise Linux Server release 7.2 (Paladin)

the moby version is :

#docker version
Client:
 Version:      18.03.1-ce
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:20:16 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:23:58 2018
  OS/Arch:      linux/amd64
  Experimental: false

the kata version is

[root@z07a05275.sqa.zth /home/huamin.thm]
#kata-runtime -v
kata-runtime  : 0.3.0
   commit   : 2245e67f934edaa2d410384b29ab07cca305f363
   OCI specs: 1.0.1

devimc · 2018-05-25T13:03:24Z

sorry @Ace-Tang I can't reproduce it in centOS

$ docker run --cgroup-parent=cleanparent --runtime=kata-runtime -d centos top
3e7ee9c50d8fe76c58a2fc8bf566772e23434aef0c710ad767335a16c8001f39
$ docker run --cgroup-parent=cleanparent --cpus 3 --runtime=kata-runtime -d centos top
ede36031486ddd7cb29cd0caaae123a13e1f0c917b840cb230bd615978495067

centOS info

$ cat /etc/os-release 
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

docker info

$ docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 1
Server Version: 18.03.1-ce
Storage Driver: overlay2
 Backing Filesystem: xfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: kata-runtime runc
Default Runtime: kata-runtime
Init Binary: docker-init
containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88
runc version: 94b7982a1ea6bb511c74c8bc18ebfb194e87b33a (expected: 4fc53a81fb7c994640722ac585fa9ca548971871)
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-862.2.3.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.701GiB
Name: thoughtful-glatisant
ID: XXXXXXXXXXXXXXXXxx
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 19
 Goroutines: 33
 System Time: 2018-05-25T13:01:42.683880895Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

sboeuf · 2018-05-31T19:54:19Z

What's the status on this ?

egernst · 2018-06-11T17:36:58Z

@Ace-Tang - any updates here?

jshachm · 2018-06-20T05:02:05Z

@Ace-Tang any new updates here ? And maybe this pr is now related to #416 since the whole hostcgroup will be refactored

egernst · 2018-07-23T14:42:02Z

@Ace-Tang - I appreciate your efforts, but see this has gone very stale. I'm going to close the PR. If you have updates, please reopen! Thanks for the contributions.

agent: add support for online memory and cpu separately

Fixes: #0 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>

devimc suggested changes May 24, 2018

View reviewed changes

egernst closed this Jul 23, 2018

Ace-Tang deleted the fix_start_fail branch July 24, 2018 01:57

zklei pushed a commit to zklei/runtime that referenced this pull request Jun 13, 2019

Merge pull request kata-containers#331 from clarecch/master

a0136e7

agent: add support for online memory and cpu separately

fidencio added a commit to fidencio/kata-runtime that referenced this pull request Sep 28, 2020

runtime: Test ci kata-containers#331

5614f3b

Fixes: #0 Signed-off-by: Fabiano Fidêncio <fidencio@redhat.com>

Conversation

Ace-Tang commented May 22, 2018

Uh oh!

jshachm commented May 22, 2018

Uh oh!

sboeuf commented May 22, 2018

Uh oh!

Ace-Tang commented May 22, 2018

Uh oh!

devimc commented May 22, 2018

Uh oh!

Ace-Tang commented May 22, 2018

Uh oh!

katabuilder commented May 23, 2018

Uh oh!

codecov bot commented May 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

katabuilder commented May 23, 2018

Uh oh!

devimc commented May 23, 2018 • edited by amshinde Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sboeuf commented May 24, 2018

Uh oh!

devimc commented May 24, 2018

Uh oh!

devimc left a comment

Choose a reason for hiding this comment

Uh oh!

Ace-Tang commented May 25, 2018

Uh oh!

devimc commented May 25, 2018

Uh oh!

sboeuf commented May 31, 2018

Uh oh!

egernst commented Jun 11, 2018

Uh oh!

jshachm commented Jun 20, 2018

Uh oh!

egernst commented Jul 23, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov bot commented May 23, 2018 •

edited

Loading

devimc commented May 23, 2018 •

edited by amshinde

Loading