OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

[yanpeipan]

machine: nvidia drive px2

/apollo/data/core/core_%e.%p
dev-aarch64-20170927_1111: Pulling from apolloauto/apollo
a1c981565bcf: Already exists 
4ecd22b74242: Already exists 
8d4841161f63: Already exists 
a93b04770247: Already exists 
Digest: sha256:10fc9daa7699f30650df05b9f3aff9762a236433e089e8755dcb0f6cfb7baab0
Status: Downloaded newer image for apolloauto/apollo:dev-aarch64-20170927_1111
RTNETLINK answers: Device or resource busy
modprobe: FATAL: Module nvidia not found in directory /lib/modules/4.4.38-rt49-tegra
modprobe: FATAL: Module nvidia-uvm not found in directory /lib/modules/4.4.38-rt49-tegra
[WARNING] Failed to find device with pattern "ttyUSB*" ...
[ OK ] Found device: /dev/ttyS0.
[ OK ] Found device: /dev/ttyS3.
[ OK ] Found device: /dev/ttyS2.
[ OK ] Found device: /dev/ttyS1.
[ OK ] Found device: /dev/can3.
[ OK ] Found device: /dev/can2.
[ OK ] Found device: /dev/can1.
[ OK ] Found device: /dev/can0.
[WARNING] Failed to find device with pattern "ram*" ...
[WARNING] Failed to find device with pattern "loop*" ...
[ OK ] Found device: /dev/nvidia-uvm-tools.
[ OK ] Found device: /dev/nvidia-uvm.
[ OK ] Found device: /dev/nvidia0.
[ OK ] Found device: /dev/nvidiactl.
bb0567aef16ae92397e23fb577cabf78a3821e1bb6cef9d598c7004947618eec
OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "open /proc/self/fd: no such file or directory": unknown

docker info

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 17.12.1-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9b55aab90508bd389d7654c4baf173a981477d55
runc version: 9f9c96235cc97674e935002fc3d78361b696a69e
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.4.38-rt49-tegra
Operating System: Ubuntu 16.04 LTS
OSType: linux
Architecture: aarch64
CPUs: 6
Total Memory: 6.504GiB
Name: nvidia
ID: C4UG:ILPT:DQFX:ZUZA:R4RE:45XP:KKV3:6XMX:X4CK:D56J:C4PL:BOFZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

[andrewrynhard]

@yanpeipan I am seeing the same issue in a Kubernetes cluster. Any luck finding out the root cause here?

[AndrianD]

#subscribe

[thaJeztah]

This could be a duplicate of #238, moby/moby#36467, which was a bug in Runc, and fixed through opencontainers/runc#1727 in RunC 1.0.0-rc5 (included in Docker 18.03 through moby/moby#36449).

Can you try updating to Docker 18.03.1 and see if you still run into this?

[samlyhuang]

I also see this problem

ubuntu@qa:~$ uname -a

Linux qa 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

ubuntu@qa:~$ docker -v

Docker version 18.03.0-ce, build 0520e24

ubuntu@qa:~$ sudo docker exec -it ihr360-one-click-provider -a

OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

ubuntu@qa:~$ sudo docker exec -it ihr360-one-click-provider sh

OCI runtime exec failed: exec failed: cannot exec a container that has stopped: unknown

this problem has fixed now?

[ruslanguns]

Me too.... looking forward for a solution ! thanks in advance

[eltiare]

I'm having this issue running Linux containers on Windows. Are these even checked anymore? :(

[tizki]

I also see this issue:

time="2018-08-06T05:05:48.591754629Z" level=error msg="Error running exec 673c1fc689e85c542d12188b7bec2819638bfedb9c05b894c072d1c6d5f5a7d2
in container: OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "process_linux.go:86: executing setns process caused \"exit status 21\"": unknown"

my environment details:

Centos 7 Linux 4.4.144-1.el7.elrepo.x86_64 #1 SMP Wed Jul 25 13:54:40 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker version 18.03.0-ce, build 0520e24

[rks92]

Bump. I'm getting this issue on OSX Sierra

[egrueter-dev]

Same here. Using Docker version 18.06.1-ce, build e68fc7a

[nauer]

Got this within a docker-compose run on Ubuntu 18.04:

OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: "Snakefile.BuildChoMine": executable file not found in $PATH": unknown

Using Docker version 18.06.1-ce
When I execute it from inside the container it works as usual.

[ronakganatra9]

The same issue is occurring in my Windows.

[eltiare]

I fixed this on Windows by running Docker as an administrator at least once. I don't recall if I had to do it every time though.

[ronakganatra9]

I'm having this issue running Linux containers on Windows. Are these even checked anymore? :(

same issue is in my system also.

[loomsen]

edit
not applicable

[thaJeztah]

@loomsen if that stat is inside the container; also check if the binary may be dynamically linked, and may be missing a dependency. Linux may show a no such file or directory if a dependency (linked library) is missing