Processes are running slow in docker container on ubuntu 18.04

[Aayush04]

• This is a bug report
• This is a feature request
• I searched existing issues before opening this one

Processes are taking time when running them in a docker container on ubuntu 18 machine. But the same process with the same docker version is running fine on ubuntu 16 machine.

I have a node application listening on some port. Accepting get requests on the path "/" and "/docker" which simply runs a command "whoami" in the host machine and in docker container respectively and returns the result. The same node application with the same docker container is running on both the machines (ubuntu16 and ubuntu18).

Now, First I tried sending 20 concurrent get request with path "/" to both the machines. And both the machines executed the command in avg of
35-40ms. With this, I concluded that the spawning process without docker has the same behaviour.

Now, Second I tried sending 20 concurrent get request with path "/docker" to both the machines. Here, ubuntu16 machine took Max 4.3 seconds and Avg 3 seconds. But ubuntu18 machine taking Max 10seconds and Avg 9 seconds.

I tried the above test multiple times. and concluded when running processes concurrently inside docker the time taken to execute is almost double in ubuntu18 machine as compare to ubuntu16.

Output of docker version:

Client:
 Version:           18.09.8
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        0dd43dd87f
 Built:             Wed Jul 17 17:40:56 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.8
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.8
  Git commit:       0dd43dd
  Built:            Wed Jul 17 17:07:25 2019
  OS/Arch:          linux/amd64
  Experimental:     false

Output of docker info which are common to both machine:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 18.09.8
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.296GiB
Name: myhostname
ID: LLLO:OMTS:PNNM:T3MP:AD2F:UMDG:IIZK:OGBO:3ZLL:YDBX:ONAO:AY5G
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 27
 Goroutines: 42
 System Time: 2019-07-25T15:25:54.991694211+05:30
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support

Output of docker info which specific to ubuntu16:

Kernel Version: 4.4.0-112-generic
Operating System: Ubuntu 16.04.3 LTS
Total Memory: 7.303GiB
ID: FOFI:RW7N:RZSP:HHKH:BKS3:LMWL:TC2J:W7V2:222Y:Q2AU:XMU3:KLU7

Output of docker info which specific to ubuntu18:

Kernel Version: 4.15.0-1040-aws
Operating System: Ubuntu 18.04.2 LTS
Total Memory: 7.296GiB
ID: LLLO:OMTS:PNNM:T3MP:AD2F:UMDG:IIZK:OGBO:3ZLL:YDBX:ONAO:AY5G

Additional environment details (AWS, VirtualBox, physical, etc.)

Machine1: Giving better performance.
node    v10.16.0
docker  18.09.8
ubuntu  16.04.3 LTS, xenial
AWS instance type c4.xlarge

Machine2: Giving poor performance.
node    v10.16.0
docker  18.09.8
ubuntu  18.04.2 LTS, bionic
AWS instance type c4.xlarge

docker image

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
aa0a31cd86a8        ubuntu:18.04        "bash"              2 days ago          Up 25 hours                             ubuntu

The command I am running in docker

docker exec ubuntu whoami

ubuntu18 machine Data:

1. Data of time taken in execution

2019-07-25 14:07:32.559 INFO  uid: 715a6af0-aeb7-11e9-a5a9-2fffd4e800d1 time: 1008 result: {"success":true,"data":"root"}
2019-07-25 14:07:32.941 INFO  uid: 7178c860-aeb7-11e9-a5a9-2fffd4e800d1 time: 1191 result: {"success":true,"data":"root"}
2019-07-25 14:07:40.363 INFO  uid: 71767e70-aeb7-11e9-a5a9-2fffd4e800d1 time: 8628 result: {"success":true,"data":"root"}
.
.
.
2019-07-25 14:07:41.970 INFO  uid: 718af0d0-aeb7-11e9-a5a9-2fffd4e800d1 time: 10101 result: {"success":true,"data":"root"}

2. logs of command strace -t docker exec ubuntu whoami

3. result of perf top --sort comm,dso

So, I need help in debugging what is wrong with docker on the ubuntu18 machine. Or if there is any limitation with docker on ubuntu18 or maybe some machine or docker configuration issue.

[mbdas]

Aayush did you find anything? We are starting to see something similar on latest Ubuntu 16 update with some 4.15 kernel and updated packages (could be same as ubuntu 18)

[crosbymichael]

Seems to be something in the docker engine causing this, not runc or containerd.

Test with Docker:

(for i in `seq 1 10`; do (time sudo docker exec test true &); done)
michael|~/development/gocode/src/github.com/containerd/containerd >
real    0m0.241s
user    0m0.031s
sys     0m0.008s

real    0m0.435s
user    0m0.026s
sys     0m0.012s

========> LONG PAUSE HERE BEFORE REST OF OUTPUT

real    0m1.887s
user    0m0.034s
sys     0m0.011s

real    0m1.907s
user    0m0.030s
sys     0m0.013s

real    0m1.923s
user    0m0.030s
sys     0m0.014s

real    0m1.940s
user    0m0.031s
sys     0m0.008s

real    0m1.961s
user    0m0.026s
sys     0m0.015s

real    0m1.987s
user    0m0.029s
sys     0m0.011s

real    0m2.003s
user    0m0.032s
sys     0m0.009s

real    0m2.025s
user    0m0.030s
sys     0m0.006s

With Containerd:

(for i in `seq 1 10`; do (time sudo ctr t  exec --exec-id $i sh true & ); done)
michael|~/development/gocode/src/github.com/containerd/containerd >
real    0m0.329s
user    0m0.015s
sys     0m0.019s

real    0m0.327s
user    0m0.011s
sys     0m0.018s

real    0m0.336s
user    0m0.011s
sys     0m0.017s

real    0m0.337s
user    0m0.012s
sys     0m0.018s

real    0m0.350s
user    0m0.019s
sys     0m0.011s

real    0m0.354s
user    0m0.010s
sys     0m0.017s

real    0m0.357s
user    0m0.014s
sys     0m0.014s

real    0m0.365s
user    0m0.012s
sys     0m0.015s

real    0m0.377s
user    0m0.018s
sys     0m0.015s

real    0m0.372s
user    0m0.015s
sys     0m0.012s

[Aayush04]

Hello @mbdas, I found that when I installed docker via binary. That is if I install docker from the static binaries as mentioned in this link I started getting better performance as one in ubuntu 16.

But when I am installing the docker through apt-get command as mentioned here, then I am getting poor performance. I tried this test with docker 18 and 19 both but found the same behaviour.

Still, I did not get any clue for whats is the difference between installation via apt-get vs installation via static binaries.

[mbdas]

An engineer in the team found the issue. libseccomp2 was causing this. Once we downgraded the lib, it was back to normal. docker needs to look at what changed in this lib.

[severfire]

I do have similar issue. Ubuntu 19.04. Docker version 19.03.2.
WordPress container runs slow. Found out that disabling WordFence helped.
Yet still funny, cause on production it works good.
Damn strange :-)

[alcaamado]

@mbdas What is the version you have installed?

[mbdas]

2.2.3-3ubuntu3 . We have pinned this version for now.

[pfalcon]

I now see a case when a simple "docker build" with a Dockerfile containing "apt-get install" for a dozen packages runs for 20 minutes before hanging completely. Just reducing that to:

FROM ubuntu:18.04
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update
RUN apt-get -y install shared-mime-info

Leads to:

Setting up libicu60:amd64 (60.2-3ubuntu3.1) ...
Setting up libglib2.0-0:amd64 (2.56.4-0ubuntu0.18.04.6) ...
No schema files found: doing nothing.
Setting up libxml2:amd64 (2.9.4+dfsg1-6.1ubuntu1.3) ...
Setting up libglib2.0-data (2.56.4-0ubuntu0.18.04.6) ...
Setting up shared-mime-info (1.9-2) ...
^C
real	4m39.560s
user	0m0.136s
sys	0m0.065s