Frequent restarts on k8s

[xarg]

After using imageproxy for 2 weeks it looks like it restarts a lot because of OOMKilled (out of memory errors).

Here's the number of restarts:

kubectl get pods -l app=imageproxy
NAME                          READY     STATUS    RESTARTS   AGE
imageproxy-3940674779-nm521   1/1       Running   54         12d
imageproxy-3940674779-swbgj   1/1       Running   57         12d

Here's the description of one pod:

Name:           imageproxy-3940674779-swbgj
Namespace:      default
Node:           NODE_HOST/XXXXX
Start Time:     Wed, 07 Jun 2017 18:21:42 XXXX
Labels:         app=imageproxy
                pod-template-hash=XXXX
Status:         Running
IP:             XXXXX
Controllers:    ReplicaSet/imageproxy-3940674779
Containers:
  imageproxy:
    Container ID: XXXX
    Image:              imageproxy
    Image ID:  XXXX
    Port:               80/TCP
    Args:
      -addr
      0.0.0.0:80
      -cache
      /tmp/imageproxycache
    State:              Running
      Started:          Tue, 20 Jun 2017 08:14:33 XXXX
    Last State:         Terminated
      Reason:           OOMKilled
      Exit Code:        137
      Started:          Mon, 19 Jun 2017 17:17:43 XXXX
      Finished:         Tue, 20 Jun 2017 08:14:32 XXX
    Ready:              True
    Restart Count:      57
    Limits:
      cpu:      2
      memory:   1Gi
    Requests:
      cpu:              200m
      memory:           512Mi
    Liveness:           http-get http://:80/health-check delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:          http-get http://:80/health-check delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:        <none>
Conditions:
  Type          Status
  Initialized   True
  Ready         True
  PodScheduled  True
QoS Class:      Burstable
Node-Selectors: <none>
Tolerations:    <none>
Events:         <none>

The logs are only after restart and there is no hint as to why there so much memory being used.
The upper memory limit (from the config) is 1Gi. Probably a memory leak right? Any idea how to diagnose this?

[willnorris]

likely related to #92? Try with a build from current master.

[xarg]

I've rebuilt from master, will let it run for a day or so and get back here

[xarg]

I'm still getting restarts even with the HEAD:

    State:		Running
      Started:		Thu, 22 Jun 2017 12:43:16 -XXXX
    Last State:		Terminated
      Reason:		OOMKilled
      Exit Code:	137
      Started:		Thu, 22 Jun 2017 11:03:39 -XXXX
      Finished:		Thu, 22 Jun 2017 12:43:15 -XXXX
    Ready:		True
    Restart Count:	5

It's been about 1 year or so since I touched Go so I'm not entirely up to date on how to diagnose this. Any help I can get with this would be great! Thanks.

[huguesalary]

I have the same issue with the current latest commit (e7f9017). I originally did not put memory limits on my pods, but imageproxy would end up hogging the memory.

I added limits to have imageproxy killed and now it gets killed all the time.

I start imageproxy with this: imageproxy -addr 0.0.0.0:80 -cache memory:100:4h

The memory limit in k8s is 512Mi.

Here's the state of things on my end currently:

NAME                                         READY  STATUS             RESTARTS    AGE
po/production-imageproxy-746b675cbc-5tg7z     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-6rp9r     0/1   CrashLoopBackOff   12          1h
po/production-imageproxy-746b675cbc-7h5xz     0/1   CrashLoopBackOff   10          1h
po/production-imageproxy-746b675cbc-7x8wx     0/1   CrashLoopBackOff   10          1h
po/production-imageproxy-746b675cbc-84brs     0/1   CrashLoopBackOff   8           1h
po/production-imageproxy-746b675cbc-9sqlg     0/1   CrashLoopBackOff   10          1h
po/production-imageproxy-746b675cbc-chpkb     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-hx7vl     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-j9d5c     0/1   CrashLoopBackOff   7           1h
po/production-imageproxy-746b675cbc-k2w28     0/1   CrashLoopBackOff   8           1h
po/production-imageproxy-746b675cbc-kfndx     0/1   CrashLoopBackOff   11          1h
po/production-imageproxy-746b675cbc-m4rxs     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-m6rsx     1/1   Running            8           1h
po/production-imageproxy-746b675cbc-phd82     0/1   CrashLoopBackOff   7           1h
po/production-imageproxy-746b675cbc-skmxx     1/1   Running            9           1h
po/production-imageproxy-746b675cbc-v94hv     0/1   CrashLoopBackOff   7           1h
po/production-imageproxy-746b675cbc-w46tq     1/1   Running            9           1h
po/production-imageproxy-746b675cbc-w52ks     1/1   Running            9           1h

[daohoangson]

I'm also running into this. With memory limit of 1Gi, very frequent restart. Increased to 2Gi, less frequent but still. Now at 3Gi and it restarted once after 20 minute...

[aaronpk]

When loading a web page with a lot of proxy URLs so that the browser attempts to download them all immediately, the process is killed because it's out of memory. I don't know how to get around this, since it's not something I can control in browsers. It seems like imageproxy should be rate limiting these requests at some point to avoid OOM issues.

[willnorris]

@aaronpk, yeah that sounds like a perfectly reasonable request. Tracking in #200