Docker Daemon dies due to healthcheck

[andrebrait]

Description

Background

I had this issue with 27.5.1 and now with 29.3.1 as well.
I've had a set of containers running in unRAID for a while now, but recently I noticed my services would go down randomly, often after a couple hours. Trying to docker ps resulted in a message saying it could not connect to the socket and asking whether the daemon was running.

It had indeed died.

Checking the error logs didn't reveal that much, but the debug logs gave me some hints: before the crash, I saw an unusual number of healthcheck commands being performed, particularly to my CrowdSec container; but that does not seem to be entirely correlated.

The only thing I know is that it is connected to healthchecks.

Things I tried and seemed not to have any influence in this:

1. Using only bridge networks
2. Using a macvlan network
3. Changing backing filesystem (zfs, btrfs)
4. Changing storage driver (zfs, btrfs, overlay2)
5. Upgrading from unRAID 7.2.4 (Docker 27.5.1) to 7.3.0 (Docker 29.3.1)
6. Not mounting the Docker socket to any containers
7. Changing RAM - also tested with memtest86+ 8.00 with 16 passes
8. Changing Motherboard
9. Changing my boot device

Disabling the healthchecks has gotten rid of the issue 100%. It's now been running for about a week whereas previously I got at most 24h.

Reproduce

1. docker run my containers with healthcheck
2. wait
3. it crashes

Expected behavior

Docker Daemon should not die because of a healthcheck, no matter how bad the healthcheck might be.

docker version

Client:
 Version:           29.3.1
 API version:       1.54
 Go version:        go1.25.8
 Git commit:        c2be9cc
 Built:             Wed Mar 25 16:12:39 2026
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.3.1
  API version:      1.54 (minimum version 1.40)
  Go version:       go1.25.8
  Git commit:       f78c987a
  Built:            Wed Mar 25 16:14:56 2026
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.2
  GitCommit:        301b2dac98f15c27117da5c8af12118a041a31d9
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    29.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.32.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.40.3
    Path:     /usr/lib/docker/cli-plugins/docker-compose

Server:
 Containers: 38
  Running: 38
  Paused: 0
  Stopped: 0
 Images: 32
 Server Version: 29.3.1
 Storage Driver: overlay2
  Backing Filesystem: zfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 301b2dac98f15c27117da5c8af12118a041a31d9
 runc version: v1.3.4-0-gd6d73eb
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.18.20-Unraid
 Operating System: Unraid OS 7.3 x86_64
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.58GiB
 Name: NAS
 ID: 7842bc14-0b75-4c17-af87-949347eb783b
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
 Firewall Backend: iptables

WARNING: No swap limit support

Additional Info

Also posting here some of the logs from when the crashes happened and a redacted debug log:

• debug.log (with crash at the end)
• crash_1.log
• crash_2.log

The crashes don't all seem to be that related, but they all stopped when I turned off all healthchecks here.

Here are all the containers I'm running as of today. All images pulled just now:

IMAGE                                                            COMMAND                  CREATED        STATUS        
lscr.io/linuxserver/bazarr:latest                                "/init"                  4 hours ago    Up 4 hours    
ghcr.io/immich-app/immich-server:v2.7.2                          "tini -- /bin/bash -…"   4 hours ago    Up 4 hours    
ghcr.io/immich-app/immich-machine-learning:v2.7.2-openvino       "tini -- python -m i…"   4 hours ago    Up 4 hours    
amir20/dozzle:latest                                             "/dozzle"                4 hours ago    Up 4 hours    
ghcr.io/goauthentik/server:2026.2.2                              "dumb-init -- ak ser…"   10 hours ago   Up 10 hours   
ghcr.io/goauthentik/server:2026.2.2                              "dumb-init -- ak wor…"   10 hours ago   Up 10 hours   
lscr.io/linuxserver/code-server:latest                           "/init"                  29 hours ago   Up 29 hours   
lscr.io/linuxserver/plex                                         "/init"                  29 hours ago   Up 29 hours   
lscr.io/linuxserver/qbittorrent:latest                           "/init"                  42 hours ago   Up 42 hours   
lscr.io/linuxserver/radarr:latest                                "/init"                  42 hours ago   Up 42 hours   
lscr.io/linuxserver/prowlarr:latest                              "/init"                  42 hours ago   Up 42 hours   
stirlingtools/stirling-pdf                                       "tini -- /scripts/in…"   42 hours ago   Up 42 hours   
lscr.io/linuxserver/sonarr                                       "/init"                  3 days ago     Up 3 days     
lscr.io/linuxserver/nextcloud:33.0.2                             "/init"                  5 days ago     Up 5 days     
redis:8.6.2-alpine                                               "docker-entrypoint.s…"   5 days ago     Up 5 days     
redis:8.6.2-alpine                                               "docker-entrypoint.s…"   5 days ago     Up 5 days     
gotenberg/gotenberg:8.29.1                                       "/usr/bin/tini -- go…"   5 days ago     Up 5 days     
apache/tika:3.3.0.0                                              "/bin/sh -c 'exec ja…"   5 days ago     Up 5 days     
crowdsecurity/crowdsec:v1.7.7                                    "/bin/bash /docker_s…"   5 days ago     Up 9 hours    
lscr.io/linuxserver/socket-proxy:latest                          "/docker-entrypoint.…"   5 days ago     Up 5 days     
redis:8.6.2-alpine                                               "docker-entrypoint.s…"   5 days ago     Up 42 hours   
redis:8.6.2-alpine                                               "docker-entrypoint.s…"   5 days ago     Up 5 days     
collabora/code:25.04.9.4.1                                       "/start-collabora-on…"   5 days ago     Up 5 days     
traefik:3.6.12                                                   "/entrypoint.sh trae…"   10 days ago    Up 42 hours   
ghcr.io/seerr-team/seerr:latest                                  "docker-entrypoint.s…"   11 days ago    Up 6 days     
santiagosayshey/profilarr:latest                                 "/entrypoint.sh guni…"   11 days ago    Up 6 days     
lscr.io/linuxserver/lidarr:latest                                "/init"                  11 days ago    Up 6 days     
flaresolverr/flaresolverr                                        "/usr/bin/dumb-init …"   11 days ago    Up 6 days     
ghcr.io/paperless-ngx/paperless-ngx:2.20.13                      "/init"                  11 days ago    Up 6 days     
postgres:16.13-alpine                                            "docker-entrypoint.s…"   11 days ago    Up 6 days     
postgres:16.13-alpine                                            "docker-entrypoint.s…"   11 days ago    Up 6 days     
ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0   "/usr/local/bin/immi…"   11 days ago    Up 6 days     
valkey/valkey:9                                                  "docker-entrypoint.s…"   11 days ago    Up 6 days     
postgres:16.13-alpine                                            "docker-entrypoint.s…"   11 days ago    Up 6 days     
maxmindinc/geoipupdate:latest                                    "/usr/bin/entry.sh"      11 days ago    Up 6 days     
qmcgaw/gluetun:v3                                                "/gluetun-entrypoint"    11 days ago    Up 6 days     
ghcr.io/jarvis2f/telegram-files:latest                           "/sbin/tini -- /bin/…"   2 weeks ago    Up 6 days     
c4illin/convertx:latest                                          "bun run dist/src/in…"   2 weeks ago    Up 6 days     

[thaJeztah]

Thanks for reporting, and for the (debug) logs!

This is the failure from the debug log, which .. at a quick glance looks like some potential race and/or missing nil-check; might be an easy fix (famous last words), but we should look if it's "expected" or something deeper is at hand (avoid tapering over the root-cause).

time="2026-03-25T16:58:52.203138295+01:00" level=debug msg="attach: stderr: begin"
time="2026-03-25T16:58:52.204999464+01:00" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/exec-added
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x312f49b]

goroutine 239571 [running]:
google.golang.org/grpc.encode({0x0?, 0x0?}, {0x855280?, 0xc00178da90?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:804 +0x3b
google.golang.org/grpc.prepareMsg({0x855280?, 0xc00178da90?}, {0x0?, 0x0?}, {0x0, 0x0}, {0x0, 0x0}, {0x1051e08, 0xc000131680})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:1881 +0xe5
google.golang.org/grpc.(*clientStream).SendMsg(0xc001c087e0, {0x855280, 0xc00178da90})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:948 +0xf1
google.golang.org/grpc.invoke({0x106d248?, 0xc002aef2f0?}, {0xba8011?, 0x2998065?}, {0x855280, 0xc00178da90}, {0x7feae0, 0xc002aef1d0}, 0x2a3e425?, {0xc000616a60, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/call.go:70 +0x99
github.com/containerd/containerd/v2/client.namespaceInterceptor.unary({{0xaf1fef?, 0x5063c60?}}, {0x106d248, 0xc002aef1a0}, {0xba8011, 0x29}, {0x855280, 0xc00178da90}, {0x7feae0, 0xc002aef1d0}, ...)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/v2/client/grpc.go:35 +0x108
google.golang.org/grpc.getChainUnaryInvoker.func1({0x106d248, 0xc002aef1a0?}, {0xba8011, 0x2998065?}, {0x855280?, 0xc00178da90?}, {0x7feae0?, 0xc002aef1d0?}, 0x312354e?, {0xc000616a60, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:544 +0x1eb
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor({0x106d248?, 0xc002aef1a0?}, {0xba8011?, 0xb13c08?}, {0x855280?, 0xc00178da90?}, {0x7feae0?, 0xc002aef1d0?}, 0x2997634?, 0xc002aef200, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/grpcerrors/intercept.go:41 +0x7e
google.golang.org/grpc.NewClient.chainUnaryClientInterceptors.func1({0x106d248, 0xc002aef1a0}, {0xba8011, 0x29}, {0x855280, 0xc00178da90}, {0x7feae0, 0xc002aef1d0}, 0xc0007b4008, 0xc77020, ...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:532 +0x10f
google.golang.org/grpc.(*ClientConn).Invoke(0xc0007b4008, {0x106d248?, 0xc002aef1a0?}, {0xba8011?, 0xc00088a0a0?}, {0x855280?, 0xc00178da90?}, {0x7feae0?, 0xc002aef1d0?}, {0x0, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/call.go:35 +0x205
github.com/containerd/containerd/api/services/tasks/v1.(*tasksClient).Start(0xc00088a0a0, {0x106d248, 0xc002aef1a0}, 0xc00178da90, {0x0, 0x0, 0x0})
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/tasks/v1/tasks_grpc.pb.go:70 +0xc8
github.com/containerd/containerd/v2/client.(*process).Start(0xc002aef170, {0x106d280, 0xc001a21310})
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/v2/client/process.go:129 +0x268
github.com/moby/moby/v2/daemon/internal/libcontainerd/remote.(*task).Exec(0xc001500660, {0x106d280, 0xc001a21310}, {0xc000860100, 0x40}, 0xc00112a100, 0x0, 0xc001281450)
	/go/src/github.com/docker/docker/daemon/internal/libcontainerd/remote/client.go:295 +0x588
github.com/moby/moby/v2/daemon.(*Daemon).ContainerExecStart(0xc000295908, {0x106d280, 0xc001a21310}, {0xc000860100?, 0x40?}, {{0x0, 0x0}, {0x1037440, 0xc00130e2c0}, {0x1037440, ...}, ...})
	/go/src/github.com/docker/docker/daemon/exec.go:308 +0x10fd
github.com/moby/moby/v2/daemon.(*cmdProbe).run.func1()
	/go/src/github.com/docker/docker/daemon/health.go:106 +0x57
created by github.com/moby/moby/v2/daemon.(*cmdProbe).run in goroutine 239450
	/go/src/github.com/docker/docker/daemon/health.go:106 +0x8df

crash2.log seems to be the same (at a glance);

Details

time="2026-03-26T08:19:08.111687178+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T08:19:51.001518805+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T08:42:42.050055350+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T08:55:40.595070007+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T09:03:19.634932116+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T09:12:50.159259590+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x95b43b]

goroutine 71733 [running]:
google.golang.org/grpc.encode({0x0?, 0x0?}, {0x1a11360?, 0xc000d35380?})
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/rpc_util.go:722 +0x3b
google.golang.org/grpc.(*Server).sendResponse(0xc0003be400, {0x1e99b88, 0xc000d35110}, 0xc00141b500, {0x1a11360, 0xc000d35380}, {0x0, 0x0}, 0xc0005e4429, {0x0, ...})
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1172 +0xa5
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0003be400, {0x1e99b88, 0xc000d34f90}, 0xc00141b500, 0xc00076e7e0, 0x2cdf160, 0x0)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1478 +0x1125
google.golang.org/grpc.(*Server).handleStream(0xc0003be400, {0x1e9a510, 0xc0008a2340}, 0xc00141b500)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1842 +0xb95
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1061 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 85
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1072 +0x11d
time="2026-03-26T09:36:43.388479358+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T09:40:04.938903964+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x312f49b]

goroutine 406187 [running]:
google.golang.org/grpc.encode({0x0?, 0x0?}, {0x97ed60?, 0xc000dad5f0?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/rpc_util.go:804 +0x3b
google.golang.org/grpc.prepareMsg({0x97ed60?, 0xc000dad5f0?}, {0x0?, 0x0?}, {0x0, 0x0}, {0x0, 0x0}, {0x1051e08, 0xc00032b4c0})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:1881 +0xe5
google.golang.org/grpc.(*clientStream).SendMsg(0xc002123440, {0x97ed60, 0xc000dad5f0})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/stream.go:948 +0xf1
google.golang.org/grpc.invoke({0x106d248?, 0xc002d81ad0?}, {0xba2b79?, 0x10?}, {0x97ed60, 0xc000dad5f0}, {0x7a27c0, 0xc002d81830}, 0xa76480?, {0xc0005a0280, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/call.go:70 +0x99
github.com/containerd/containerd/v2/client.namespaceInterceptor.unary({{0xaf1fef?, 0x14d2b8827cc8?}}, {0x106d248, 0xc002d816e0}, {0xba2b79, 0x28}, {0x97ed60, 0xc000dad5f0}, {0x7a27c0, 0xc002d81830}, ...)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/v2/client/grpc.go:35 +0x108
google.golang.org/grpc.getChainUnaryInvoker.func1({0x106d248, 0xc002d816e0?}, {0xba2b79, 0x2998065?}, {0x97ed60?, 0xc000dad5f0?}, {0x7a27c0?, 0xc002d81830?}, 0x312354e?, {0xc0005a0280, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:544 +0x1eb
github.com/moby/buildkit/util/grpcerrors.UnaryClientInterceptor({0x106d248?, 0xc002d816e0?}, {0xba2b79?, 0xc001b60300?}, {0x97ed60?, 0xc000dad5f0?}, {0x7a27c0?, 0xc002d81830?}, 0x2997634?, 0xc002d81860, ...)
	/go/src/github.com/docker/docker/vendor/github.com/moby/buildkit/util/grpcerrors/intercept.go:41 +0x7e
google.golang.org/grpc.NewClient.chainUnaryClientInterceptors.func1({0x106d248, 0xc002d816e0}, {0xba2b79, 0x28}, {0x97ed60, 0xc000dad5f0}, {0x7a27c0, 0xc002d81830}, 0xc00093e008, 0xc77020, ...)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:532 +0x10f
google.golang.org/grpc.(*ClientConn).Invoke(0xc00093e008, {0x106d248?, 0xc002d816e0?}, {0xba2b79?, 0xc002b139a0?}, {0x97ed60?, 0xc000dad5f0?}, {0x7a27c0?, 0xc002d81830?}, {0x0, ...})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/call.go:35 +0x205
github.com/containerd/containerd/api/services/tasks/v1.(*tasksClient).Exec(0xc002b139a0, {0x106d248, 0xc002d816e0}, 0xc000dad5f0, {0x0, 0x0, 0x0})
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/api/services/tasks/v1/tasks_grpc.pb.go:124 +0xc8
github.com/containerd/containerd/v2/client.(*task).Exec(0xc00077d600, {0x106d280, 0xc0026cd450}, {0xc001978340, 0x40}, 0xc001b60300, 0xc002d816b0)
	/go/src/github.com/docker/docker/vendor/github.com/containerd/containerd/v2/client/task.go:457 +0x4ee
github.com/moby/moby/v2/daemon/internal/libcontainerd/remote.(*task).Exec(0xc0014ff5c0, {0x106d280, 0xc0026cd450}, {0xc001978340, 0x40}, 0xc001b60300, 0x0, 0xc002b13910)
	/go/src/github.com/docker/docker/daemon/internal/libcontainerd/remote/client.go:277 +0x2fb
github.com/moby/moby/v2/daemon.(*Daemon).ContainerExecStart(0xc0002ff408, {0x106d280, 0xc0026cd450}, {0xc001978340?, 0x40?}, {{0x0, 0x0}, {0x1037440, 0xc002da4c80}, {0x1037440, ...}, ...})
	/go/src/github.com/docker/docker/daemon/exec.go:308 +0x10fd
github.com/moby/moby/v2/daemon.(*cmdProbe).run.func1()
	/go/src/github.com/docker/docker/daemon/health.go:106 +0x57
created by github.com/moby/moby/v2/daemon.(*cmdProbe).run in goroutine 406074
	/go/src/github.com/docker/docker/daemon/health.go:106 +0x8df

crash1.log looks more ... interesting; I'd have to dig if this is an error we produce, or if it's (e.g.) a Go runtime error; panic: id (473) <= evictCount (549) doesn't immediately ring a bell and the stacktrace may mean it's a bug in the Go runtime? (Maybe it's one of the issues fixed in go1.26.2)

panic: id (473) <= evictCount (549)

goroutine 127 [running]:
golang.org/x/net/http2/hpack.(*headerFieldTable).idToIndex(0xc0001c0880, 0xc0007032f0?)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/tables.go:118 +0xbd
golang.org/x/net/http2/hpack.(*headerFieldTable).search(0xc0001c0880, {{0x900161, 0x5}, {0x996592, 0x28}, 0x0})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/tables.go:105 +0xe5
golang.org/x/net/http2/hpack.(*Encoder).searchTable(0xc0001c0880, {{0x900161, 0x5}, {0x996592, 0x28}, 0x0})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/encode.go:97 +0x85
golang.org/x/net/http2/hpack.(*Encoder).WriteField(0xc0001c0880, {{0x900161, 0x5}, {0x996592, 0x28}, 0x0})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/encode.go:62 +0x145
google.golang.org/grpc/internal/transport.(*loopyWriter).writeHeader(0xc0001c0980, 0x4346d, 0x0, {0xc000daf448?, 0x10?, 0x5f6a60?}, 0x100000001?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:720 +0x119
google.golang.org/grpc/internal/transport.(*loopyWriter).originateStream(0xc0001c0980, 0xc0016cd040, 0xc00276b2c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:707 +0x6a
google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler(0xc0001c0980, 0xc00276b2c0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:693 +0x298
google.golang.org/grpc/internal/transport.(*loopyWriter).handle(0xc000780840?, {0x462e80?, 0xc00276b2c0?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:872 +0x211
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0001c0980)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:579 +0x9f
google.golang.org/grpc/internal/transport.newHTTP2Client.func6()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:470 +0xd2
created by google.golang.org/grpc/internal/transport.newHTTP2Client in goroutine 88
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:468 +0x2472

[thaJeztah]

Looks like that last one has an open ticket in the Go issue tracker; also mentions gRPC involved;

• x/net/http2/hpack: "id (106) <= evictCount (117)" panic in headerFieldTable.idToIndex golang/go#69316

[thaJeztah]

Hm... actually the other ones seem to be happening somewhere in the gRPC stack

[andrebrait]

Would it be easier if I were to detail all the networking part (custom networks, usage of DNS to resolve the address of other containers, the healthcheck commands of each image and/or compose, etc.)?

[thaJeztah]

At a first glance, I don't immediately suspect it's due to how it's used. I wonder if there's an issue with the containerd stack; one thing I notice is that a file involved has a build-tag, and I seem to recall containerd made changes for the protobuf implementations;

moby/vendor/github.com/containerd/containerd/api/services/tasks/v1/tasks_grpc.pb.go

Line 1 in d80e341

//go:build !no_grpc

other implementation fro the same;

moby/vendor/github.com/containerd/containerd/api/services/tasks/v1/tasks_ttrpc.pb.go

Lines 1 to 3 in d80e341

	// Code generated by protoc-gen-go-ttrpc. DO NOT EDIT.
	// source: github.com/containerd/containerd/api/services/tasks/v1/tasks.proto
	package tasks

Wondering if there's any discrepancy there (which could also relate to the Unimplemented errors showing up;

time="2026-03-26T08:19:08.111687178+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events" time="2026-03-26T08:19:51.001518805+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T08:42:42.050055350+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events" time="2026-03-26T08:55:40.595070007+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-03-26T09:03:19.634932116+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events" time="2026-03-26T09:12:50.159259590+01:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"

cc @dmcgowan @cpuguy83 @vvoland any thoughts at a quick glance?

[andrebrait]

But when were those changes made? Because I've been having these crashes in both Docker 27.5.1 and 29.3.1.

I also have a Debian VM running Portainer and I've never had any issues with it, though they do run very different sets of containers.

[thaJeztah]

I think containerd/ttrpc 1.2 was when some of the rewrites landed; containerd/ttrpc#99

which was already used in docker 27.5.1;
https://github.com/moby/moby/blob/v27.5.1/vendor.mod#L149

[andrebrait]

Adding the latest crash I just got today.
After multiple days of uptime, I fired up musicseer and that has a HEALTHCHECK layer, as seen in its Dockerfile.

Note that the container was using another container (Gluetun) for its networking.

Now, interestingly, the crash happened exactly when I moved the container in unRAID to place it after Gluetun (so logically Gluetun would start first). Now, I think this is perhaps just a coincidence, since I wasn't doing that when I faced those crashes in the past.

But who knows, the stack trace is a bit different this time.

Log file: docker.log

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x95c8fb]

goroutine 119380 [running]:
google.golang.org/grpc.encode({0x0?, 0x0?}, {0x1a22960?, 0xc001585ac0?})
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/rpc_util.go:804 +0x3b
google.golang.org/grpc.(*Server).sendResponse(0xc00046d688, {0x1eab8e8, 0xc000fbbb30}, 0xc000fbc680, {0x1a22960, 0xc001585ac0}, {0x0, 0x0}, 0xc000012769, {0x0, ...})
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1174 +0x9f
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00046d688, {0x1eab8e8, 0xc000fbba10}, 0xc000fbc680, 0xc000665080, 0x2d00428, 0x0)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1475 +0x12c5
google.golang.org/grpc.(*Server).handleStream(0xc00046d688, {0x1eac2b0, 0xc0003b36c0}, 0xc000fbc680)
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1832 +0xdc6
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1063 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 62
	/go/src/github.com/containerd/containerd/vendor/google.golang.org/grpc/server.go:1074 +0x11d
nohup: ignoring input
nohup: ignoring input
time="2026-04-13T07:10:52.585381646+02:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-04-13T08:14:57.450853224+02:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
time="2026-04-13T19:47:50.497998662+02:00" level=error msg="post event" error="rpc error: code = Unimplemented desc = service containerd.services.events.ttrpc.v1.Events"
panic: id (480) <= evictCount (519)

goroutine 74 [running]:
golang.org/x/net/http2/hpack.(*headerFieldTable).idToIndex(0xc000cc4080, 0xc001404840?)
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/tables.go:118 +0xbd
golang.org/x/net/http2/hpack.(*headerFieldTable).search(0xc000cc4080, {{0xaf3b0b, 0x5}, {0xbd1cc5, 0x31}, 0x0})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/tables.go:105 +0xe5
golang.org/x/net/http2/hpack.(*Encoder).searchTable(0xc000cc4080, {{0xaf3b0b, 0x5}, {0xbd1cc5, 0x31}, 0x0})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/encode.go:97 +0x74
golang.org/x/net/http2/hpack.(*Encoder).WriteField(0xc000cc4080, {{0xaf3b0b, 0x5}, {0xbd1cc5, 0x31}, 0x0})
	/go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/hpack/encode.go:62 +0x125
google.golang.org/grpc/internal/transport.(*loopyWriter).writeHeader(0xc0007b80a0, 0x3f8dd, 0x0, {0xc000dc0588?, 0x10?, 0x75d000?}, 0xc000c50101?)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:747 +0x113
google.golang.org/grpc/internal/transport.(*loopyWriter).originateStream(0xc0007b80a0, 0xc002628120, 0xc001c6a140)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:734 +0x69
google.golang.org/grpc/internal/transport.(*loopyWriter).headerHandler(0xc0007b80a0, 0xc001c6a140)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:720 +0x1ee
google.golang.org/grpc/internal/transport.(*loopyWriter).handle(0xc001516040?, {0x517d60?, 0xc001c6a140?})
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:902 +0x211
google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0007b80a0)
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:606 +0x99
google.golang.org/grpc/internal/transport.NewHTTP2Client.func6()
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:469 +0xd2
created by google.golang.org/grpc/internal/transport.NewHTTP2Client in goroutine 226
	/go/src/github.com/docker/docker/vendor/google.golang.org/grpc/internal/transport/http2_client.go:467 +0x244b