After one dockerd crash, systemd restarted Docker, and the new daemon stopped all 46 running containers and flagged each hasBeenManuallyStopped=true. 11 came back on their own. The other 35 stayed off until I started them by hand.
Request: enable live-restore by default, or restart containers that were running before a daemon failure.
I suspect the crash itself is my hardware (see below). The aftermath looks like a ZimaOS default that turns any daemon failure into a major outage.
Environment
| Item | Detail |
|---|---|
| ZimaOS | v1.8.0 |
| Kernel | 6.18.9 |
| Docker | 27.5.1 (overlay2, nvidia runtime via systemd drop-in) |
| CPU | AMD Threadripper 1950X (32 threads) |
| RAM | 47 GiB, plus 15 GiB swap |
| Containers | 50 total, 46 running at the time of the crash |
| daemon.json | Empty ({}), not modified by me |
| Live restore | LiveRestore=false |
What happened (Oct 30, local time)
| Time | Event |
|---|---|
| 04:35:39 | dockerd crashed: SIGILL: illegal instruction, exit status 2/INVALIDARGUMENT (41 min CPU, 13.3 GB peak memory) |
| 04:35:41 | systemd scheduled the restart |
| 04:35:44 | New dockerd began loading containers. For each of the 46 it logged ShouldRestart failed ... error="restart canceled" hasBeenManuallyStopped=true |
| 04:35:54 | Six containers: failed to exit within 10s of signal 15 - using the force |
| 04:36:04 | 11 containers restarted automatically. Docker reported started at 04:36:05. |
| 04:40:12 | I started the remaining 35 by hand. Nothing alerted me. |
Restart policies work in general: after a full reboot on Oct 29, nearly everything came back by itself.
Log line from the new daemon, one per container (container ID removed):
dockerd[2662390]: level=warning msg="ShouldRestart failed, container will not be restarted" container=<id> daemonShuttingDown=false error="restart canceled" execDuration=22h29m15s hasBeenManuallyStopped=true restartCount=0
Why most containers stayed off
Without live-restore, a daemon restart stops every container and sets the manually-stopped flag. Docker’s unless-stopped policy then skips flagged containers, while always restarts them on daemon start. That matches what I saw: 11 recovered, 35 did not. [I’ll confirm which policy each group uses and add the list below.]
Result: any dockerd crash takes down most apps, with no automatic recovery and no alert.
Request
- Enable
"live-restore": truein the default daemon config, so containers survive a daemon crash or restart. - Or add a post-start step that restarts containers that were running before the daemon went down.
- If edits to
/etc/docker/daemon.jsonaren’t preserved across updates or reboots, please say where a persistent override should go.
Check if you’re affected
docker info --format '{{.LiveRestoreEnabled}}'
false means any Docker crash or restart takes down your apps.
About the crash itself
dockerd died inside runtime.memmove, called from its gRPC connection to containerd. The bytes at the faulting address are ordinary x86-64 (jb, jbe, cmp rbx,8), so this is not a missing CPU instruction.
Starting an hour earlier, unrelated programs faulted across at least six different cores. That points to unstable hardware (RAM, CPU, or power) on my side, which I’m testing. I’m not asking ZimaOS to fix the SIGILL, but pointers from anyone who has seen this on Threadripper 1000-series are welcome.
[details=“dmesg faults, 03:00-05:00 (click to expand)”]
| Time | Program | Fault | CPU |
|---|---|---|---|
| 03:18:10 | .NET (libcoreclr.so) | general protection fault | n/a |
| 04:05:11 | av1 decoder (libc.so.6) | segfault | 29 |
| 04:05:11 | av1 decoder (libc.so.6) | segfault | 12 |
| 04:09:02 | av1 decoder (libc.so.6) | segfault | 5 |
| 04:15:43 | matroska demuxer (libavformat) | segfault | 27 |
| 04:15:43 | matroska demuxer (libavformat) | segfault | 9 |
| 04:37:57 | .NET (AMP_Linux_x86_64) | invalid opcode | n/a |
| 04:40:25 | .NET (libclrjit.so) | segfault | 7 |
| 04:41:43 | h264 decoder (libnvcuvid.so) | invalid opcode | n/a |
| 04:56:46 | .NET (libcoreclr.so) | general protection fault | n/a |
[/details]
[details=“dockerd crash trace (click to expand)”]
dockerd[3584]: SIGILL: illegal instruction
dockerd[3584]: PC=0x14f1a40 m=237 sigcode=2
dockerd[3584]: instruction bytes: 0xf 0x82 0xd 0x1 0x0 0x0 0xf 0x86 0x2 0x1 0x0 0x0 0x48 0x83 0xfb 0x8
goroutine 125 [running]:
runtime.memmove() runtime/memmove_amd64.s:59
bufio.(*Reader).Read bufio/bufio.go:258
golang.org/x/net/http2.(*Framer).ReadFrame
google.golang.org/grpc/internal/transport.(*http2Client).reader http2_client.go:1645
[/details]
Separate issue: BTRFS errors on the 12TB drive
| Counter | Value |
|---|---|
| write / read / flush errors | 0 |
| corruption_errs | 7438 (and rising) |
| generation_errs | 0 |
The counter was already 7125 at 04:12, so the errors predate the crash by a long way. They continue at about 5 per minute, always at the same two spots (inode 273896 at offset 2441891840, inode 274304 at offset 2926067712), which something re-reads about once a minute. The data profile is single, so BTRFS has no second copy to repair from. The filesystem is 98.4% full with 1 MiB unallocated. I don’t believe this is related to the Docker issue.
Attachments
docker-journal-excerpt.txt: crash trace, restart, and container shutdown (the full journal is 17,900 lines, mostly goroutine dumps)incident-logs-full.tar.gz: full docker and containerd journals plus the kernel log for 03:00-05:00
Happy to provide more logs or run diagnostics.