On my ZimaOS system, after a while, it suddenly stops working; the control panel freezes. The web panel freezes, and neither login nor the SSH terminal works. Docker containers continue to function. The system returns to normal after a restart, but after a while the problem returns. Does anyone know anything about this? I’m using version 1.6.1.
Yeah that sounds like the system itself is still running (since your containers stay up), but the WebUI and core services are freezing.
Quick things to check:
- can you still ping the Zima when it locks up?
- any response locally on the machine or completely frozen?
Feels like a 1.6.x issue rather than Docker.
If you can, grab logs after a reboot so we can see what’s happening:
journalctl -xe
dmesg | tail -100
Without logs it’s a bit of a guess, but I’ve seen similar where the UI layer crashes while Docker keeps going.
If it keeps happening, worth testing with fewer containers or even rolling back just to confirm it’s version related.
Yes, I can ping it!
It only blocks access to the web panel and SSH.
Follow the log: Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F62696E2F646F636B657264002D2D686F73743D66643A2F2F002D2D6164642D72756E74696D653D6E76696469613D2F7573722F62696E2F6E76696469612D636F6E7461696E65722D72756E74696D65
Apr 24 10:12:31 ZimaOS NetworkManager[1021]: <info> [1777036351.2245] manager: (vethceded42): new Veth device (/org/freedesktop/NetworkManager/Devices/9)
Apr 24 10:12:31 ZimaOS (udev-worker)[2333]: Network interface NamePolicy= disabled on kernel command line.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS audit: ANOM_PROMISCUOUS dev=veth8eae157 prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
Apr 24 10:12:31 ZimaOS systemd-networkd[714]: vethceded42: Link UP
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS kernel: br-f17c48621b87: port 1(veth8eae157) entered blocking state
Apr 24 10:12:31 ZimaOS kernel: br-f17c48621b87: port 1(veth8eae157) entered disabled state
Apr 24 10:12:31 ZimaOS kernel: veth8eae157: entered allmulticast mode
Apr 24 10:12:31 ZimaOS kernel: veth8eae157: entered promiscuous mode
Apr 24 10:12:31 ZimaOS audit[1574]: SYSCALL arch=c000003e syscall=44 success=yes exit=40 a0=c a1=c0005a6a20 a2=28 a3=0 items=0 ppid=1 pid=1574 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F62696E2F646F636B657264002D2D686F73743D66643A2F2F002D2D6164642D72756E74696D653D6E76696469613D2F7573722F62696E2F6E76696469612D636F6E7461696E65722D72756E74696D65
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-networkd[714]: veth8eae157: Link UP
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS NetworkManager[1021]: <info> [1777036351.2273] manager: (vethb11ebed): new Veth device (/org/freedesktop/NetworkManager/Devices/10)
Apr 24 10:12:31 ZimaOS NetworkManager[1021]: <info> [1777036351.2288] manager: (veth8eae157): new Veth device (/org/freedesktop/NetworkManager/Devices/11)
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS NetworkManager[1021]: <info> [1777036351.2525] manager: (vetha152749): new Veth device (/org/freedesktop/NetworkManager/Devices/12)
Apr 24 10:12:31 ZimaOS (udev-worker)[2335]: Network interface NamePolicy= disabled on kernel command line.
Apr 24 10:12:31 ZimaOS audit: ANOM_PROMISCUOUS dev=vethfd8c2ed prom=256 old_prom=0 auid=4294967295 uid=0 gid=0 ses=4294967295
Apr 24 10:12:31 ZimaOS NetworkManager[1021]: <info> [1777036351.2537] manager: (vethfd8c2ed): new Veth device (/org/freedesktop/NetworkManager/Devices/13)
Apr 24 10:12:31 ZimaOS audit[1574]: SYSCALL arch=c000003e syscall=44 success=yes exit=40 a0=c a1=c000506600 a2=28 a3=0 items=0 ppid=1 pid=1574 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="dockerd" exe="/usr
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F62696E2F646F636B657264002D2D686F73743D66643A2F2F002D2D6164642D72756E74696D653D6E76696469613D2F7573722F62696E2F6E76696469612D636F6E7461696E65722D72756E74696D65
Apr 24 10:12:31 ZimaOS kernel: docker0: port 2(vethfd8c2ed) entered blocking state
Apr 24 10:12:31 ZimaOS kernel: docker0: port 2(vethfd8c2ed) entered disabled state
Apr 24 10:12:31 ZimaOS kernel: vethfd8c2ed: entered allmulticast mode
Apr 24 10:12:31 ZimaOS kernel: vethfd8c2ed: entered promiscuous mode
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-networkd[714]: vethfd8c2ed: Link UP
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS audit[2387]: NETFILTER_CFG table=nat family=2 entries=21 op=xt_replace pid=2387 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2387]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=55e34433e8d0 items=0 ppid=1574 pid=2387 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100444F434B4552002D7000746370002D6400302F30002D2D64706F72740038303830002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E31372E302E323A383
Apr 24 10:12:31 ZimaOS audit[2398]: NETFILTER_CFG table=nat family=2 entries=22 op=xt_replace pid=2398 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2398]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=5602ec4889d0 items=0 ppid=1574 pid=2398 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100504F5354524F5554494E47002D7000746370002D73003137322E31372E302E32002D64003137322E31372E302E32002D2D64706F7274003830002D6A004D415351554552414445
Apr 24 10:12:31 ZimaOS systemd-timesyncd[660]: Network configuration changed, trying to establish connection.
Apr 24 10:12:31 ZimaOS audit[2406]: NETFILTER_CFG table=filter family=2 entries=57 op=xt_replace pid=2406 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2406]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=55b578b64b80 items=0 ppid=1574 pid=2406 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4100444F434B45520000002D6900646F636B657230002D6F00646F636B657230002D7000746370002D64003137322E31372E302E32002D2D64706F7274003830002D6A0041434
Apr 24 10:12:31 ZimaOS audit[2407]: NETFILTER_CFG table=nat family=2 entries=23 op=xt_replace pid=2407 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2407]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=5627e2d3dd00 items=0 ppid=1574 pid=2407 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100444F434B4552002D7000746370002D6400302F30002D2D64706F72740035343332002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E31382E302E323A353
Apr 24 10:12:31 ZimaOS audit[2410]: NETFILTER_CFG table=nat family=2 entries=24 op=xt_replace pid=2410 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2410]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=559dd4931f30 items=0 ppid=1574 pid=2410 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100444F434B4552002D7000746370002D6400302F30002D2D64706F7274003130343433002D6A00444E4154002D2D746F2D64657374696E6174696F6E003137322E31372E302E323A3
Apr 24 10:12:31 ZimaOS audit[2411]: NETFILTER_CFG table=nat family=2 entries=25 op=xt_replace pid=2411 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2411]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=564eb2929030 items=0 ppid=1574 pid=2411 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D74006E6174002D4100504F5354524F5554494E47002D7000746370002D73003137322E31382E302E32002D64003137322E31382E302E32002D2D64706F72740035343332002D6A004D415351554552414
Apr 24 10:12:31 ZimaOS audit[2414]: NETFILTER_CFG table=filter family=2 entries=58 op=xt_replace pid=2414 subj=unconfined comm="iptables"
Apr 24 10:12:31 ZimaOS audit[2414]: SYSCALL arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=5577a9a80d60 items=0 ppid=1574 pid=2414 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe=
Apr 24 10:12:31 ZimaOS audit: PROCTITLE proctitle=2F7573722F7362696E2F69707461626C6573002D2D77616974002D740066696C746572002D4100444F434B45520000002D690062722D663137633438363231623837002D6F0062722D663137633438363231623837002D7000746370002D64003137322E31382E302E32002
Apr 24 10:12:31 ZimaOS audit[2416]: NETFILTER_CFG table=nat family=2 entries=26
op=xt_replace pid=2416 subj=unconfined comm="iptables"
[ 10.340377] Generic FE-GE Realtek PHY r8169-0-300:00: attached PHY driver (mii_bus:phy_addr=r8169-0-300:00, irq=MAC)
[ 10.502456] r8169 0000:03:00.0 eth0: Link is Down
[ 10.570411] Loading iSCSI transport class v2.0-870.
[ 10.733380] md127: detected capacity change from 1874876416 to 0
[ 10.733384] md: md127 stopped.
[ 10.752656] md: md0 stopped.
[ 10.793989] md0: detected capacity change from 0 to 1874876416
[ 10.830483] BTRFS: device fsid 80bbaec7-58e7-368d-ae72-9c81b95516f0 devid 1 transid 714 /dev/md0 (9:0) scanned by zimaos-local-st (839)
[ 10.830754] BTRFS info (device md0): first mount of filesystem 80bbaec7-58e7-368d-ae72-9c81b95516f0
[ 10.830761] BTRFS info (device md0): using crc32c (crc32c-intel) checksum algorithm
[ 10.830765] BTRFS info (device md0): using free-space-tree
[ 11.020155] Rounding down aligned max_sectors from 4294967295 to 4294967288
[ 13.876603] loop6: detected capacity change from 0 to 416
[ 13.895870] loop7: detected capacity change from 0 to 2192
[ 13.933233] loop0: detected capacity change from 0 to 9184
[ 13.969635] loop0: detected capacity change from 0 to 416
[ 13.975048] loop1: detected capacity change from 0 to 2192
[ 13.976669] loop2: detected capacity change from 0 to 9184
[ 14.117145] r8169 0000:03:00.0 eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 14.474000] tun: Universal TUN/TAP device driver, 1.6
[ 14.835828] nvidia-nvlink: Nvlink Core is being initialized, major device number 237
[ 14.835834] NVRM: The NVIDIA GeForce G210 GPU installed in this system is
NVRM: supported through the NVIDIA 340.xx Legacy drivers. Please
NVRM: visit http://www.nvidia.com/object/unix.html for more
NVRM: information. The 580.105.08 NVIDIA driver will ignore
NVRM: this GPU. Continuing probe...
[ 14.841416] NVRM: No NVIDIA GPU found.
[ 14.842869] nvidia-nvlink: Unregistered Nvlink Core, major device number 237
[ 15.060210] NFSD: Using nfsdcld client tracking operations.
[ 15.060214] NFSD: no clients to reclaim, skipping NFSv4 grace period (net f0000000)
[ 15.813105] Initializing XFRM netlink socket
[ 15.938603] r8169 0000:03:00.0 eth0: entered promiscuous mode
[ 16.280345] docker0: port 1(vethceded42) entered blocking state
[ 16.280349] docker0: port 1(vethceded42) entered disabled state
[ 16.280355] vethceded42: entered allmulticast mode
[ 16.285206] vethceded42: entered promiscuous mode
[ 16.293919] br-f17c48621b87: port 1(veth8eae157) entered blocking state
[ 16.293923] br-f17c48621b87: port 1(veth8eae157) entered disabled state
[ 16.293931] veth8eae157: entered allmulticast mode
[ 16.294004] veth8eae157: entered promiscuous mode
[ 16.321438] docker0: port 2(vethfd8c2ed) entered blocking state
[ 16.321442] docker0: port 2(vethfd8c2ed) entered disabled state
[ 16.321451] vethfd8c2ed: entered allmulticast mode
[ 16.321904] vethfd8c2ed: entered promiscuous mode
[ 16.566865] eth0: renamed from vetha152749
[ 16.569077] docker0: port 2(vethfd8c2ed) entered blocking state
[ 16.569081] docker0: port 2(vethfd8c2ed) entered forwarding state
[ 16.586436] eth0: renamed from vethb11ebed
[ 16.586857] br-f17c48621b87: port 1(veth8eae157) entered blocking state
[ 16.586860] br-f17c48621b87: port 1(veth8eae157) entered forwarding state
[ 16.616358] eth0: renamed from vethb454bce
[ 16.618925] docker0: port 1(vethceded42) entered blocking state
[ 16.618930] docker0: port 1(vethceded42) entered forwarding state
[ 52.679017] nvidia-nvlink: Nvlink Core is being initialized, major device number 236
[ 52.679023] NVRM: The NVIDIA GeForce G210 GPU installed in this system is
NVRM: supported through the NVIDIA 340.xx Legacy drivers. Please
NVRM: visit http://www.nvidia.com/object/unix.html for more
NVRM: information. The 580.105.08 NVIDIA driver will ignore
NVRM: this GPU. Continuing probe...
[ 52.683248] NVRM: No NVIDIA GPU found.
[ 52.683943] nvidia-nvlink: Unregistered Nvlink Core, major device number 236
We followed the recommendation of some collaborators and downgraded to version 1.5.4. Now, let’s analyze it.