Persistent Rescue/Emergency Mode Boot Loop on Specific Node

Operating System ZimaOS
1

Disclaimer: I’ve been using Gemini to help me troubleshoot and pull these diagnostics. Some of the technical assumptions about the root cause might be slightly off, but the bottom line is that this specific server is refusing to boot after a clean shutdown and I’m trying to avoid another full reinstall.

I run three ZimaOS nodes (v1.7+), and one specific machine continuously falls into a Rescue/Emergency mode boot loop after standard system operations.

Background: A while back, during an update to 1.7+, this specific server dropped into rescue mode. No mount fixes or command-line troubleshooting worked, so I eventually gave up, wiped the drive, performed a clean reinstall, and restored my containers from backup. Everything appeared fine for a while.

Current Issue: Last night, due to a severe storm, I executed a clean shutdown via the UI, powered off all containers gracefully, and unplugged the machine. Upon powering back up this morning, the same node immediately dropped back into Rescue Mode / Emergency Mode. (My other two nodes booted back up perfectly without issue).

Here is what I have observed and tested so far on the failing node:

  • System Logs: journalctl -xb shows casaos-overlay[652]: ldconfig: Can't create temporary cache file /etc/ld.so.cache~: Read-only file system, indicating the overlay/systemd target is failing because the environment is mounted read-only.

  • Storage Layout: Booting off NVMe (/dev/nvme0n1). Partitions p3 (SquashFS root), p7 (/mnt/overlay), and p8 (/var/lib/casaos_data).

  • Filesystem Status: Running e2fsck -fy on the writable ext4 partitions (p7 and p8) completes without critical errors, but the node still panics upon boot.

  • Boot Parameters: Editing GRUB to force filesystem checks (fsck.mode=force fsck.repair=yes) runs cleanly, but the system continuously drops back into the emergency target right after Reached target ZFS startup target / Finished CasaOS overlay setup.

It appears RAUC slot verification or the SquashFS overlay binding is getting permanently stuck in a failed boot state on this specific hardware configuration after a reboot/shutdown sequence.

Has anyone encountered this specific overlay/RAUC lockup, or is there a way to reset the boot slot flag from the emergency shell without having to re-image the system every time it reboots? Any insight from the dev team would be appreciated.