Boot NVME with ZimaOS crashed? Where to start and how to repair?

rmeneer · June 24, 2026, 7:34pm

I’m trying to figure out what happened here. During the day my running instance of ZimaOS crashed. Via the web interface i choose to reboot but nothing happened. I Plugged the system (Minisforum N5) to the screen and I was presented with these errors:

GRUB

Slot A (try 3)
Slot B (try 3)
Slot A (rescue Shell
Slot B (Rescue Shell)

When I try the Slot A/B optionl I get this error:

1 Filesystem checktem: /deu/nume1n1p?..•
Starting Mounting device: /deu/nume1n1p? to /mnt/overlay…mount: /nt/overlay: can’t read supert kon /dev/numeln1p?.
dmesg (1) may have more information after failed mount system call.
[FAILED] Failed to mount overlay partition
17.6027591 Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
IMMMM
17.6027991 CPU: 5 UID: 0 PID: 1 Comm: init Not tainted 6.12.25 #3
17.6028181 Hardware name: Micro Computer (HK) Tech Limited N5/F8NAA, BIOS 1.04 09/02/2025
17.6028381 Call Trace:

When I try the rescue shell I get this error:
(sorry for the many enters, its a copy/paste from an OCR-picure)

[

Starting Virtual Console Setup…

Finished File System Check on /deu/disk/by-label/casaos-boot.

OK OK OK

1 Finished File System Check on /deu/disk/by-label/casaos-overlay.

1 Finished

File System Check on /deu/disk/by-label/casaos-data.

Starting

CasaOS data resizing•••

OK OK OK

1 Finished Virtual Console Setup.

1 Finished CasaOS data resizing.

1 Finished Load Kernel Modules.

Starting Apply Kernel Variables…

OK

1 Finished Apply Kernel Variables.

Mounting

CasalS boot partition.•

Mounting

CasaDS overlay partition…

Mounting CasalS data partition..•

[FAILEDI Failed to mount CasalS ouerlay partition.

See

'systemctl status mnt-overlay.mount"

for details.

OK

1 Mounted CasaDS boot partition.

OK

i Mounted CasaDS data partition.

Mounting Log persistent data…

Starting Grow File Systen on /var/lib/casaos_data..

OK OK

1 Mounted Log persistent data.

1 Reached target Local File Systens.

Starting CasalS over lay setup..

Starting Flush Journal to Persistent Storage.

L

OK OK

Finished Grow File System on /var/lib/casaos_data.

Stopped Virtual Console Setup.

Stopping

Virtual Console Setup…

Starting

Virtual Console Setup.••

[

OK OK OK

1 Finished

CasalS overlay setup.

1

Finished

Virtual Console Setup

Finished

Flush Journal to Persistent Storage.

Starting

Create System Files and Directories…

OK

Finished

Create System Files and Directories.

Starting

Network Tine Synchronization..

OK

Started Network Tine Synchronization.

OK

1 Reached

target System Initialization.

L

OK

1 Reached target System Time Set.

L

OK

1 Started Rescue Shell.

1 Reached target Rescue Mode.

You are in rescue bode. After 1o to ng or tore “Sumalett-b” to view system logs, “systencti reboot” to reboot, or to continue bootup.

Press Enter for naintenance

Cor press Control-D to continue): -

Now my question is: is my boot NVME corrupt or is the the memory and where do start from to repair the OS so I can access my datapool.

gelbuilding · June 24, 2026, 8:15pm

The crash and lockout may not mean the whole NVMe or data pool is dead.

From the log, the failure appears to be around the writable ZimaOS overlay partition:

Failed to mount CasaOS overlay partition
can’t read superblock

That usually points to filesystem corruption on the overlay partition, or an NVMe read/write issue. A hard crash, forced power-off, unsafe shutdown, or NVMe problem can cause this.

I would not assume RAM first. RAM is possible, but the log points to storage/overlay first.

Also, because ZimaOS cannot mount the overlay partition, it cannot complete normal boot. That is why the Web UI, SSH, and normal terminal are not available. The only access at this stage is either the local rescue shell from the attached screen/keyboard, or a Linux live USB.

When you see:

Press Enter for maintenance

press Enter. If it gives you a rescue prompt, check the partitions first:

lsblk -o NAME,SIZE,FSTYPE,LABEL,UUID,MOUNTPOINTS

blkid

If possible, also check the NVMe health:

smartctl -a /dev/nvme1n1

If those commands are not available in rescue mode, boot from an Ubuntu live USB and run the checks from there.

Do not reinstall or format yet. First confirm whether casaos-data and your data pool are still visible. The boot/overlay may be damaged while the data pool is still recoverable.

piterdan · June 25, 2026, 4:46am

If you can get into the rescue shell, please run these commands and send us the output:

lsblk -o NAME,PATH,SIZE,FSTYPE,LABEL,PARTLABEL,PARTUUID,MOUNTPOINTS,MODEL,SERIAL

blkid

cat /proc/cmdline

dmesg -T | grep -Ei 'nvme|I/O error|Buffer I/O|EXT4|superblock|reset|timeout|critical|media'

smartctl -x /dev/nvme1n1

nvme smart-log /dev/nvme1n1

e2fsck -fn /dev/nvme1n1p7

Note: the device name may be different on your system. Please confirm the correct boot NVMe and the partition labeled `casaos-overlay` from `lsblk` before running any repair command.

rmeneer · June 27, 2026, 12:34pm

**here are the results of the commands. I tried to format the text from the screenshots with chatgpt.

lsblk -o NAME,SIZE,FSTYPE,LABEL,UUID,MOUNTPOINTS**

NAME SIZE FSTYPE LABEL UUID MOUNTPOINTS
sda 114.6G

sdb 10.9T linux_raid_member zimaos:fc5616382c017331 407a8dc6-4fb3-84e2-1832-b7da3249d9bd
└─md127 43.7T btrfs 407a8dc6-4fb3-84e2-1832-b7da3249d9bd

sdc 12.7T linux_raid_member zimaos:fc5616382c017331 407a8dc6-4fb3-84e2-1832-b7da3249d9bd
└─md127 43.7T btrfs 407a8dc6-4fb3-84e2-1832-b7da3249d9bd

sdd 12.7T linux_raid_member zimaos:fc5616382c017331 407a8dc6-4fb3-84e2-1832-b7da3249d9bd
└─md127 43.7T btrfs 407a8dc6-4fb3-84e2-1832-b7da3249d9bd

sde 10.9T linux_raid_member zimaos:fc5616382c017331 407a8dc6-4fb3-84e2-1832-b7da3249d9bd
└─md127 43.7T btrfs 407a8dc6-4fb3-84e2-1832-b7da3249d9bd

sdf 10.9T linux_raid_member zimaos:fc5616382c017331 407a8dc6-4fb3-84e2-1832-b7da3249d9bd
└─md127 43.7T btrfs 407a8dc6-4fb3-84e2-1832-b7da3249d9bd

nvme1n1 119.2G
├─nvme1n1p1 32M vfat casaos-boot 62EB-BDA7 /mnt/boot
├─nvme1n1p2 24M squashfs
├─nvme1n1p3 6G squashfs
├─nvme1n1p4 24M squashfs
├─nvme1n1p5 6G squashfs
├─nvme1n1p6 8M
├─nvme1n1p7 96M ext4 casaos-overlay 8ed454ec-8396-48ca-96c0-a72ca3265f52 /
└─nvme1n1p8 107.1G ext4 casaos-data 5bc1654b-e46f-43f4-9bd0-537bc50bdad3 /var/log
/var/lib/casaos_data

nvme2n1 465.8G btrfs 0b8c6c22-027d-49b3-bb8e-9af66f43f738
nvme0n1 465.8G btrfs a3676d6a-01f8-40d7-adf3-3e6bdb079acb

Device	Size	Purpose
sdb–sdf	57.1 TB raw	RAID member disks
md127	43.7 TB usable	Btrfs RAID array
nvme1n1	119.2 GB	OS/CasaOS boot drive
nvme0n1	465.8 GB	Btrfs SSD
nvme2n1	465.8 GB	Btrfs SSD

from the blkid command

/dev/sdf:
UUID=“407a8dc6-4fb3-84e2-1832-b7da3249d9bd”
UUID_SUB=“b2d666b8-4030-bd48-90bb-cb01352dc292”
LABEL=“zimaos:fc5616382c017331”
TYPE=“linux_raid_member”

/dev/nvme0n1:
UUID=“a3676d6a-01f8-40d7-adf3-3e6bdb079acb”
UUID_SUB=“e02deffd-20f8-41df-991f-774a5d155c97”
BLOCK_SIZE=“4096”
TYPE=“btrfs”

/dev/sdd:
UUID=“407a8dc6-4fb3-84e2-1832-b7da3249d9bd”
UUID_SUB=“00528f66-fdab-fbc7-9187-df436a3bb38c”
LABEL=“zimaos:fc5616382c017331”
TYPE=“linux_raid_member”

/dev/md127:
UUID=“407a8dc6-4fb3-84e2-1832-b7da3249d9bd”
UUID_SUB=“19c1c2cc-d8bc-419d-a1e5-890d14ca7bcc”
BLOCK_SIZE=“4096”
TYPE=“btrfs”

/dev/sdb:
UUID=“407a8dc6-4fb3-84e2-1832-b7da3249d9bd”
UUID_SUB=“9cd908c8-1fc7-72b6-1941-576a5475c0d3”
LABEL=“zimaos:fc5616382c017331”
TYPE=“linux_raid_member”

/dev/nvme2n1:
UUID=“0b8c6c22-027d-49b3-bb8e-9af66f43f738”
UUID_SUB=“4cfaba81-02cc-17e1-4205-9349-57c70224ba2”
BLOCK_SIZE=“4096”
TYPE=“btrfs”

/dev/sdc:
UUID=“407a8dc6-4fb3-84e2-1832-b7da3249d9bd”
UUID_SUB=“1978cf72-f1f0-50a5-6e3b-10cc8bc0a271”
LABEL=“zimaos:fc5616382c017331”
TYPE=“linux_raid_member”

/dev/nvme1n1p4:
BLOCK_SIZE=“131072”
TYPE=“squashfs”
PARTLABEL=“casaos-kernel1”

/dev/nvme1n1p2:
BLOCK_SIZE=“131072”
TYPE=“squashfs”
PARTLABEL=“casaos-kernel0”

/dev/nvme1n1p7:
LABEL=“casaos-overlay”
UUID=“8ed454ec-8396-48ca-96c0-a72ca3265f52”
TYPE=“ext4”
PARTLABEL=“casaos-overlay”

/dev/nvme1n1p5:
BLOCK_SIZE=“131072”
TYPE=“squashfs”
PARTLABEL=“casaos-system1”

/dev/nvme1n1p1:
SEC_TYPE=“msdos”
LABEL_FATBOOT=“casaos-boot”
UUID=“62EB-BDA7”
BLOCK_SIZE=“512”
TYPE=“vfat”
PARTLABEL=“casaos-boot”

/dev/nvme1n1p8:
LABEL=“casaos-data”
UUID=“5bc1654b-e46f-43f4-9bd0-537bc50bdad3”
BLOCK_SIZE=“4096”
TYPE=“ext4”
PARTLABEL=“casaos-data”

/dev/nvme1n1p6:
PARTLABEL=“casaos-bootstate”

The result from the cat command was extremely hard to capture from a 27” 4k screen
extractable part from the cat command:

BOOT_IMAGE=root=PARTUUID=
ro
result
fsrro.enabled=1
sram_dev.board=3
net.naming-scheme=v250
systemd.machine_id=
fsck.repair=yes
console=tty0
quiet
splash
loglevel=3
systemd.show_status=1
rd.udev.log_level=3
vt.handoff=1
biosdevname=0
…

from the dmesg command:

[Thu Jun 25 16:57:05 2026] Command line: BOOT_IMAGE=(hd3,gpt4)/bzImage root=PARTUUID=…
[Thu Jun 25 16:57:05 2026] Kernel command line: BOOT_IMAGE=(hd3,gpt4)/bzImage root=PARTUUID=…
[Thu Jun 25 16:57:06 2026] systemd[1]: systemd.unit=recovery.target
[Thu Jun 25 16:57:06 2026] nvme nvme0: pci function 0000:10:00.0
[Thu Jun 25 16:57:06 2026] nvme nvme1: pci function 0000:02:00.0
[Thu Jun 25 16:57:06 2026] nvme nvme2: pci function 0000:05:00.0
[Thu Jun 25 16:57:06 2026] nvme nvme1: allocated 8 MiB host memory buffer.
[Thu Jun 25 16:57:06 2026] nvme nvme1: 8/0/0 default/read/poll queues
[Thu Jun 25 16:57:06 2026] nvme nvme0: D3 entry latency set to 8 seconds
[Thu Jun 25 16:57:06 2026] nvme nvme1: Ignoring bogus Namespace Identifiers
[Thu Jun 25 16:57:06 2026] nvme1n1: p1 p2 p3 p4 p5 p6 p7
[Thu Jun 25 16:57:06 2026] nvme nvme2: 15/0/0 default/read/poll queues
[Thu Jun 25 16:57:06 2026] nvme nvme0: allocated 64 MiB host memory buffer.
[Thu Jun 25 16:57:06 2026] nvme nvme0: 12/0/0 default/read/poll queues
[Thu Jun 25 16:57:06 2026] ata1: SATA link down (SStatus 0 SControl 300)

[Thu Jun 25 16:57:24 2026] nvme1n1: I/O Cmd(0x2) @ LBA 25348616, 256 blocks, I/O Error (sct 0x2 / sc 0x81)
[Thu Jun 25 16:57:24 2026] critical medium error, dev nvme1n1, sector 25348616 op 0x0:(READ) flags 0x80700 phys_seg 18 prio class 0
[Thu Jun 25 16:57:24 2026] Buffer I/O error on dev nvme1n1, logical block 67, async page read

[Thu Jun 25 16:57:24 2026] critical medium error, dev nvme1n1, sector 25348632 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Thu Jun 25 16:57:24 2026] Buffer I/O error on dev nvme1n1, logical block 67, async page read

[Thu Jun 25 16:57:24 2026] nvme1n1: I/O Cmd(0x2) @ LBA 25348616, 256 blocks, I/O Error (sct 0x2 / sc 0x81)
[Thu Jun 25 16:57:24 2026] critical medium error, dev nvme1n1, sector 25348616 op 0x0:(READ) flags 0x80700 phys_seg 3 prio class 0
[Thu Jun 25 16:57:24 2026] Buffer I/O error on dev nvme1n1, logical block 67, async page read

[Thu Jun 25 16:57:24 2026] nvme1n1: I/O Cmd(0x2) @ LBA 25348632, 8 blocks, I/O Error (sct 0x2 / sc 0x81)
[Thu Jun 25 16:57:24 2026] critical medium error, dev nvme1n1, sector 25348632 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
[Thu Jun 25 16:57:24 2026] Buffer I/O error on dev nvme1n1p7, logical block 67, async page read

[Thu Jun 25 16:57:24 2026] nvme1n1: I/O Cmd(0x2) @ LBA 25360904, 64 blocks, I/O Error (sct 0x2 / sc 0x81)
[Thu Jun 25 16:57:24 2026] critical medium error, dev nvme1n1, sector 25360904 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0

[Thu Jun 25 16:57:25 2026] EXT4-fs (nvme1n1p7): warning: mounting fs with errors, running e2fsck is recommended
[Thu Jun 25 16:57:25 2026] EXT4-fs (nvme1n1p7): mount failed
[Thu Jun 25 16:57:25 2026] EXT4-fs (nvme1n1p8): couldn’t mount because of unsupported optional features …

[Thu Jun 25 16:57:25 2026] amd_pmc … SMU timeout …

[Thu Jun 25 17:00:22 2026] nvme1n1: I/O Cmd(0x2) …
[Thu Jun 25 17:00:22 2026] critical medium error, dev nvme1n1 …
[Thu Jun 25 17:00:22 2026] Buffer I/O error on dev nvme1n1p7, logical block 67, async page read

[Thu Jun 25 17:09:12 2026] nvme1n1: I/O Cmd(0x2) …
[Thu Jun 25 17:09:12 2026] critical medium error, dev nvme1n1 …
[Thu Jun 25 17:09:12 2026] Buffer I/O error on dev nvme1n1p7, logical block 67, async page read

[Thu Jun 25 17:14:53 2026] nvme1n1: I/O Cmd(0x2) …
[Thu Jun 25 17:14:53 2026] critical medium error, dev nvme1n1 …
[Thu Jun 25 17:14:53 2026] Buffer I/O error on dev nvme1n1p7, logical block 67, async page read

[Thu Jun 25 17:46:30 2026] nvme1n1: I/O Cmd(0x2) …
[Thu Jun 25 17:46:30 2026] critical medium error, dev nvme1n1 …
[Thu Jun 25 17:46:30 2026] Buffer I/O error on dev nvme1n1p7, logical block 67, async page read

From the sortctl command:

bash-5.2# smartctl -x /dev/nvme1n1

smartctl 7.4 2023-08-01 r5530 [x86_64-linux-6.12.25] (local build)
Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number: AirDisk 128GB SSD
Serial Number: QGB656B007477P110N
Firmware Version: SN1509B
PCI Vendor/Subsystem ID: 0x1e4b
IEEE OUI Identifier: 0x3a5a27
Total NVM Capacity: 128,035,676,160 [128 GB]
Unallocated NVM Capacity: 0
Controller ID: 0
NVMe Version: 1.4
Number of Namespaces: 1
Namespace 1 Size/Capacity: 128,035,676,160 [128 GB]
Namespace 1 Formatted LBA Size: 512
Namespace 1 IEEE EUI-64: 3a5a27 03500004d5
Local Time is: Thu Jun 25 17:22:12 2026 UTC
Firmware Updates (0x1a): 5 Slots, no Reset required
Optional Admin Commands (0x0017): Security Format Frmw_DL Self_Test
Optional NVM Commands (0x001f): Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Log Page Attributes (0x06): Cmd_Eff_Lg Ext_Get_Lg
Maximum Data Transfer Size: 128 Pages
Warning Comp. Temp. Threshold: 90 Celsius
Critical Comp. Temp. Threshold: 95 Celsius

Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 6.50W - - 0 0 0 0 0 0
1 + 5.00W - - 1 1 1 1 0 0
2 + 3.60W - - 2 2 2 2 0 0
3 - 0.0500W - - 3 3 3 3 5000 10000
4 - 0.0025W - - 4 4 4 4 8000 45000

Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 0 512 0 0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x00
Temperature: 33 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 3%
Data Units Read: 1,380,022 [706 GB]
Data Units Written: 3,013,555 [1.54 TB]
Host Read Commands: 133,991,379
Host Write Commands: 120,782,615
Controller Busy Time: 276
Power Cycles: 66
Power On Hours: 238
Unsafe Shutdowns: 16
Media and Data Integrity Errors: 76
Error Information Log Entries: 0
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 33 Celsius
Temperature Sensor 2: 35 Celsius

Error Information (NVMe Log 0x01, 16 of 64 entries)
No Errors Logged

Read Self-test Log failed: Invalid Field in Command (0x002)

bash-5.2#

from the nvme smart-log /dev/nvme1n1 command

bash-5.2# nvme smart-log /dev/nvme1n1

Smart Log for NVME device:nvme1n1 namespace-id:ffffffff
critical_warning : 0
temperature : 32 C
available_spare : 100%
available_spare_threshold : 10%
percentage_used : 3%
endurance_group_critical_warning_summary : 0
data_units_read : 1380022
data_units_written : 3013556
host_read_commands : 133991379
host_write_commands : 128782771
controller_busy_time : 276
power_cycles : 66
power_on_hours : 238
unsafe_shutdowns : 16
media_errors : 76
num_err_log_entries : 0
Warning Temperature Time : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1 : 32 C
Temperature Sensor 2 : 35 C
Thermal Management T1 Trans Count : 0
Thermal Management T2 Trans Count : 0
Thermal Management T1 Total Time : 0
Thermal Management T2 Total Time : 0

and the final e2fsck command:

bash-5.2# e2fsck -fn /dev/nvme1n1p7

e2fsck 1.47.2 (1-Jan-2025)
Warning: skipping journal recovery because doing a read-only filesystem check.

Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information

Error reading block 271 (Input/output error) while reading inode and block bitmaps. Ignore error? no

e2fsck: Can’t read an inode bitmap while retrying to read bitmaps for casaos-overlay
e2fsck: aborted

bash-5.2#

BentFender · June 27, 2026, 6:39pm

I had a SIMILAR but different problem with new install today. I was not able to install ZimaOS on my newly inserted NVMe drive. It seems banal but I have never checked the CMOS battery on my motherboard and it kept no time - just reset to the production date every boot. As soon as I changed the CR 2032 battery on the motherboard it went smoothly and now my server is up and running again. I hope this helps someone, cheers guys !

rmeneer · June 27, 2026, 9:39pm

I will check it, just to be on the safe side. But my crash happened when all was up and running. Thanks!

rmeneer · June 28, 2026, 11:16am

I checked the time/ date, adjusted the time into summer time (would expect that the os would automatically adjust using ntp). This small adjustment didn’t do anything and resulted in the same error…. Thanks for helping

piterdan · June 29, 2026, 7:26am

Thanks for sending all of that over. This is really helpful.

Looking at the logs, I don’t think this is just ZimaOS choosing the wrong Slot A/B, and it doesn’t look like a normal memory issue either.

The important part is this:

critical medium error, dev nvme1n1
Buffer I/O error on dev nvme1n1p7
EXT4-fs (nvme1n1p7): mount failed
e2fsck: Error reading block 271 (Input/output error)
media_errors: 76

nvme1n1 is your 128GB boot SSD, and nvme1n1p7 is the small ZimaOS overlay partition. That partition is where the system keeps some persistent system configuration. Both Slot A and Slot B need this same partition, so that explains why both boot options fail.

Your large data pool looks separate from this:

sdb-sdf → md127 → btrfs 43.7T

So the good news is: this does not look like your main datapool is gone. The problem appears to be on the small boot NVMe.

The SMART result saying “PASSED” is a bit misleading here. The drive has still logged Media and Data Integrity Errors: 76, and the kernel is reporting real read errors. So I would not trust this boot SSD anymore.

What I would do next:

Power the machine off completely.

Reseat the 128GB boot NVMe if you are comfortable doing that.

Try booting once more.

If the same error comes back, replace that 128GB boot SSD.

Reinstall ZimaOS onto the new boot SSD.

Be very careful during reinstall not to wipe or initialize the existing data disks/pool.

There is a possible temporary workaround where the small overlay partition is recreated, but because the drive is already returning medium errors, I would treat that only as a last-resort boot attempt, not a real fix.

I wouldn’t spend too much time trying to repair this boot SSD. Once a drive starts returning medium errors, it’s usually better to replace it and avoid risking more weird failures later.

rmeneer · June 30, 2026, 11:29am

I replaced the drive, reinstalled and the datapool is still there. The only caveat is the activation code for, which now ‘is in use by another device’? How can I restore my licence?