Thanks for continuing with this. I believe we have now taken the normal logging investigation as far as the current ZimaOS kernel allows.
The latest crash followed exactly the same pattern:
Complete host lockup, not a Docker or application failure
No panic, OOM, i915, SATA/NVMe, or watchdog error
Nothing captured by the persistent journal, pstore, or remote logger
Removing the IOMMU and VFIO parameters made no difference
Crash-dump support exists in the kernel, but no crash kernel is currently reserved or active
Rather than continuing to request more commands that produce the same result, I suggest one controlled isolation test.
If RataNAS uses SATA storage, test only the libata.force=nolpm parameter suggested by piterdan. Do not change anything else, and monitor it beyond the usual crash window. If the system does not use SATA storage, that parameter will not be relevant.
If it crashes again, I do not believe collecting the same logs will help. The next meaningful test would be running a standard Debian installation from a spare drive under a comparable workload.
If Debian also crashes, hardware or firmware becomes the leading suspect. If Debian remains stable beyond the established crash window, the ZimaOS kernel or driver stack becomes the leading suspect, and IceWhale will need to investigate or provide a suitable debugging kernel.
Just had another crash. Added the sata parameter. I am using sata, although through a m2 pcie to sata adapter as my motherboard has only 1 sata port. On that ‘real’ sata port the OS disk is located, on the m2 to 6x sata both 4tb data disks are located. This ran fine for months on OMV. If I’m not mistaken, that’s based of debian.
libata.force=nolpm doesn’t help either, another crash this morning. If there’s no other options I’m afraid I’ll have to go back to OMV again. This is unfortunately unworkable😟
Thanks you so much @gelbuilding for all the help.and insights so far! Really appreciated.
@Zima-Jerry do you have other options at this point?
I completely understand your decision. You have been extremely patient and have tested every reasonable suggestion we made.
The same hardware running reliably for months under OMV is important comparative evidence. It does not completely rule out hardware, but after removing the IOMMU/VFIO parameters, testing the SATA LPM workaround, checking persistent logs, pstore and remote logging, I believe a ZimaOS kernel or driver compatibility issue is now the leading suspect. The PCIe-to-SATA controller path may be involved, but the current evidence cannot confirm that.
I don’t want to send you through more commands that produce the same result. At this stage, IceWhale needs to review the evidence in this thread and either provide a targeted test, a debugging kernel, or another supported way to capture this silent lockup.
Returning to OMV is completely reasonable because the system needs to be reliable. If @Zima-Jerry has another controlled test or debug build, I believe that is the only worthwhile next step.
Thank you as well for your patience, detailed testing and clear feedback. It has been genuinely appreciated.
Same problem—I have a Blackview MP80 mini PC. Every 2–3 days, it becomes unresponsive; I can’t see anything on the screen—it’s just black. By all accounts, it’s a freeze. I’ve checked to see if it was the apps, but I don’t think that has anything to do with it—I only have two installed, and they’re from the app store. This didn’t happen to me before with Fygo.
All the points I currently suspect have been ruled out. If possible, would this situation still occur when using a completely new system, without installing any applications at all? Due to the many uncertain factors present, I can only start by installing the system from a clean state, to determine whether the problem lies with the system itself, or with certain applications when combined with it.
Well, back when I started with Zima I had a lockup when configuring samba, without anything else installed. But I don’t remember what version that was, 1.5.something I suppose. However, as stated, with the very same 3 containers running, OMV runs perfectly fine. Is Casa based on Debian?
Maybe I can source another one of those mini pc’s, I am sure I had two of those. As I can’t run too long without either backup (duplicati) and the security cams (frigate) I can’t use my main system for that. But in addition, as per this thread we have also tested with unifi and frigate completely disabled (not uninstalled) to no avail. Duplicati kept running if I remember correctly to keep my files backed up.
There’s multiple people reporting these kinds of lockups though, and all of them as far as I’ve seen use intel iGPU. While lots of things have been tested, I feel I can’t rule that out yet. When I’m back I’ll try to find if OMV uses specific kernel parameters maybe. Who knows.
Hi, I just got my Zimaboard2 2 weeks ago. The first week was running ok and then… I’m experiencing the same issue reported here, suddenly just disconnects and disappears from the network. It’s still powered on but hidden from the network and inaccessible by any means. — complete host lockup, board stays powered (all lights on) but disappears entirely from the network: not pingable, not visible in router’s connected-devices list, not SSH-able, requires a full power cycle to recover.
Device details:
- Hardware: ZimaBoard 2
- ZimaOS version: v 1.7.0
- Symptom: intermittent full lockup, roughly every 1-2 days, no pattern tied to specific app usage
`coredumpctl list` shows 8 separate segfault/coredump events across 3 different days (Aug 5-7), hitting different binaries: /usr/local/bin/python3.11, /usr/bin/python3.14, /usr/sbin/xtables-legacy-multi, and /bin/dbus-broker. This isn’t one app crashing — it’s a recurring pattern across the system.
The two closest to my most recent freeze (04:09:13 and 04:12:22 CST, ~6 minutes before the journal stopped logging entirely) were both Python 3.14 segfaults. I pulled the full coredump info on the second one:
- Run as root under systemd unit `zimaos-welcome@tty1.service` (the local console welcome screen)
- Signal 11 (SIGSEGV), crashing deep inside libpython3.14.so.1.0 itself (_PyEval_EvalFrameDefault / import machinery), not in application code
This points at a possible Python 3.14 interpreter-level bug or instability (3.14 is a very recent release), triggered repeatedly by ZimaOS’s own crudini/welcome-service Python wrapper, rather than anything in my configuration, network, or storage. I separately ruled out my NVMe (Kingston KC3000) and RAM as direct causes — SMART data shows no critical warnings or media errors (one minor thermal throttle event, likely unrelated).
Also worth noting: I had ZimaNet (built-in remote access) enabled at the time of this crash. Disabling it via the UI toggle did NOT actually stop the systemd service — it kept running and cycling failed peer connections in the background. I had to stop it manually with `systemctl stop znet`. Currently running with ZimaNet and Tailscale both fully disabled to see if lockups continue.
Thanks for that input! Really interesting to see it crashing on actual Zimaboard hardware as well. Might be completely different things ofcourse but it might give some insights.
Mine hadn’t crashed for about a week. It turned out Duplicati (my backup tool in a docker container) ran into an issue and was generating lots of cpu load. As soon as I fixed that and cpu load was normal again, within 3 hours or so it crashed again. So I thought maybe something with the cpu governor. I set that to performance just to test that out, but no cigar either, another crash within hours.
What interests me though is that zimaboard 2 is N150 based, so that’s also using the i915 driver. I just came back from vacation. Next week I’ll have some time to compare to omv. Is CasaOS debian based?
btw @gelbuilding, just for reference, I had the kernel-logger running all the time, and it didn’t log anything interesting besides what we already saw. So I’m shutting that down for now.
[edit]
just to rule things out once more, I have now disabled the i915 driver once again. This renders quite some things unusable as they don’t have a rendering device. So I also had to disable Frigate for that. Frigate itself turned out not to be the issue; with Frigate disabled (and i915 enabled) it still crashed. But I’ll let it run like this for a while just to rule things out.
Unfortunately, also with i915 disabled, it just crashed again. Given the number of other users reporting issues lately, even with official Zima hardware, and the lack of solutions from Icewhale, I can’t deny I’m losing confidence by now. Is anyone actually working on these issues from the Icewhale side? For me at least stability has priority over new features.
[edit]
after yet another crash, just hours after rebooting, I switched back to OMV again. I have seperate OS SSD’s for both Zima and OMV, so I can switch quickly without too much hassle. I will report back in a few weeks if it’s still up and running, just to confirm hardware is not the issue here.
We are currently working on addressing the issue where the zimaos-welcome function calls crudini, resulting in a crash in Python 3.14. We are already in the process of resolving this problem.
We will remove the dependencies on crudini, and also reduce the frequency at which zimaos-welcome requests various types of information. The changes have been made and will be released in the next test version.
Well I guess that’s good news. Being a professional MS engineer and administrator for 25+ years now, in the Linux realm I’m a mere user, maybe just a bit more. For my understanding, how could a userspace package like python halt a whole system? Even if python crashes, shouldn’t at least kernel-stuff like networking still be running?
You’re right. The crudini issue was just a trigger; the actual cause of the system crash is currently under investigation. However, the zimaos-welcome service will no longer cause system resource waste or system crashes.
Sidenote is that according to my Shelly smartplug this device is connected to (because I needed to be able to restart the machine remotely when Zima crashed) reports it has slightly lower power consumption with OMV compared to ZimaOS. I still feel it has to do with how it handles C-states or something.
1. We removed the dependency on Crudini, which had a chance of causing Python to crash in Python 3.14, and optimized the frequency of resource requests by the zimaos-welcome service to reduce CPU usage.
2. We rolled back the Docker Engine version, which improved the startup speed of Docker containers and reduced the likelihood of DBus broker message blocking.
Not yet but I will this weekend and let ya know if it crashes again. So far it’s been about every 10-15 days on 1.6.1
Btw it looks like you have a similar smart plug if not the same one as I did and that was causing the crashes way more often for me but didn’t prevent them entirely
How could a smartplug cause a system to crash? For me I started using it because of the crashes, and it didn’t make it better or worse. As they are just microcontroller operated relays, they shouldn’t bring a lot of interference. I use Shelly Plug S Gen3, and since moving back to OMV I’ve not had a single crash anymore.
I still wonder how something like Crudini (whatever that is or does) or Python or even docker can halt the whole kernel. Sure processes could crash because of bugs, but when they can halt or crash the kernel, there’s a fundamental issue or bug. Preventing the trigger from happening is good, but solving the kernel won’t crash is better.