eth0 UP 192.168.6.251/24
192.168.6.252 dev eth0 lladdr 00:15:5d:06:02:03 STALE
From ZimaOS:
echo “Hello world!”|nc -u 192.168.6.252 6666
message received by ZimaLoggers socat and logged to the logfile.
Hi, could you try adding the following parameter to cmdline.txt and monitor the system?
libata.force=nolpm
I suspect SATA LPM may be causing the drive or SATA link to become unresponsive.
@gelbuilding would that interfere with your ongoing investigation or should I wait for your instructions?
Thanks, Piterdan. I suggest we finish the netconsole setup first while keeping RataNAS unchanged. libata.force=nolpm may be worth testing later, but the current logs contain no verified SATA link resets or ATA timeout errors. Adding it now would make it harder to know which change affected the result.
Rataplan626, provided ZimaLogger is running on a different physical machine, keep its socat receiver open and run this on RataNAS:
modprobe netconsole netconsole=6665@192.168.6.251/eth0,6666@192.168.6.252/00:15:5d:06:02:03
If that returns without an error, run:
dmesg -n 8 && echo '<6>RataNAS netconsole test' > /dev/kmsg
ZimaLogger should immediately display RataNAS netconsole test and save it in /var/log/ratanas-netconsole.log.
This configuration is temporary and will disappear after a reboot. Please confirm whether the test message arrives. Once verified, we can make netconsole persistent. After that, libata.force=nolpm can be tested separately without losing the crash capture.
root@RataNAS:/root ➜ # modprobe netconsole netconsole=6665@192.168.6.251/eth0,6666@192.168.6.252/00:15:5d:06:02:03
modprobe: FATAL: Module netconsole not found in directory /lib/modules/6.18.9
root@RataNAS:/root ➜ #
I tested the same ZimaOS kernel and confirmed that netconsole is built directly into the kernel with CONFIG_NETCONSOLE=y. Therefore, the modprobe error is expected and does not mean netconsole is unavailable.
Please keep ZimaLogger’s socat receiver running before rebooting RataNAS.
In RataNAS cmdline.txt, remove:
quiet splash loglevel=3
Replace it with:
loglevel=8 netconsole=6665@192.168.6.251/eth0,6666@192.168.6.252/00:15:5d:06:02:03
Keep everything on the same single line. Do not add libata.force=nolpm yet because we want the next test to contain only the logging change.
After rebooting, run these two commands on RataNAS:
cat /proc/cmdline
echo '<6>RataNAS persistent netconsole test' > /dev/kmsg
Please confirm that the netconsole parameter appears in /proc/cmdline and that ZimaLogger receives the test message. If both succeed, leave ZimaLogger running for the next lockup. Netconsole is specifically designed for capturing kernel messages when local disk logging fails.
admin@RataNAS:~ ➜ $ cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt2)/bzImage root=PARTUUID=8d3d53e3-6d49-4c38-8349-aff6859e82fd rootwait net.naming-scheme=v250 systemd.machine_id=4073bb4f60c14bbeb7016546f3b3098c fsck.repair=yes console=tty1 loglevel=8 netconsole=6665@192.168.6.251/eth0,6666@192.168.6.252/00:15:5d:06:02:03 systemd.show_status=1 rd.udev.log_level=3 net.ifnames=0 biosdevname=0 thunderbolt.host_reset=false rauc.slot=A
the test message is not received by the logger. When I use
echo "Hello world!"|nc -u 192.168.6.252 6666
from RataNAS to ZimaLogger it is received, so the listener is working fine. The messages do end up in /dev/kmsg on Ratanas itself, but they aren’t sent out to the logger system.
root@RataNAS:/mnt/boot ➜ # dmesg|grep netconsole
[ 0.000000] Command line: BOOT_IMAGE=(hd1,gpt2)/bzImage root=PARTUUID=8d3d53e3-6d49-4c38-8349-aff6859e82fd rootwait net.naming-scheme=v250 systemd.machine_id=4073bb4f60c14bbeb7016546f3b3098c fsck.repair=yes console=tty1 loglevel=8 netconsole=6665@192.168.6.251/eth0,6666@192.168.6.252/00:15:5d:06:02:03 systemd.show_status=1 rd.udev.log_level=3 net.ifnames=0 biosdevname=0 thunderbolt.host_reset=false rauc.slot=A
[ 0.048952] Kernel command line: BOOT_IMAGE=(hd1,gpt2)/bzImage root=PARTUUID=8d3d53e3-6d49-4c38-8349-aff6859e82fd rootwait net.naming-scheme=v250 systemd.machine_id=4073bb4f60c14bbeb7016546f3b3098c fsck.repair=yes console=tty1 loglevel=8 netconsole=6665@192.168.6.251/eth0,6666@192.168.6.252/00:15:5d:06:02:03 systemd.show_status=1 rd.udev.log_level=3 net.ifnames=0 biosdevname=0 thunderbolt.host_reset=false rauc.slot=A
[ 0.973964] netconsole: netconsole: local port 6665
[ 0.973968] netconsole: netconsole: local IPv4 address 192.168.6.251
[ 0.973971] netconsole: netconsole: interface name ‘eth0’
[ 0.973974] netconsole: netconsole: local ethernet address ‘ff:ff:ff:ff:ff:ff’
[ 0.973977] netconsole: netconsole: remote port 6666
[ 0.973979] netconsole: netconsole: remote IPv4 address 192.168.6.252
[ 0.973982] netconsole: netconsole: remote ethernet address 00:15:5d:06:02:03
[ 0.973985] netpoll: netconsole: eth0 doesn’t exist, aborting
[ 0.973988] netconsole: Not enabling netconsole for cmdline0. Netpoll setup failed
[ 0.973991] netconsole: cleaning up
[ 166.797774] RataNAS persistent netconsole test
[ 405.804601] RataNAS persistent netconsole test
[ 416.500351] RataNAS persistent netconsole test
however, the nic is certainly eth0:
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.6.251 netmask 255.255.255.0 broadcast 192.168.6.255
ether d8:bb:c1:92:fe:31 txqueuelen 1000 (Ethernet)
RX packets 1574604 bytes 2360600022 (2.1 GiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 786542 bytes 55542956 (52.9 MiB)
TX errors 0 dropped 2 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xb1100000-b1120000
Thank you. This now clearly identifies why netconsole did not work.
The netconsole settings were parsed correctly, including the IP addresses, ports and ZimaLogger MAC address. However, netconsole started at 0.973985 seconds, before eth0 existed:
netpoll: netconsole: eth0 doesn’t exist, aborting
netconsole: Not enabling netconsole for cmdline0
Your latest output confirms that eth0 becomes available and operates normally later in the boot. It currently shows no RX errors, overruns or carrier failures. The two dropped TX packets are insignificant and do not explain the complete system lockups.
This is why the ordinary nc test reaches ZimaLogger while kernel netconsole does not. Netconsole disabled itself before the Ethernet interface became available.
I now want to check whether the ZimaOS kernel supports configuring netconsole dynamically after eth0 has started. Please run:
grep -E ' /sys/kernel/config |configfs' /proc/mounts || echo 'configfs is not mounted'
zcat /proc/config.gz | grep -E 'CONFIG_CONFIGFS_FS|CONFIG_NETCONSOLE'
Please paste both results. Do not add libata.force=nolpm yet. If dynamic configuration is available, we can enable netconsole after eth0 exists while keeping the rest of the system unchanged.
root@RataNAS:/mnt/boot ➜ # grep -E ' /sys/kernel/config |configfs' /proc/mounts || echo 'configfs is not mounted'
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
root@RataNAS:/mnt/boot ➜ # zcat /proc/config.gz | grep -E 'CONFIG_CONFIGFS_FS|CONFIG_NETCONSOLE'
CONFIG_NETCONSOLE=y
# CONFIG_NETCONSOLE_EXTENDED_LOG is not set
CONFIG_CONFIGFS_FS=m
This confirms the limitation in the current ZimaOS kernel.
Netconsole is built into the kernel, while configfs is provided as a separate module. Netconsole starts before eth0 exists, disables itself, and this kernel does not provide the dynamic netconsole interface needed to attach it later.
We will use live userspace kernel-log forwarding as the fallback. This may capture messages immediately before the system locks, although it cannot transmit after the kernel has completely stopped scheduling processes.
Keep the ZimaLogger socat receiver running. On RataNAS, start:
journalctl -kf -n 0 -o short-precise | nc -u 192.168.6.252 6666
Leave that terminal open. From a second RataNAS SSH session, run:
echo '<4>RataNAS live kernel forwarding test' > /dev/kmsg
Please confirm whether ZimaLogger receives RataNAS live kernel forwarding test. If successful, I will provide a small systemd service so forwarding starts automatically and survives SSH disconnection and reboot.
Keep libata.force=nolpm out for this test. Once remote logging is persistent, we can test that parameter separately.
ah, that makes sense. And hence the given commands work and log to socat. Testmessages arrived at socat when sending to /dev/kmsg.
Excellent. This confirms that live kernel messages from /dev/kmsg are reaching ZimaLogger and being written by socat.
We will now make the ZimaLogger receiver start automatically. First, stop the existing foreground socat command with Ctrl+C, then run these two commands on ZimaLogger.
Create the receiver service:
sudo tee /etc/systemd/system/ratanas-kernel-receiver.service >/dev/null <<'EOF'
[Unit]
Description=Receive RataNAS live kernel messages
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
ExecStart=/usr/bin/socat -u UDP-RECV:6666,reuseaddr -
StandardOutput=append:/var/log/ratanas-kernel.log
StandardError=journal
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target
EOF
Enable it and verify its status:
sudo systemctl daemon-reload && sudo systemctl enable --now ratanas-kernel-receiver.service && sudo systemctl --no-pager --full status ratanas-kernel-receiver.service
Please paste the service status. Once it is active, we will create the matching automatic sender service on RataNAS and send another test message.
● ratanas-kernel-receiver.service - Receive RataNAS live kernel messages
Loaded: loaded (/etc/systemd/system/ratanas-kernel-receiver.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-07-22 07:17:19 CDT; 1min 10s ago
Invocation: caf05539195843ec97dab0a5a75ef45e
Main PID: 702 (socat)
Tasks: 1 (limit: 2224)
Memory: 1.8M (peak: 2.1M)
CPU: 4ms
CGroup: /system.slice/ratanas-kernel-receiver.service
└─702 /usr/bin/socat -u UDP-RECV:6666,reuseaddr -
It’s running, but it’s not receiving messages when I send a testmessage to /dev/kmsg on RataNAS/ZimaOS, but it does log when I simply nc to it. So the socat-service runs and works.
on Ratanas I created a systemd service with
ExecStart=/bin/sh -c ‘/usr/bin/journalctl -kf -n 0 -o short-precise | /usr/bin/nc -u 192.168.6.252 6666’
I’ve rebooted RataNAS and indeed a lot of messages are received by the logger. If I did wrong and you need some other command to run on RataNAS, please inform. If anything happens in between I will report the log.
Perfect. What you created on RataNAS is correct. The fact that messages arrived after reboot confirms that:
- The ZimaLogger receiver service starts automatically.
- The RataNAS sender service starts automatically.
- Live kernel journal messages are reaching
/var/log/ratanas-kernel.log.
We do not need another command now. Please leave both services running and keep libata.force=nolpm out for the moment so the system remains unchanged.
If RataNAS locks again, note the exact time. After rebooting RataNAS, do not restart or alter ZimaLogger. Let me know the crash time and we will extract only the relevant section from /var/log/ratanas-kernel.log.
The failed boot-time netconsole= parameter is no longer required because the userspace forwarder has replaced it. You can remove that parameter and restore quiet splash loglevel=3 at the next convenient reboot. There is no need to reboot solely for this cleanup.
This forwarding method may capture the activity leading into the lockup. If the kernel stops completely without producing another message, the remote log will stop abruptly at the same moment, which is also useful evidence.
Exciting times! Another crash! However… unfortunately no logs. Crash occured at 18:42 CET, and I pulled the plug at 18:55. Zimalogger logs attached. There has been almost no logging at all at socat by the way, but it was still running as after the reboot it logged some things again.
ratanas-kernel.log (5.9 KB)
I believe the reason is that ZimaLogger was working, but the sender uses journalctl and nc, which are userspace processes. When RataNAS completely freezes, those processes and the network stack stop running. If the kernel produces no warning before the lockup, nothing can be transmitted.
The entries at 18:55 are from the new boot. The last pre-crash entry was the previous evening, and the perf sampling messages do not identify the crash cause.
The current ZimaOS kernel also cannot use true netconsole correctly because netconsole starts before eth0 exists, then disables itself. We now need to check whether persistent kernel crash storage or lockup detection is available.
Please run these two commands on RataNAS:
zcat /proc/config.gz | grep -E 'CONFIG_(PSTORE|PSTORE_RAM|PSTORE_CONSOLE|PSTORE_PMSG|KEXEC|CRASH_DUMP|HARDLOCKUP_DETECTOR|SOFTLOCKUP_DETECTOR|WQ_WATCHDOG)='
dmesg | grep -Ei 'pstore|ramoops|efi.*pstore|watchdog|hard lockup|soft lockup'
Please post both outputs. That will tell us whether we can capture the next lockup through pstore, kdump, or the kernel watchdog. Changing socat will not solve this limitation.
root@RataNAS:/root ➜ # zcat /proc/config.gz | grep -E 'CONFIG_(PSTORE|PSTORE_RAM|PSTORE_CONSOLE|PSTORE_PMSG|KEXEC|CRASH_DUMP|HARDLOCKUP_DETECTOR|SOFTLOCKUP_DETECTOR|WQ_WATCHDOG)='
CONFIG_KEXEC=y
CONFIG_CRASH_DUMP=y
CONFIG_PSTORE=y
CONFIG_PSTORE_RAM=y
CONFIG_SOFTLOCKUP_DETECTOR=y
CONFIG_HARDLOCKUP_DETECTOR=y
root@RataNAS:/root ➜ # dmesg | grep -Ei 'pstore|ramoops|efi.*pstore|watchdog|hard lockup|soft lockup'
[ 0.136099] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
[ 0.976059] pstore: Using crash dump compression: deflate
[ 0.976063] pstore: Registered efi_pstore as persistent store backend
[ 14.696047] systemd[1]: Load Kernel Module efi_pstore skipped, unmet condition check ConditionKernelModuleLoaded=!efi_pstore
[ 14.772523] systemd[1]: Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore
root@RataNAS:/root ➜ #
I’m learning quite a few new things in Linux with all this 
I’m not new to it, and can usualy find my way, but while in the Microsoft realm I’m a pretty darn good troubleshooter (I earn my living with that), I like learning some of it in Linux as well For my understanding, are you actually a ZimaOS / Casa dev? Or ‘just’ a very knowledgable user?
Good output. It confirms that the kernel includes pstore, crash-dump support, and hard/soft lockup detection. The NMI watchdog is already enabled, and EFI pstore is registered.
However, pstore remained empty after the crash, so the freeze did not produce a panic or crash record that EFI could save. CONFIG_KEXEC and CONFIG_CRASH_DUMP only confirm kernel support. They do not confirm that a crash kernel is reserved and loaded.
Could you please run these two checks:
grep -o 'crashkernel=[^ ]*' /proc/cmdline || echo "No crashkernel parameter configured"
cat /sys/kernel/kexec_crash_size 2>&1
Regarding your question, I don’t work for IceWhale and I’m not a ZimaOS or CasaOS developer. I’m simply a user who enjoys investigating problems and helping where I can.
I started using computers back in the 1980s, when DOS was all we had. What a pain that was! Most of what I know has come from years of troubleshooting, testing and refusing to let a computer win.
Just had another crash, indeed the same type of logging as before, and none when it actually crashed.
oot@RataNAS:/root ➜ # grep -o ‘crashkernel=[^ ]*’ /proc/cmdline
|| echo “No crashkernel parameter configured”
No crashkernel parameter configured
root@RataNAS:/root ➜ # cat /sys/kernel/kexec_crash_size 2>&1
0