ZimaOS sporadic very slowly?

Operating System
,
1

Hey zima community, i need your help.

I’m new to Linux and ZimaOS.
I set up my server about four weeks ago, and the NAS has been running on the system ever since.
From the very beginning, I’ve had a problem where the operating system runs extremely slowly at times.
It often starts right at the login screen; loading the page in my local network browser can take several minutes before I can even log in. Once I do log in, the dashboard seems to take forever to load—or never finishes loading at all. The apps don’t appear, so I end up rebooting; sometimes it runs significantly faster after a reboot.

It often happens that the server has been running for a few hours and I want to save data to the NAS (I’ve set up a Samba share for this). Creating a new folder takes several minutes before I can actually save a file.
This issue also improves drastically after a reboot.

Does anyone have any idea what might be going wrong here or how I can fix this problem?

On Windows, I’d suspect driver issues or missing drivers, but I have no idea how to troubleshoot this on Linux.

System specs:
CPU: Intel i7 8700K
RAM: 32GB DDR4
Motherboard: MSI MPG Z390 Carbon
System SSD: Samsung 850 256GB
Storage: 2x Seagate IronWolf 8TB (RAID 1)

ZimaOS Version: 1.6.1

Thanks in advance =)

2

From what you describe, I would not assume a missing driver straight away.

Because the slowdown affects both the ZimaOS web login/dashboard and Samba file creation, I would first check for system load, memory/swap pressure, disk problems, or a storage/RAID issue. A reboot temporarily clearing the issue usually points to something building up over time.

Before reinstalling anything, please run these checks when the system is slow and post the output here.

free -h

uptime

df -h

Then also check disk and RAID status:

lsblk -o NAME,SIZE,MODEL,SERIAL,FSTYPE,MOUNTPOINT

cat /proc/mdstat

And check recent system errors:

dmesg | tail -n 80

The important things to look for are:

  1. RAM or swap being heavily used
  2. Very high load average
  3. One disk causing I/O errors
  4. RAID 1 rebuilding, degraded, or stuck
  5. The system SSD or HDDs reporting errors

Since you are using 2x 8TB drives in RAID 1, a slow or failing disk can make the whole NAS feel extremely slow, especially with Samba writes.

Also, if this started from the beginning on ZimaOS 1.6.1, it would be useful to know whether you installed fresh or upgraded from an earlier version.

Post the outputs above and we can narrow it down properly.

3

Thanks for your answer.

it has taken time how to open the terminal to try your suggested commands.

Today i boot up the server in the morning around 8:00 am, now im back from work and want to connect from my pc to the web UI but it wont load, again. I cant also open the terminal on IP:7681

directly on server pc, the start screen terminal view “zimaos hat sarted successfully…. you can connect via browser…” isnt displayed.

so i tried to connect with zima client on my phone and see i got Access. I can open some files from my NAS, browse between folders and so on, just with a little more loading time, but i have access.

I have really no idea whats wrong….

While I was messing around with the terminal to test things out, I went ahead and entered your commands to see what the output would be. I’ve saved the information that was displayed.
Could that help with troubleshooting?

I Uploaded the .txt file.

Thanks in advance.

TestingTerminal.txt (92,6 KB)

4

Thanks for uploading the log, that definitely helps.

From what I can see, the basic system checks looked normal at the time you ran them. The CPU load was low, RAM was fine, swap was not being used, /DATA had plenty of free space, and the RAID 1 array was active with both disks online.

The only issue is that these checks were taken shortly after reboot, when the system was running normally again. So they may not show the actual problem state.

One thing I did notice in the log is a BTRFS warning showing one recorded corruption error on md0. I am not saying this is definitely the cause of the Web UI hanging, but it is something worth checking properly.

Since Zima Client on your phone still works and you can still browse files, the whole server does not sound completely frozen. It sounds more like the ZimaOS Web UI, terminal service, Docker service, or possibly disk I/O is getting stuck after the system has been running for a while.

Next time after a reboot, could you please run these and save the output again?

journalctl -b --no-pager | tail -n 200

dmesg | grep -Ei "error|fail|btrfs|ata|reset|timeout|i/o|corrupt|md0"

docker ps

And if possible, when the system becomes slow again, before rebooting, run:

uptime && free -h && cat /proc/mdstat

At this stage I would not reinstall yet. First I would try to confirm whether the issue is coming from the Web UI services, Docker, the RAID/BTRFS filesystem, or a disk/SATA timeout after the machine has been running for a few hours.

5

while slow.txt (158,7 KB)

Hello again and thanks for your help.

Today i got the situation again that the system is slow. The web ui in the browser needed to load the login page around 10 minutes. After login io opened the terminal and execute your commands. Now the apps in the dashboard are still not loaded.

After i do a reboot of the system the login page will load directly and the apps on the dashboard are instantly shown after login.

i uploaded the terminal save “while slow.txt” for your checking.

I hope thats already helpful to find the problem and thanks a lot for your help.

6

Thanks, this log is more useful because it was taken while the system was slow.

From what I can see, the main system itself does not look overloaded:

CPU load is very low
RAM is fine
Swap is not being used
/DATA has plenty of free space
RAID 1 is still active with both drives online
The Docker containers are running

So at this stage I do not think this is a simple CPU, RAM, swap, or full disk issue.

The problem looks more focused around the ZimaOS Web UI / app-management side. That would also match what you are seeing: the NAS files can still be accessed, but the browser UI takes a very long time and the dashboard apps do not load properly.

I also still see the BTRFS message showing one recorded corruption count on md0, but I do not want to say that is the cause yet because your RAID is online and the general system load looks normal.

Can you please run these while the Web UI is slow again?

journalctl -b -u zimaos-app-management -u zimaos-gateway -u zimaos-user -u zimaos-message-bus --no-pager | tail -n 200

docker stats --no-stream

docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}"

The first command should show whether one of the ZimaOS Web UI services is hanging or throwing errors.
The second command will show if one container is quietly using high CPU or memory.
The third command gives a cleaner container status list.

For now, I would not reinstall yet. The logs are starting to narrow this down, and it looks more like a service/UI issue than the whole server failing.

7

Hello, and thanks for your reply.
Unfortunately, I’ve been away for work over the past few weeks and haven’t been able to capture the diagnostic data yet.
I did actually have to reboot the system once or twice because everything was running quite slowly—even via the Zima client—though that could also be partly attributed to the data connection.

As soon as the web interface starts running slowly again, I’ll capture and post the diagnostic data.

In the meantime, I’d like to resolve this btrfs error. I’ve created a backup of the NAS data on an external hard drive.
Since I don’t have much experience with Linux yet: can I simply run the following command, and will that fix the error?

btrfs check --repair

Thanks in advance

8

Thanks for the update.

Good that you already made a backup first. That is the correct first step.

I would not run this command yet:

btrfs check --repair

btrfs check --repair is not a normal first repair step. It should only be used as a last resort, and only when the filesystem is unmounted. Running it without confirming the real state of the filesystem can make the damage worse.

The safer first step is to check the filesystem status and run a BTRFS scrub, because scrub is designed to verify data and metadata while the filesystem is mounted.

Please run these first:

btrfs filesystem show
btrfs device stats /media/YOUR-NAS-MOUNT-PATH
btrfs scrub status /media/YOUR-NAS-MOUNT-PATH

Then, if no scrub is currently running, start a scrub:

btrfs scrub start -Bd /media/YOUR-NAS-MOUNT-PATH

Replace /media/YOUR-NAS-MOUNT-PATH with the actual mounted NAS path.

After it finishes, please post the output of:

btrfs scrub status /media/YOUR-NAS-MOUNT-PATH
btrfs device stats /media/YOUR-NAS-MOUNT-PATH
journalctl -k --no-pager | grep -iE "btrfs|checksum|corrupt|error|md0" | tail -n 100

Because your RAID 1 was still online and the system load looked normal in the earlier log, I would first verify whether BTRFS can correct or isolate the recorded error through scrub before trying any offline repair command.

So for now: backup is good, but please do not run btrfs check --repair yet.

9

Hi, I have run the commands.
I assume that this was just the step to gather information, and that the repair will take place in the next step!?

I hope everything was done correctly; I can’t really tell whether the scrub was running or not, but I assume there would have been a notification if it hadn’t.

here is the output:

root@ZimaOS:/root ➜ # btrfs filesystem show
Label: none  uuid: 9f4acc3d-3df7-a677-48ab-0612830fbb17
        Total devices 1 FS bytes used 2.03TiB
        devid    1 size 7.28TiB used 2.04TiB path /dev/md0


root@ZimaOS:/root ➜ # btrfs device stats /media/_NAS1_
[/dev/md0].write_io_errs    0
[/dev/md0].read_io_errs     0
[/dev/md0].flush_io_errs    0
[/dev/md0].corruption_errs  611
[/dev/md0].generation_errs  0


root@ZimaOS:/root ➜ # btrfs scrub status /media/_NAS1_
UUID:             9f4acc3d-3df7-a677-48ab-0612830fbb17
        no stats available
Total to scrub:   2.03TiB
Rate:             0.00B/s
Error summary:    no errors found
root@ZimaOS:/root ➜ # journalctl -k --no-pager | grep -iE "btrfs|checksum|corrupt|error|md0" | tail -n 100
Jul 09 07:26:50 ZimaOS kernel: ACPI: Early table checksum verification disabled
Jul 09 07:26:50 ZimaOS kernel: xor: automatically using best checksumming function   avx       
Jul 09 07:26:50 ZimaOS kernel: Btrfs loaded, zoned=no, fsverity=no
Jul 09 07:26:54 ZimaOS kernel: md: md0 stopped.
Jul 09 07:26:54 ZimaOS kernel: md/raid1:md0: active with 2 out of 2 mirrors
Jul 09 07:26:54 ZimaOS kernel: md0: detected capacity change from 0 to 15627788928
Jul 09 07:26:54 ZimaOS kernel: BTRFS: device fsid 9f4acc3d-3df7-a677-48ab-0612830fbb17 devid 1 transid 1043 /dev/md0 (9:0) scanned by zimaos-local-st (852)
Jul 09 07:26:54 ZimaOS kernel: BTRFS info (device md0): first mount of filesystem 9f4acc3d-3df7-a677-48ab-0612830fbb17
Jul 09 07:26:54 ZimaOS kernel: BTRFS info (device md0): using crc32c (crc32c-intel) checksum algorithm
Jul 09 07:26:54 ZimaOS kernel: BTRFS info (device md0): using free-space-tree
Jul 09 07:26:55 ZimaOS kernel: BTRFS info (device md0): bdev /dev/md0 errs: wr 0, rd 0, flush 0, corrupt 611, gen 0

thanks for your help =)

10

Thanks for posting the output.

Yes, I believe this step was mainly to gather information first. I would still not run this command at this stage:

btrfs check --repair

From your output:

[/dev/md0].write_io_errs    0
[/dev/md0].read_io_errs     0
[/dev/md0].flush_io_errs    0
[/dev/md0].corruption_errs  611
[/dev/md0].generation_errs  0

So BTRFS has recorded 611 corruption errors on /dev/md0, but it is not showing read/write/flush/generation errors. Your RAID1 also appears to be online:

md/raid1:md0: active with 2 out of 2 mirrors

The important part is that I do not see the output from the actual scrub start command. btrfs scrub status only shows the scrub status. It does not start the scrub.

I suggest running this now:

btrfs scrub start -Bd /media/_NAS1_

This may take a while because it has about 2 TiB to check. The -B option keeps it in the terminal until it finishes, so please let it complete.

After it finishes, I suggest posting the output of:

btrfs scrub status /media/_NAS1_
btrfs device stats /media/_NAS1_

If the scrub reports corrected errors, that is useful information. If it reports uncorrectable errors, then the next step is different.

For now, I would not run btrfs check --repair. I believe that is still a last-resort offline repair command, not the next normal step.