Memory leak in icewhale-files-backup after ZimaOS update causing OOM kills

Insight & Feedback
1

After updating to the latest version of ZimaOS, the icewhale-files-backup service began consuming an excessive amount of RAM. Memory usage rapidly increases from approximately 3.5 GB to more than 10 GB, eventually triggering the Linux OOM Killer, which terminates the process. Since the service is configured with Restart=always, it automatically restarts and the same behavior repeats indefinitely. This causes server instability, including freezes and unexpected shutdowns. The issue did not occur before the update.

2

After updating to the latest version of ZimaOS, my server became unstable and experienced unexpected shutdowns.

After investigating the logs (journalctl), I found multiple Out Of Memory (OOM) events where the Linux kernel was repeatedly killing the icewhale-files-backup process due to excessive memory usage.

The service was consuming several gigabytes of RAM shortly after startup and eventually exhausted all available memory, triggering the OOM Killer. Since the service is configured with Restart=always, it restarted automatically and the same cycle repeated indefinitely.

Because I do not use the built-in Backup feature in ZimaOS, I decided to temporarily disable the service.

I ran:



sudo systemctl stop icewhale-files-backup.service
sudo systemctl disable icewhale-files-backup.service

Then I verified that it was disabled:



systemctl is-enabled icewhale-files-backup.service

Expected output:



disabled

After disabling the service:

  • Memory usage returned to normal.

  • The server became stable again.

  • No additional OOM events have occurred so far.

  • Docker, Immich, Syncthing, SMB, and Tailscale continue to work normally.

  • I have not noticed any negative side effects, since I do not use the Backup feature.

This appears to be a temporary workaround, not a permanent fix. If you rely on the built-in Backup functionality, disabling this service will prevent backup tasks from running.

I have also opened a bug report so the ZimaOS developers can investigate the issue. Hopefully this information helps anyone experiencing the same problem.

2 Likes
3

Did you try to copy/ back-up from or towared an external harddrive? I’m observant similair behaviour, but only when copying files from an external attached USB drive towards the storage pool.

4

Hello,
I am experiencing exactly the same behavior.
Paperless-ngx 3.0.4 and Immich 3.1.0 fail to start after the update from ZimaOS 1.6.2 to 1.7.0. RAM usage sits at 95–100% and briefly drops to 5% every few minutes.
I have configured several backup jobs, all are writing to USB drives.
After stopping the backup service as described above, everything is running stably again. However, I did have to open the settings for Paperless and Immich and save them without making any changes.

5

Update

I can confirm that masking the Backup service completely stopped the restart loop on my system.

The service was still restarting automatically because of Restart=always, so simply stopping or disabling it was not enough.

I executed:



sudo systemctl stop icewhale-files-backup.service
sudo systemctl disable icewhale-files-backup.service
sudo systemctl mask icewhale-files-backup.service

After masking it:

  • The service remains inactive.

  • No icewhale-files-backup process is running anymore.

  • RAM usage returned to normal.

  • The server is stable again.

Before stopping the service, it reached 4.2 GB of RAM in about 8 seconds, and previous OOM logs show it growing to around 10 GB RSS before being killed by the Linux OOM Killer.

This seems to confirm that the Backup service is responsible for the excessive memory usage on my system.

2 Likes
6

This can be confirmed !

7

Update:
I’ve run several tests, but unfortunately, the backup keeps causing problems.
Apparently, the backup process restarts sporadically in the background, and memory usage fluctuates between 85 and 100 percent.
The backup itself isn’t being performed correctly or completely.
This morning, I couldn’t even launch any applications, despite not having made any changes to the system.

Error: “Cannot connect to the Docker daemon at unix:/var/run/docker.sock. Is the docker daemon running?” (Yes, I tried to restart and reenable docker.socket, but issue clouldn’t be fixed)

I’ve now rolled back to version 1.6.2, and everything is running stable again.

Unfortunately, version 1.7.0 seems to still be in beta. :frowning:

Backup after 1.70 not working