A reliable rsync backup from ZimaOS to Synology (when the built-in Backup app lets you down)

Insight & Feedback
1

For people comfortable with SSH and a terminal.

Disclaimer: this works for me, but I take no responsibility for your data or hardware. Test it on your own setup first, verify the results yourself (see the verification notes — that’s the whole point), and especially triple-check the dd target disk in the OS-backup part — writing to the wrong disk destroys it.

(This is an updated version of an earlier post. Some of my first advice turned out to be wrong once I tested it at scale — I’ve corrected it below and flagged the mistakes, because the mistakes are the useful part.)

Why

ZimaOS’s built-in Backup app didn’t work for me. It reported a source count of ~225k files when the real count was ~1.2M — it silently skipped everything the backup process couldn’t read (my Nextcloud data dir is owned by www-data with restrictive perms, and the tool couldn’t traverse into it). It also hung after reporting “done,” and its Stop/Delete buttons didn’t reliably stop anything.

The specific bugs matter less than the pattern: a backup tool whose status you can’t verify is worse than none — it gives you false confidence. Every problem below, I only found because my replacement is transparent (real exit codes, real file counts, a log I can read, link counts I can check). I even caught a serious bug in my own setup this way (see “the include-order trap”). So the real lesson isn’t “rsync good, built-in bad” — it’s verify on disk, never trust a green checkmark.

Two parts: a versioned DATA backup (daily) and an OS-image backup (weekly).

Two ZimaOS constraints that shape everything

  1. Anything outside /DATA is wiped on reboot — including the system crontab; even the Zima-Cron app loses its tasks. Only Docker state (on your pool) survives. So scheduling must live in a Docker container, not cron. I use Ofelia (a tiny Docker job scheduler) as a Portainer stack.

  2. Your data is on the pool, not the small system disk. Docker’s data-root and AppData live on my SSD pool (/media/ZIMASSD); /DATA/AppData is just a symlink to it. Back up the pool path, and image the system disk separately (Part 2).

Part 1 — Versioned DATA backup

The architecture (corrected)

rsync mirrors the pool into a single CURRENT/ folder on the Synology. Versioning is done by snapshotting CURRENT into a dated folder with cp -al (hardlink copy) — near-zero extra space, since unchanged files are hardlinks, not copies. You get N daily browsable snapshots costing roughly one full copy + daily changes.

The critical correction from my first post: I originally ran cp -al from the ZimaOS side through the SMB/CIFS mount. Do not do this. Over the mount, every single hardlink is a separate network round-trip — for my ~1.2M files, cp -al was on track to take ~26 hours. The same cp -al run locally on the Synology takes ~2 minutes. So the snapshot must run server-side on the Synology, not across the mount.

Why not rsync’s own --link-dest? Because I lock the transfer key down with rrsync (below), and rrsync and --link-dest are fundamentally incompatible — rrsync anchors paths at its restricted root and forbids .., while --link-dest needs a ..-style relative path. rrsync will accept a mangled path and then silently full-copy, or reject it outright. Don’t fight it. Server-side cp -al sidesteps the whole problem.

Two locked-down SSH keys, each doing exactly one job

I use two separate keys, each forced (via command= in authorized_keys) to run one thing only, with no-pty,no-port-forwarding,no-agent-forwarding,no-X11-forwarding. Even if a key is stolen, it can’t get a shell or do anything else:

  • Mirror key → forced to rrsync (the official restriction wrapper), scoped to the one backup folder. Only rsync-into-that-folder, nothing else.

  • Snapshot key → forced to a tiny snapshot script on the Synology that does cp -al CURRENT/ → <timestamp>/ locally and prunes old snapshots. No arguments, self-contained, so there’s nothing to inject.

Enable rsync-over-SSH on the Synology (Control Panel → File Services → rsync), note the SSH port it gives you, and install both public keys with their forced commands.

The nightly flow (script run by the Ofelia container)

  1. Snapshot first — SSH with the snapshot key; the Synology runs cp -al locally (~2 min) and prunes to N. Taken before the mirror, so an interrupted transfer never costs you a good snapshot.

  2. Mirror — rsync -aH --delete the pool into CURRENT/ over the rrsync key. Only changed files move.

Excludes — and the include-order trap that silently dropped my Docker volumes

Exclude Docker’s runtime guts (overlay2, containers, image, buildkit, …) — they’re un-backuppable live objects (they throw rsync “exit 23”) and pointless to back up, since container filesystems rebuild from images. Also exclude .trash (the SMB recycle bin — mine was hiding 23 GB of junk) and @eaDir (Synology metadata).

But keep docker/volumes/ — that’s where your actual persistent container data lives (databases, Nextcloud data, etc.). Losing it would be catastrophic, and this is exactly where I nearly got burned:

rsync applies filter rules first-match-wins, in order. My first attempt was:

--exclude='/docker/**'
--include='/docker/volumes/***'   # DEAD RULE - never reached

The broad exclude matched docker/volumes/... first, so the include never fired — and my Docker volumes (including all my Nextcloud data) were silently left out while the backup reported success. The run said exit 0. Everything looked fine. Only checking the destination contents by hand revealed the gap. Correct order is includes before the broad exclude:

--include='/docker/'
--include='/docker/volumes/***'
--include='/docker/nginx-proxy-manager/***'
--exclude='/docker/**'

This is the single best argument for the “verify on disk” rule. A tool that just reports success would have hidden this for months.

Verify it actually works (do this, don’t skip it)

Hardlinks are real (snapshots share storage, not duplicating):

find "/volume2/.../<a-dated-snapshot>/AppData/<something>" -type f -printf '%n %p\n' | head

Link count 2 = sharing with CURRENT, correct. 1 = it’s secretly full-copying.

The data you care about is actually present (catches the include-order bug):

ls /volume2/.../CURRENT/AppData/          # your app configs
ls /volume2/.../CURRENT/docker/volumes/   # your container DATA - must be here
ls /volume2/.../CURRENT/docker/overlay2/  # should NOT exist (correctly excluded)

Part 2 — Weekly OS-image backup

Separate job, separate purpose: a bootable clone of the system disk for bare-metal recovery.

  • dd the system disk (mine: the 57.7 GB eMMC, /dev/mmcblk0) piped through gzip to a local file, write a sha256sum beside it, then rsync both to the Synology via the same restricted mirror key. Keep the newest few.

  • Mine compresses to ~23 GB — dd images empty-but-not-zeroed blocks too, so it’s bigger than “used space” suggests. Runs in well under an hour.

  • It’s a live image (crash-consistent), which is fine here because the active Docker data lives on the pool, not this disk.

  • Restore: verify the checksum, then gunzip -c image.img.gz | dd of=/dev/<system-disk>. Confirm the target disk three times — this is the one command that can destroy the wrong drive.

Recovery in one line

OS disk dies but pool survives → reimage the OS, reconnect the pool drives, boot; Docker stacks come back on their own because their data (volumes) is on the pool. Pool lost too → rsync a dated snapshot (not CURRENT) back down, redeploy the stacks — volumes restored means data intact.

Gotchas that cost me the most time

  • cp -al over the SMB mount is a trap. ~26 hours vs ~2 minutes run locally on the Synology. Do the snapshot server-side (a dedicated forced-command key), never across the mount.

  • rsync include/exclude order is first-match-wins. Put your --include rules before the broad --exclude, or they’re dead and you silently lose data. Verify the destination actually contains your volumes.

  • rrsync vs --link-dest: incompatible. rrsync forbids .. and mangles root-anchored paths; --link-dest then silently full-copies. Use server-side cp -al for versioning instead.

  • Killed transfers leave orphans on the Synology. A stopped run can leave an rrsync/rsync --server process alive on the Synology side, still holding rrsync’s directory lock (an flock, released only when the process dies). The next run fails with “another instance is already accessing this directory.” Fix: SSH to the Synology, ps aux | grep rsync, kill the leftover PIDs — not the SynoRsy+ … --daemon one (that’s Synology’s own service).

  • Don’t touch the Portainer/Ofelia stack while a backup runs. The transfer runs in a sibling container; updating the stack orphans it — it keeps running unsupervised while the script that manages it dies.

  • rrsync -wo (write-only) blocks read-back. Fine for pure pushing, but breaks anything that needs to read the destination. I use plain rrsync (still folder-locked).

  • CURRENT is not a backup — it’s the live mirror and can be mid-update. Always restore from a dated snapshot.

  • Above all: verify on disk. File counts, “done” messages, exit codes, progress bars — none of them prove your data is there. Check the destination contents and link counts yourself. Every real bug I hit (theirs and my own) was invisible at the “it said success” level.

Happy to share the actual scripts if anyone wants them.