ZimaOS-HD fills up unexpectedly on ZimaBoard 2: root cause is rclone VFS cache runaway, plus a cleanup gotcha when re-authorizing cloud drives

Insight & Feedback
1

Summary

If you’re seeing the “Device built-in storage is almost full” or “Running out of space” warning on ZimaOS-HD (the eMMC) despite having all your app data on an SSD, the most likely culprit is the rclone VFS cache growing without bounds under /DATA/.cache/rclone/. This can fill the entire 45 GB /DATA partition on a ZimaBoard 2 in a matter of hours, especially when you have OneDrive (or any cloud drive) mounted and any process that walks the file tree (backups, indexers, sync jobs) touches many files.

This post covers two things:

  1. How to prevent the runaway with a systemd drop-in (recommended for anyone using cloud drives).

  2. How to recover if your eMMC is already full.

All commands assume ZimaOS 1.6.2 on a ZimaBoard 2. Adjust paths if your storage layout differs. Tested against actual production behavior.

Why this happens

ZimaOS runs a persistent rclone rcd daemon (rclone.service) that talks over a Unix socket at /run/rclone/rclone.sock. When you add a cloud drive from the UI, icewhale-files sends a mount/mount RC call with these VFS options (base64-decoded from actual daemon logs):

json

{
  "CacheMode": "full",
  "ChunkSize": "1M",
  "ChunkStreams": 8,
  "ReadAhead": "128M",
  "FastFingerprint": true,
  "DirCacheTime": "30s"
}

Notice what’s not there: no VfsCacheMaxSize, no VfsCacheMaxAge, no VfsCacheMinFreeSpace. This means the cache has no upper limit and no eviction policy beyond rclone’s defaults, which effectively don’t kick in until the disk is completely full.

Also, the daemon runs with Environment=HOME=/DATA in its unit file. Since rclone’s default cache directory is ~/.cache/rclone/, all VFS cache material lands on /DATA/.cache/rclone/ — that is, on the eMMC. On a ZimaBoard 2, that’s a 45 GB partition shared with system state, logs, databases, and container data. It fills quickly.

The good news: because icewhale-files does not pass cache-limit flags in the RC call, environment variables on the daemon process take effect. That gives us a clean and reversible way to fix this.

Part 1 — Preventive tuning (do this once, forget about it)

This creates a systemd drop-in that moves the rclone VFS cache to your SSD (if you have one) and enforces hard limits. It survives reboots and rclone restarts.

Prerequisites:

  • Root access (via SSH or the web SSH console).

  • An SSD mounted at /media/SSD-Storage (adjust the path if yours differs — for example, if it’s a HDD RAID at /media/Safe-Storage, use that instead).

  • ZimaOS 1.6.2 (should also work on later versions unless the service unit changes).

Steps:

bash

# 1. Create the cache directory on the SSD
mkdir -p /media/SSD-Storage/rclone-cache
chmod 700 /media/SSD-Storage/rclone-cache

# 2. Create the systemd drop-in
mkdir -p /etc/systemd/system/rclone.service.d/
cat > /etc/systemd/system/rclone.service.d/override.conf <<'EOF'
# Prevent rclone VFS cache from filling ZimaOS-HD (eMMC).
# Moves cache to SSD, applies size cap, age eviction, and free-space guardrail.
[Service]
Environment=RCLONE_CACHE_DIR=/media/SSD-Storage/rclone-cache
Environment=RCLONE_VFS_CACHE_MAX_SIZE=10G
Environment=RCLONE_VFS_CACHE_MAX_AGE=24h
Environment=RCLONE_VFS_CACHE_MIN_FREE_SPACE=20G
EOF

# 3. Reload systemd and verify the drop-in was picked up
systemctl daemon-reload
systemctl show rclone.service --property=Environment --no-pager

# 4. Restart the daemon
systemctl restart rclone.service

# 5. Verify the mount comes back (may need a click in Files UI)
mount | grep -iE "onedrive|gdrive|dropbox"

Tuning notes:

  • RCLONE_VFS_CACHE_MAX_SIZE=10G — Soft ceiling for cache size. Adjust based on your access patterns. If you regularly re-access the same large files, raising this to 20 GB or 50 GB improves hit rate. On a spacious SSD, being generous here is fine.

  • RCLONE_VFS_CACHE_MAX_AGE=24h — Evicts entries older than 24 hours. Good balance between hit rate for daily workflows and not accumulating stale material for weeks.

  • RCLONE_VFS_CACHE_MIN_FREE_SPACE=20G — The critical safety net. Forces eviction whenever the filesystem containing the cache drops below 20 GB free. This is the guardrail that catches any edge case where MAX_SIZE isn’t respected. Set this to whatever you consider a comfortable minimum free space on your SSD.

  • RCLONE_CACHE_DIR=/media/SSD-Storage/rclone-cache — Moves the cache off the eMMC. Also better for eMMC endurance since VFS cache is hot data with lots of writes.

Verification after applying:

Let the mount run under normal use for a while (or trigger a backup that reads many files). Then check:

bash

# Cache should be growing on the SSD, not the eMMC
du -sh /media/SSD-Storage/rclone-cache/
du -sh /DATA/.cache/rclone/

# /DATA usage should stay stable
df -h /DATA

Expected: the SSD cache dir grows to whatever your workload demands (up to your MAX_SIZE), and the eMMC path stays at a few KB (metadata only).

Part 2 — Recovery: what to do when /DATA is already at 100%

If you hit the “48 MB Available” (or similar) warning, here’s how to recover before applying the preventive fix.

Verify the diagnosis first:

bash

# Confirm /DATA is really full
df -h /DATA

# Confirm the rclone cache is the culprit
du -xh --max-depth=2 /DATA/.cache/rclone/ 2>/dev/null | sort -rh | head -10

If the biggest consumer is /DATA/.cache/rclone/vfs, this is your problem. If it’s something else (a rogue log, a container volume, an OTA bundle), the recovery below won’t apply directly.

Recovery steps (for the rclone VFS cache case):

bash

# 1. Stop the rclone daemon (this unmounts all cloud drives)
systemctl stop rclone.service

# 2. Force-unmount any lingering FUSE mounts
mount | grep fuse.rclone | awk '{print $3}' | xargs -I {} umount {} 2>/dev/null

# 3. Verify nothing rclone-related is mounted
mount | grep fuse.rclone
# (should return empty)

# 4. Purge the VFS cache
rm -rf /DATA/.cache/rclone/vfs/*

# 5. Confirm space is reclaimed
df -h /DATA

# 6. Start the daemon back up
systemctl start rclone.service

# 7. Reconnect cloud drives from the Files UI (they should remount automatically)

Important: After recovery, immediately apply Part 1 (the preventive tuning). Without it, the cache will refill and you’ll be back in the same situation within hours or days.

Bonus: cleanup gotcha when re-authorizing a cloud drive

There’s a related issue worth mentioning. If you delete and re-authorize a cloud drive from the UI, the reconnection can fail with this error in the logs:

rclone[]: ERROR : rc: "mount/mount": error: failed to mount FUSE fs: 
"/media/<remote_id>" is not empty, use --allow-non-empty to mount anyway

Root cause: icewhale-files doesn’t fully clean up the mount point directory when removing a cloud drive. Residual content stays behind. FUSE (correctly) refuses to mount over a non-empty directory to prevent silent shadowing of user data.

Workaround:

bash

# Verify what's inside the residual directory (should be leftover metadata, not user data)
ls -la /media/<remote_id>/
find /media/<remote_id> -maxdepth 3 2>/dev/null | head -20

# If confirmed to be residual only, empty it
rm -rf /media/<remote_id>/*
rm -rf /media/<remote_id>/.[!.]* 2>/dev/null

# Now re-trigger the mount from the Files UI

Note that IceWhale reuses the remote ID across re-authorizations (same hash), so the mount point path stays the same. The residual data is metadata from the previous session, not your actual cloud content (which lives on the cloud provider, not locally).

Suggestions for IceWhale

If anyone from the ZimaOS team is reading this, three things would meaningfully improve the experience:

  1. Add sane cache limits by default to the mount/mount RC call, or expose them as user-configurable settings in the Files UI. VfsCacheMinFreeSpace would catch this class of failure without requiring users to know rclone flags.

  2. Move the default cache location off the eMMC when an SSD is present. This alone would prevent most incidents, especially given ZimaBoard 2’s modest 45 GB eMMC partition.

  3. Fix the cleanup path when a cloud drive is deleted from the UI: unmount, remove the remote from rclone.conf, and remove the mount point directory contents before the operation returns 200. Right now step 3 is missing (or silently failing), which turns re-authorization into a support incident.

Happy to provide additional logs or reproducers if it helps. This was diagnosed from actual daemon logs on ZimaOS 1.6.2 on ZimaBoard 2, and both the tuning and the workaround have been validated in production.

1 Like