ZimaBrain Snapshot / ZimaOS App Recovery feature

[gelbuilding]

Hi All,

We are developing and actively testing a new ZimaBrain Snapshot / ZimaOS App Recovery feature.

The goal is to restore installed ZimaOS applications and their persistent state—not to create another backup system for personal media, documents, downloads, or other user files.

This is currently an experimental development build. We plan to release a public test version soon after completing Cube and ZimaBoard verification.

What the recovery point currently captures

The current recovery bundle includes:

• Application persistent data under /DATA/AppData
• Saved ZimaOS Custom App definitions under /var/lib/casaos/apps
• Docker Engine named volumes discovered from verified Docker mountpoints
• File metadata, logical sizes, file counts and per-file SHA-256 evidence
• Separate checksummed archives for each recovery component
• A downloadable verified manifest describing exactly what was captured

This is an archive-based recovery point. It is not currently a native Btrfs copy-on-write snapshot.

Current Cube test

Our current ZimaCube test recovery point contains:

• AppData: 38,739 files and 27,291,439,327 logical bytes
• Custom App definitions: 38 files and 410,002 bytes
• Docker named volumes: 21 volumes, 2,103 files and 643,743,067 logical bytes
• Total: 40,880 files and approximately 27.94 GB of logical recovery data
• Runtime exclusions: 36 FIFO or Unix socket entries safely excluded and recorded

Runtime endpoints such as sockets and FIFOs are not persistent application data, so they are excluded with explicit manifest evidence rather than copied into the recovery bundle.

Verification process

The system does more than check whether an archive exists.

It currently performs these steps:

1. Measures all recovery sources and confirms destination capacity.
2. Requires the destination to be on a different filesystem from every captured source.
3. Creates separate archives and manifests for AppData, Custom App definitions and Docker named volumes.
4. Calculates archive and per-file SHA-256 checksums.
5. Fails verification if a captured source changes during creation.
6. Restores everything into a new isolated directory.
7. Validates archive paths before extraction.
8. Compares the restored file set, byte totals, metadata and per-file checksums against the captured manifest.
9. Records a persistent green VERIFIED state only after every check passes.

The isolated recovery test never overwrites live AppData and never starts applications from restored data.

New recovery progress interface

Large recovery points can take time because the verifier reads and checks the data multiple times.

The latest interface now displays:

• Current recovery phase
• Green progress bar
• Percentage completed
• Processed and total verification bytes
• Restored logical bytes
• Restored and total file counts
• Elapsed time
• Estimated time remaining
• Live heartbeat
• Safe cancellation control

Cancellation stops only the exact isolated recovery worker and removes only its incomplete isolated output. It does not alter the verified recovery bundle or live ZimaOS data.

The recovery page also opens immediately while stored recovery evidence is reverified in the background.

Current limitations

We are deliberately not claiming complete system recovery yet.

The current recovery point does not include:

• Core CasaOS or ZimaOS databases
• User accounts
• Extensions
• Firewall rules
• Scheduled-task state
• Running or stopped container state
• Docker network definitions
• Docker image layers or image digests
• Container writable layers
• User media, documents, downloads, VM disks, AI models or external backup repositories
• Application-consistent database dumps

Saved Custom App definitions are intended to recreate containers, but Docker images are expected to be pulled again.

Applications are stopped during controlled testing to improve data consistency. Proper application-level database backup hooks still need to be designed.

Performance observations

Our current test performs approximately 111.8 GB of verification work for about 27.94 GB of logical recovery data because archives and restored files are read several times:

1. Archive checksum verification
2. Archive-entry validation
3. File extraction
4. Independent restored-file checksum verification

Small files are significantly slower than large sequential files because each file requires creation, metadata handling and checksum work.

The current /DATA/AppData directory is stored on Btrfs but is not itself a Btrfs subvolume. /var/lib/casaos/apps is stored on ext4. Therefore, an instant native Btrfs snapshot cannot currently cover the complete recovery scope.

Feedback requested

We would appreciate guidance from IceWhale developers and experienced ZimaOS users:

1. Is /DATA/AppData, /var/lib/casaos/apps and Docker named volumes the correct foundation for application recovery?
2. Which additional CasaOS or ZimaOS databases are required to restore the installed-app inventory and Custom App metadata safely?
3. Is there an official or preferred API for exporting and recreating installed applications?
4. Which Docker network, image digest or runtime configuration should be retained?
5. What application-consistency hooks would you recommend for PostgreSQL, MariaDB, Redis and similar services?
6. Would a future ZimaOS storage layout using dedicated Btrfs subvolumes be practical?
7. Should recovery target the same ZimaOS installation, a clean installation, or both?
8. Are there additional safety checks you would require before allowing controlled live restoration?

We want to confirm that the recovery architecture is heading in the right direction before enabling live replacement.

This remains an experimental test branch. Live AppData restoration is still disabled while we complete verification, cancellation, performance and recovery-scope testing.

We hope to release a test build soon and would appreciate technical feedback from the community.

[isanto1306]

First of all, I think this is a really promising feature. It already feels much more like an application recovery system than a traditional backup solution, and I like that the focus is on verification and reliability instead of just creating archives.

The isolated restore test is probably my favorite part. Verifying that a recovery point actually works before touching live data is something many backup solutions don’t even do.

I do have a few suggestions that I think could make the feature even stronger over time.

Preserve the complete container configuration

Saving the Custom App definitions is a great start, but I’d also consider preserving everything needed to recreate containers exactly as they were.

That could include things like environment variables, restart policies, labels, network settings, devices, port mappings and security options. The closer the restored container is to the original one, the smoother recovery will be.

Include Docker networks

Docker networks are one thing I’d definitely like to see added in the future.

A lot of self-hosted setups rely on custom bridge networks, macvlan or ipvlan. Without those, containers may restore successfully but still won’t be able to communicate properly.

Store image digests

I completely understand why Docker images themselves aren’t included.

However, it would be really useful to save the image digest and registry information. That way the recovery process could pull the exact same image version instead of whatever happens to be tagged as latest months later.

Database consistency

This is probably the biggest area I’d look at next.

Simply archiving a running PostgreSQL or MariaDB data directory doesn’t always produce a consistent backup. In the future it might be worth supporting application-aware backup hooks like pg_dump, mysqldump, Redis SAVE, or temporarily pausing supported containers during the backup.

That would make recovery much more reliable.

Detect writable container layers

Some Docker containers unfortunately still write data outside mounted volumes.

Even if that isn’t technically ZimaOS’ fault, it would be great if ZimaBrain could detect this and warn the user before creating the recovery point.

Something as simple as:

“Container X contains data inside its writable layer that will not be recovered.”

would probably save a lot of confusion later.

Support bind mounts

Many users don’t use Docker named volumes for everything.

Instead they often store data in directories like:

• /DATA/docker

• /media/storage/docker

Supporting bind mounts would make the recovery feature useful for many more real-world installations.

Preserve filesystem metadata

It may also be worth preserving ownership, permissions, ACLs and extended attributes where possible.

Most users won’t notice the difference, but for more advanced deployments it can make restores much smoother.

Better reporting

If files change while a recovery point is being created, instead of only reporting that verification failed, it would be helpful to identify which application or directory changed.

That would make troubleshooting much easier.

Future ideas

One thing that could be really useful is a small metadata file stored with every recovery point.

Besides the manifest, it could include information like:

• ZimaOS version

• CasaOS version

• Docker version

• Kernel version

• Architecture

• Creation date

• Hostname

• Number of installed applications

That would make recovery and troubleshooting much easier, especially months later.

Another idea would be a Recovery Readiness Report.

After creating a recovery point, ZimaBrain could automatically highlight anything that might prevent a complete restore.

For example:

• ✓ AppData protected

• ✓ Docker volumes protected

• ✓ Custom Apps protected

• Docker networks not included

• PostgreSQL running without an application-aware backup

• Two containers using writable layers

• Three containers using the latest image tag

That would give users a clear picture of how complete their recovery really is and what they could improve before they actually need it.

Overall, I think you’re building something that could become one of the standout features of ZimaOS. The current foundation already looks very solid, and adding things like database consistency, Docker network recovery, bind mount support and image reproducibility would make it even more complete.

[gelbuilding]

@isanto1306 Thank you for taking the time to write such a detailed and thoughtful response. You have helped us enormously with ZimaBrain already, and your feedback continues to push the project in the right direction.

You have understood exactly what we are trying to build. This is not intended to be another tool that creates an archive and simply assumes it will work. The goal is verified application recovery, with an isolated restore proving the recovery point before anything can touch live data.

Your suggestions align very closely with our roadmap.

The current recovery point protects:

• /DATA/AppData
• Saved Custom App definitions
• Docker named volumes
• File ownership, permissions, timestamps and symbolic links
• A detailed manifest with file counts, logical bytes, archive sizes and SHA-256 checksums

However, we clearly identify that it is not yet a complete system recovery point.

The next important areas are exactly what you identified:

• Full container configuration and runtime settings
• Docker networks, including custom bridge, macvlan and ipvlan
• Image registry details and immutable image digests
• External bind mounts outside /DATA/AppData
• Per-container writable-layer detection
• Application-aware PostgreSQL, MariaDB and Redis consistency
• ACL and extended-attribute preservation
• Better identification of files or applications that change during capture
• Host metadata such as ZimaOS, CasaOS, Docker and kernel versions

Your point about database consistency is particularly important. We have already confirmed that simply copying a live database directory cannot be described as application-consistent. We are currently testing with application writers stopped, but future versions need controlled hooks such as pg_dump, mysqldump, Redis persistence commands or safe container quiescing.

I also really like your Recovery Readiness Report idea. It fits ZimaBrain’s verifier-first design perfectly. Instead of simply saying “snapshot created,” it could clearly report:

• What is protected
• What was successfully restored and verified
• What is excluded
• Which containers rely on writable layers
• Which databases need application-aware protection
• Which images use mutable tags such as latest
• What could prevent a complete recovery

That would give the user an honest recovery score rather than a false sense of security.

We are currently running repeated real isolated recovery tests and fixing every failure we find. The recovery interface now shows the active phase, percentage completed, bytes and files processed, elapsed time, ETA and a safe cancellation control.

We still consider this experimental and will not release it publicly until the verified restore completes correctly across AppData, Custom App definitions and Docker named volumes.

Thank you again. You have been one of the people who has genuinely helped shape ZimaBrain, and this feedback gives us a very strong technical roadmap for the next stages.

[isanto1306]

Thanks, I really appreciate that.

It’s great to hear that my feedback has been useful. I honestly enjoy following the progress of ZimaBrain because it’s trying to solve a real problem instead of just adding another backup feature. The focus on verified recovery is what makes it stand out for me.

I also think the Recovery Readiness Report could become a really helpful feature. Sometimes users don’t even realize a container is writing outside persistent storage or that a database may not restore cleanly. Showing those things up front would give people a much better idea of how recoverable their system actually is.

I’m looking forward to trying future builds and seeing how the project evolves. If I can help by testing new features, breaking things, or sharing feedback from real-world use, I’m always happy to help.

Keep up the great work!

[gelbuilding]

Thank you, @isanto1306. That genuinely means a lot.

You have understood the purpose of ZimaBrain perfectly. Creating a backup is easy; proving that it can actually recover the applications correctly is the difficult and important part.

I agree that the Recovery Readiness Report could become one of its strongest features. Users should know before a failure occurs which containers use writable layers, which databases may be inconsistent, which image versions cannot be reproduced and what data remains outside the recovery point.

We would be honoured to have you help test future builds. Careful testers who are willing to break things and explain what happened are exactly what this project needs. We are already finding and fixing issues through real isolated recovery tests, and that evidence is making the system stronger with every build.

Once we have a controlled test release ready, I will contact you. Thank you again for your support and for helping shape ZimaBrain’s direction.

[isanto1306]

Thank you, I really appreciate it.

I’d be happy to help test future builds. I use ZimaOS every day, so I can try things in real-world scenarios and hopefully catch issues that don’t always show up in a controlled environment.

I’ll make sure to provide detailed feedback, logs and clear reproduction steps whenever I find something. That usually makes it much easier to track problems down.

Looking forward to the test build, and thanks again for the opportunity.