ZimaBrain CE — Local ZimaOS Diagnostic Assistant

ZimaBrain CE — Local ZimaOS Diagnostic Assistant

Hi everyone,

I wanted to share a community diagnostic tool for ZimaOS called ZimaBrain CE.

This project was built with strong guidance from Holger / Lintux, and I want to give him proper credit for that. His direction around verifier-first thinking, evidence layers, security awareness, firewall checks, SMART interpretation, and safe diagnostic flow shaped a lot of how this tool works.

The purpose of ZimaBrain CE is simple: make ZimaOS troubleshooting more structured, evidence-based, and safer.

Many ZimaOS problems are not one single issue. They usually involve a mix of storage, Docker, app paths, ports, firewall rules, SMART health, mount points, containers, and sometimes assumptions made from incomplete evidence.

ZimaBrain CE was built to slow that process down and help users work from evidence first.

The basic principle is:

Analyze first → Verifier second → Explainer third → Repair guide last

That means ZimaBrain should first look at local evidence from the system, explain what it found, and then suggest the safest next step.

What it does

ZimaBrain CE runs locally on your ZimaOS machine and checks things like:

  • System and storage evidence
  • SMART / NVMe health
  • Docker containers
  • App ports and exposed services
  • LAN reachability hints
  • Firewall / DOCKER-USER / ZFW-style checks
  • Local dashboard evidence
  • Installed app awareness
  • Redacted support report export

It also includes a built-in native dashboard, so you can quickly see detected disks, container state, and local evidence without needing a separate visual dashboard container.

Why we built it

A common issue is that users see something like SMART PASSED and assume the disk is healthy.

But SMART PASSED by itself is not enough.

For example, a disk can show overall SMART passed while still having risky attributes such as:

  • Reallocated sectors
  • Reported uncorrectable errors
  • Pending sectors
  • Offline uncorrectable sectors
  • CRC errors
  • Command timeouts
  • NVMe unsafe shutdowns
  • NVMe error log entries

ZimaBrain CE checks those detailed values instead of trusting the top-level SMART result alone.

Another common issue is apps not opening. Sometimes the app is running, but the port is not exposed. Sometimes it is bound to localhost. Sometimes Docker published the port but firewall or LAN reachability is still the problem. ZimaBrain tries to separate those cases using local evidence.

Security note

ZimaBrain CE runs in Elevated Local Verifier mode.

That means it needs host visibility and read-only Docker socket access so it can verify ZimaOS disks, containers, ports, firewall state, and system evidence.

Because of that:

  • Keep it on your trusted LAN
  • Set a password on first run
  • Do not expose it publicly
  • Use the redacted support report when sharing output

Install through ZimaOS Custom App

Install this through ZimaOS → App Store → Custom App.

Paste the compose file below into Custom App and deploy it.

After install:

  • Open the tile
  • Set the first-run password
  • Access ZimaBrain on port 8601
  • Ask a test question such as Are my disks healthy?

If you already installed an older test build, delete the old app first and reinstall so Docker pulls the latest image.

Current status

This is still a community beta, but it has now been tested on multiple ZimaOS systems.

Recent testing confirmed:

  • Custom App install works
  • Tile opens on port 8601
  • First-run password setup works
  • Native dashboard loads
  • ZimaBrain logo appears on the tile
  • ZimaBrain CE tips popup appears on start
  • SMART / disk health answer works
  • Dashboard now flags risky SMART attributes correctly
  • Redacted support report downloads
  • App-specific awareness has improved

Docker image

gelbuilding/zimabrain-ce:1.6.0-beta

Compose

name: zimabrain-ce

services:
  zimabrain-ce:
    image: gelbuilding/zimabrain-ce:1.6.0-beta
    container_name: zimabrain-ce
    restart: unless-stopped
    privileged: true
    pid: host
    ports:
      - "8601:8601"
    volumes:
      - /DATA/AppData/zimabrain-ce:/data
      - /DATA/AppData/zimabrain-ce/docs:/app/docs:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/hostname:/host/etc/hostname:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
    extra_hosts:
      - "host.docker.internal:host-gateway"
    labels:
      icon: https://raw.githubusercontent.com/Jacko88888/zimabrain-ce/main/app/assets/zimabrain-ce-logo.svg
      title: ZimaBrain CE
      description: Local ZimaOS diagnostic assistant
      index: /
      scheme: http
      port_map: "8601"

Good first questions to try

Are my disks healthy?
Is port 8601 reachable on LAN and is firewall blocking it?
Why is Immich not opening?
Which containers are exited?

Final note

This is a verifier-first tool. It will not always say “confirmed” unless the evidence is strong enough. Sometimes it will say Partially Verified, and that is intentional.

For example, LAN reachability can often be checked from the host side, but it cannot be fully proven from another device unless an external LAN client also tests access.

The aim is to reduce unsafe advice, improve troubleshooting flow, and make support reports easier to understand.

Again, special thanks to Holger / Lintux for his explicit guidance and technical direction on how this should be structured.

7 Likes

Thank you for the effort @gelbuilding, it has a ton of potential to help users troubleshoot their own issues and also upload relevant data here whenever asking for help. Keep pushing it!

2 Likes

Feedback after testing ZimaBrain CE on my ZimaOS system

First of all, great work. I spent quite a while testing ZimaBrain CE on my production ZimaOS system and it actually helped me find several real configuration problems that had gone unnoticed.

Issues ZimaBrain helped me fix

  • Removed an old startup.service that referenced a missing /DATA/startup.sh.

  • Fixed my plex-delay.service, which still referenced an old Docker container name (plex instead of big-bear-plex).

  • Identified an auditd.service failure caused by incorrect ownership of /var/log/audit and audit.log.

  • Confirmed that my SMART data was healthy while still warning about CRC counters and NVMe unsafe shutdowns instead of relying only on “SMART PASSED”.

  • Correctly identified exited containers.

  • Correctly detected published Docker ports and firewall configuration.

These were real issues, not false positives.

Suggestion: Differentiate historical boot failures from current problems

The biggest improvement I would suggest is handling watchdog services more intelligently.

In my case:


tailscaled-watchdog.service  -> FAILED
tailscaled.service           -> RUNNING

zfw-ui-watchdog.service      -> FAILED
zfw-ui.service               -> RUNNING

The watchdog units fail during boot because of boot ordering, but the actual services are started successfully afterwards and continue running normally.

Currently ZimaBrain reports them as:

Failed systemd unit detected

Technically this is correct, but it may lead users to believe that Tailscale or ZFW are currently broken, when in reality both services are healthy.

A more useful message could be something like:

Historical boot watchdog failure detected.

Monitored service is currently active and healthy.

No current outage detected.

That would make the report much easier to interpret.

Suggestion: Correlate related services

When a failed systemd unit is detected, it would be helpful if ZimaBrain also checked whether a related service is currently running.

Examples:

  • tailscaled-watchdog.service → check tailscaled.service

  • zfw-ui-watchdog.service → check zfw-ui.service

If the monitored service is active, the alert severity could be reduced automatically.

Overall impression

For a community beta, I’m genuinely impressed.

During a single testing session ZimaBrain helped me discover and fix:

  • one obsolete startup service

  • one broken Docker restart service

  • one auditd configuration problem

It also correctly analyzed SMART health, Docker state, ports, firewall exposure and storage.

The verifier-first approach works well because it encourages users to investigate actual evidence instead of making assumptions.

Overall this is already a very useful diagnostic tool for ZimaOS, and I think adding service correlation and better classification of historical boot failures would make it even better.

Great work, and thanks for sharing it with the community.

2 Likes

Thanks mate, I really appreciate that.

That is exactly the goal. I am trying to make it easier for users to collect proper evidence first, instead of everyone guessing from screenshots or half the information.

One of the useful parts is the local Zima visual dashboard. You can see your own machine locally, including detected disks, container state, ports, and system evidence, without needing to rely only on the ZimaOS WebUI or separate screenshots.


The support report side is important too. If someone has a disk, Docker, port, firewall, app path, or SMART issue, ZimaBrain should help them ask the right question and share useful redacted data without exposing private details.

Some good questions users can test with are:

Are my disks healthy?
Do any of my disks have risky SMART attributes?
Which containers are exited?
Why is Immich not opening?
Is port 8601 reachable on LAN?
Is my firewall blocking this app?
Which apps have exposed ports?
Show me a safe support report I can share on the forum.

Still early beta, but I will keep improving it and tightening the verifier-first flow.

Another important part is how ZimaBrain answers.

It does not just give a generic answer. It tries to answer from real local evidence collected from the machine and the dashboard. So when it talks about disks, containers, ports, SMART values, firewall state, or exposed services, it should be based on what it can actually verify from the system.

That is the main idea behind the verifier-first flow. If the evidence is strong, it can say confirmed. If the evidence is incomplete, it should say partially verified instead of pretending to know.

1 Like

Thank you very much for taking the time to test it properly and write such detailed feedback.

This is exactly the kind of testing I was hoping for. The most important part for me is that ZimaBrain helped find real issues on a production ZimaOS system, not just theoretical warnings.

The points you raised about the watchdog services are very good.

I agree that there is a difference between:

watchdog failed during boot

and:

the real service is currently down

If tailscaled-watchdog.service failed during boot, but tailscaled.service is now running normally, then ZimaBrain should not make that look like a current outage.

Same with ZFW. If the watchdog failed because of boot timing, but the actual service is active and healthy, then the report should reduce the severity and explain it more clearly.

I like your suggested wording:

Historical boot watchdog failure detected.
Monitored service is currently active and healthy.
No current outage detected.

That is much clearer for users.

Service correlation is also a very good suggestion. A failed helper/watchdog unit should be checked against the real service it is meant to protect before the warning is classified as serious.

So the better logic should be something like:

Failed watchdog/helper unit detected
Related service checked
Related service is currently active
Classify as historical boot issue, not current outage

This fits very well with the verifier-first direction of the tool.

Thank you again for testing it this deeply. Feedback like this helps improve the diagnostic flow and makes the tool safer for normal users, because it reduces panic from old boot-time failures while still keeping real problems visible.

1 Like

Since upgrading to ZimaOS 1.6.2 Beta 2, casaos-every-boot.service fails on every boot.

Root cause:

/usr/libexec/every-boot.d/07-targetcli-sync-users

tries to execute:

/usr/libexec/targetcli-sync-users

but the executable is missing:

/usr/libexec/targetcli-sync-users: No such file or directory

As a result, casaos-every-boot.service exits with:

[ERROR] One or more every-boot scripts failed

This appears to be a regression introduced in Beta 2, since the issue did not exist before the update.

2 Likes

Thanks for the feedback, this is exactly the type of real testing we need.

The next version of ZimaBrain CE will focus on improving the failed-service and root-cause side of the verifier.

At the moment, ZimaBrain CE is already doing the main verifier-first job well. It can detect things like failed systemd units, Docker state, published ports, firewall exposure, SMART warnings, storage signals, and update/version evidence.

But the beta testing has shown where we need to improve next.

The main improvements planned for the next version are:

1. Failed systemd unit deep-dive

When ZimaBrain detects a failed unit, it should go further than just saying:

Failed systemd unit detected

It should help identify the exact failed script, command, missing file, or service dependency where the evidence supports it.

Example:

casaos-every-boot.service failed
/usr/libexec/every-boot.d/07-targetcli-sync-users calls missing /usr/libexec/targetcli-sync-users
2. Historical boot failure vs current live problem

Some watchdog units may fail during boot, but the actual service is running fine afterwards.

Example:

tailscaled-watchdog.service  -> failed
tailscaled.service           -> running

zfw-ui-watchdog.service      -> failed
zfw-ui.service               -> running

ZimaBrain should not make users think Tailscale or ZFW is currently broken if the real service is active.

A better result would be:

Historical boot watchdog failure detected.
Monitored service is currently active and healthy.
No current outage detected.
3. Related service correlation

When a failed service is detected, ZimaBrain should check related services where possible.

Examples:

tailscaled-watchdog.service -> check tailscaled.service
zfw-ui-watchdog.service     -> check zfw-ui.service
plex-delay.service          -> check the actual Docker container name
startup.service             -> check the referenced script path

This should help reduce false concern and make the report more useful.

4. Missing file and obsolete script detection

Several real issues from testing involved services pointing to files or scripts that no longer existed.

Examples:

startup.service referenced missing /DATA/startup.sh
07-targetcli-sync-users referenced missing /usr/libexec/targetcli-sync-users

ZimaBrain should detect this pattern more clearly.

5. Docker container name mismatch detection

If a systemd service references an old Docker container name, ZimaBrain should detect that.

Example:

plex-delay.service referenced old container name: plex
Actual container name: big-bear-plex

That is the type of issue ZimaBrain should be able to explain clearly.

6. auditd ownership / permission checks

If auditd fails because /var/log/audit or audit.log has the wrong ownership or permissions, ZimaBrain should identify that from evidence before suggesting any repair.

7. SMART health classification

ZimaBrain should continue not relying only on:

SMART PASSED

It should still show warning counters separately, such as:

CRC errors
NVMe unsafe shutdowns
pending sectors
reallocated sectors
temperature

That way users do not panic, but also do not miss real warning signs.

8. Files / AppData / media issue checks

With the ZimaOS 1.6.2 beta testing, we also need a stronger Files/AppData/media diagnostic layer.

This should help separate:

real backend path problems
mount problems
permission problems
UI-only Files app issues
9. Better routing and evidence classification

If ZimaBrain has real local evidence, it should show the evidence.

If it only has enough information to suggest commands, it should clearly say:

Guidance only
Not verified from current report

It should not make command suggestions look like a verified diagnosis.

10. Cleaner forum-ready output

The forum-ready section should be shorter and clearer:

Confirmed:
- What is actually proven

Not confirmed:
- What still needs logs or UI testing

Next safest step:
- One or two read-only checks

The main direction for the next version is:

Detect failed unit
Read exact evidence
Identify failed script or missing helper
Check related live service
Classify the issue correctly

So instead of only saying “failed service detected”, ZimaBrain should be able to say whether it is:

a current outage
a historical boot failure
an obsolete local config
a missing file
a Docker container name mismatch
a permission issue
or a ZimaOS packaging regression

This feedback has been very useful.

ZimaBrain CE is still beta, but the verifier-first approach is working. The next step is making the root-cause layer deeper and more accurate.

We are open to any suggestions from the community, especially real examples with logs, failed services, Docker issues, storage warnings, or ZimaOS update regressions.

11. ZimaOS regression detection

If a failure first appears immediately after a ZimaOS update and the referenced executable or helper script is missing from the OS image, ZimaBrain should classify it as a possible ZimaOS packaging regression instead of a local configuration problem.

Example:

  • casaos-every-boot.service

  • /usr/libexec/every-boot.d/07-targetcli-sync-users

  • Missing /usr/libexec/targetcli-sync-users

  • First appeared after upgrading to ZimaOS 1.6.2 Beta 2

Suggested classification:

Possible ZimaOS packaging regression

This failure appears to be introduced by the current ZimaOS version rather than by user configuration.

2 Likes

Yes, that is a very good point, and I agree with you.

That distinction is important.

There is a big difference between a user-created service pointing to an old or missing custom file, and an OS-provided service or script calling a helper that is missing from the current ZimaOS image.

The casaos-every-boot.service case is a good example.

If this appeared immediately after the 1.6.2-beta2 update, and the missing helper looks like something that should belong to the OS image, then ZimaBrain should not treat it the same way as a normal local configuration problem.

I will add this to the next version plan as item 11:

ZimaOS regression detection

The classification should be something like:

Possible ZimaOS packaging regression detected.

That would make the diagnosis clearer for users and also more useful for IceWhale, because it separates local user mistakes from possible update regressions.

This is exactly the kind of feedback that helps improve ZimaBrain CE. Thanks again for testing it properly and giving detailed examples.

1 Like

Wow, what a fantastic tool. It will help me further understand ZimaOS and how it works on my home server. What is present and what is gone. Just Pure GOLD !:1st_place_medal:

It didn’t answer any of my questions other than the 2 dockers I know are stopped, and I only run on demand. It couldn’t answer about the 1 Real alert, or the 5 Info.

### ZimaBrain Answer

## ❓ Question asked
### What are the real alerts, container alerts, and info alerts?

#### Verification status
@@VERIFY:VERIFIED@@ ✅ VERIFIED FROM SAME-REPORT EVIDENCE
- This answer is based on evidence found in the current report.
- Active layer: Containers Layer
- Layer file: `app/brain/layers/containers.py`


#### Direct answer / severity
- This is a container state question.
- The answer comes from the Containers Layer using dashboard-parsed Docker state.

Dashboard container count:
- 56/58 containers running.
- 2 container(s) are not running according to the visual dashboard count.

Exited containers parsed from the dashboard layer:
- gphotos2immich (ghcr.io/warreth/gphotos2immich:latest)
- rclone-rclone-1 (rclone/rclone)

#### Next safest step
- Inspect only the affected stopped/exited container logs/status. Do not remove containers in bulk.

#### Forum-ready summary
Based on the verified dashboard evidence, review any stopped or exited containers one by one. Do not remove containers in bulk or run Docker prune commands.

---

## 5. What is the 1 Real  Alert Real Alerts
1
Verified host/storage priority

Time: 2026-07-12 20:16:07

### ZimaBrain Answer

## ❓ Question asked
### What is the 1 Real  Alert Real Alerts
1
Verified host/storage priority

#### Verification status
@@VERIFY:PARTIALLY VERIFIED@@ ⚠️ PARTIALLY VERIFIED
- Some local evidence was found, but one or more key facts still need confirmation.
- Active layer: Dashboard Alerts Layer
- Layer file: `app/brain/layers/dashboard_alerts.py`


#### Direct answer / severity
- This is a dashboard alert question.
- The answer comes from the Dashboard Alerts Layer after the Evidence Normalizer Layer has cleaned it.

Real alerts:
- No real hardware/storage alerts were parsed.

Container/service alerts:
- Dashboard container count: 56/58 running.
- 2 container(s) are not running according to the visual dashboard count.
- No named exited container/service alert was parsed, but the dashboard count still shows a container mismatch.

Info only / unsupported metrics:
- No unsupported/N/A SMART values were parsed.

#### Next safest step
- If the container count is not full, inspect Docker status for the missing stopped/exited container first. Then check any hardware, storage, or service alerts one by one.

#### Forum-ready summary
Based on the verified report, compare the dashboard container count with named container alerts. If the count is not full but no name is parsed, verify Docker status before changing or removing anything.

---

## 6. What is the Info only o Only
5
Unsupported/N/A metrics

Time: 2026-07-12 20:17:28

### ZimaBrain Answer

## ❓ Question asked
### What is the Info only o Only
5
Unsupported/N/A metrics

#### Verification status
@@VERIFY:NOT VERIFIED@@ ❌ NOT VERIFIED
- The question did not match a verified diagnostic layer. This answer is guidance only.
- Active layer: Fallback Guidance Route
- Layer file: `app/brain/answer_builder.py`


#### Direct answer / severity
- This Flask cockpit currently answers dashboard evidence questions reliably.
- Try: show me dashboard alerts, explain sda CRC errors, why is sdd filesystem usage 100%, which containers are exited, which disks are healthy, or is my system protected?

#### Next safest step
- Ask a dashboard-specific question, or extend the Flask verifier with additional ZimaOS layers.

#### Forum-ready summary
Ask a dashboard-specific question so ZimaBrain can route the answer through the correct verifier layer.

---

That is all it told me. Nothing about the 1 Real alert, or the 5 sensors

You are right. ZimaBrain detected the alert counts, but it did not identify the actual 1 Real Alert or explain the 5 Info Only items. It also treated your two on-demand containers as faults even though that is expected in your setup.

That means the current dashboard alert layer is incomplete. It can count alerts, but it is not yet reliably mapping each alert back to the exact disk, sensor, or condition that triggered it.

Thanks for testing it and posting the full output. This is useful because it shows exactly what needs fixing.

Hi everyone,

Thanks again for the testing. The report showing that ZimaBrain counted alerts but did not explain them properly helped expose a real gap.

I pushed an update today with these fixes:

  • Real Alerts now try to map back to the source, object, and condition where the dashboard evidence allows it.
  • Info Only items are now listed instead of only showing the count.
  • Unsupported / N/A metrics are treated as info, not real faults.
  • On-demand or intentionally stopped containers are separated from possible container faults.
  • If ZimaBrain can see an alert count but cannot safely parse the underlying detail, it should now say that clearly instead of acting like there is no alert.
  • Added Files / AppData / media path diagnostics, including odd /media mount names, AppData symlink targets, read-only media, and ZimaOS media mirror path checks.
  • Added auditd ownership / permission checks.
  • Improved answer trust labels so answers now show whether they are verified, partially verified, or guidance only.
  • Added a small UI panel showing the new verifier improvements.

How to update:

cd /DATA/AppData/zimabrain-ce-flask-8601
git pull
docker compose build zimabrain-ce-flask-8601
docker compose up -d zimabrain-ce-flask-8601

Then check health:

curl -fsS http://127.0.0.1:8601/health && echo
docker ps --filter name=zimabrain-ce-flask-8601

Expected health output should include:

"ok": true

After updating, please test the same questions again:

What are the real alerts, container alerts, and info alerts?
What is the 1 Real Alert?
What are the 5 Info Only items?

This still depends on what the ZimaOS dashboard exposes in the local evidence, but the new build should be much clearer and should no longer hide the alert-count mismatch.

I also fixed the global question route. Questions like “what needs attention” now show VERIFIED FROM SAME-REPORT EVIDENCE when ZimaBrain finds real same-report verifier findings, instead of incorrectly falling back to guidance-only wording.