Warning: ZimaOS 1.8 Photos stores previews in RAM (/tmp). With a large library it exhausted RAM + swap and took down my whole home network's DNS

ZimaOS 1.8 Photos fills RAM via /tmp with large libraries – workaround (previews on disk + resource limits)

Setup: ZimaBoard 2 (16 GB RAM), ZimaOS 1.8.0, ~100,000 photos on a NAS share, Pi-hole and Nextcloud AIO on the same box.

Why

Photos writes its previews to /tmp/zimaos-mds-gallery-preview. On ZimaOS /tmp is a tmpfs, so it lives in RAM (50 % of RAM). The folder grows by ~80 KB per photo and is never trimmed. Within three days it held 6 GB / 75,000 files: RAM and swap were full, the CPU was busy swapping, and everything on the box stalled. My Pi-hole on the same box answered DNS in 1–12 s or timed out, so the whole house had “slow internet”. Photos is not a Docker app, so it doesn’t show up in docker stats.

Check your own box:

df -h /tmp

sudo du -sh /tmp/zimaos-mds-gallery-preview

free -h

Quick relief

sudo systemctl stop zimaos-photos-ai-worker.service zimaos-photos.service

sudo find /tmp/zimaos-mds-gallery-preview -mindepth 1 -delete

Permanent workaround

This keeps Photos running, but puts the previews on disk and limits its CPU and RAM. Replace /media/YourDisk with a disk that has room (100k previews ≈ 8 GB). Run as root:

CACHE=/media/YourDisk/AppData/photos-preview-cache

TARGET=/tmp/zimaos-mds-gallery-preview

# Wait for the disk (systemd retries), then bind-mount the preview folder onto it

BIND="grep -q ' /media/YourDisk ' /proc/mounts || exit 1; mkdir -p $TARGET $CACHE; grep -q ' $TARGET ' /proc/mounts || mount --bind $CACHE $TARGET"

mkdir -p "$CACHE"

mkdir -p /etc/systemd/system/zimaos-photos.service.d

cat > /etc/systemd/system/zimaos-photos.service.d/limits.conf <<EOF

[Service]

ExecStartPre=+/bin/sh -c "$BIND"

Nice=19

CPUWeight=10

MemoryHigh=3584M

MemoryMax=5G

EOF

mkdir -p /etc/systemd/system/zimaos-photos-ai-worker.service.d

cat > /etc/systemd/system/zimaos-photos-ai-worker.service.d/limits.conf <<EOF

[Service]

ExecStartPre=+/bin/sh -c "$BIND"

Nice=19

EOF

mkdir -p /etc/systemd/system/zimaos-photos.slice.d

cat > /etc/systemd/system/zimaos-photos.slice.d/limits.conf <<EOF

[Slice]

CPUWeight=10

CPUQuota=150%

MemoryHigh=2560M

MemoryMax=3G

EOF

systemctl daemon-reload

systemctl restart zimaos-photos.service

Why these settings:

  • Bind mount: previews go to disk instead of RAM. Both services set it up, because the AI worker starts first.
  • No CPU cap on zimaos-photos.service: it also runs the Manticore search database. A 1-core cap throttled it so hard that the gallery showed “No matching photos”. Nice=19 + CPUWeight=10 is enough: it only gets CPU nobody else needs.
  • Memory 3.5 GB soft / 5 GB hard: with MemoryHigh=2G the cgroup was throttled 29,000+ times and the rebuild crawled.
  • Cap on the parent zimaos-photos.slice: the AI and media slices are its children. Photos overrides limits on zimaos-photos-media.slice at startup, but leaves the parent alone.

Verify:

grep gallery-preview /proc/mounts     # must show your disk, not tmpfs

grep high /sys/fs/cgroup/system.slice/zimaos-photos.service/memory.events /sys/fs/cgroup/zimaos.slice/zimaos-photos.slice/memory.events # should barely grow

To undo: delete the three limits.conf files, then systemctl daemon-reload and restart Photos.

If Photos breaks after the memory crash

“This feature is temporarily unavailable”: journalctl -u zimaos-photos.service showed Manticore failing on a missing binlog file (binlog.0000: No such file or directory). Moving the binlog files aside fixed it; the data itself is kept:

sudo systemctl stop zimaos-photos.service

sudo mkdir -p /media/ZimaOS-HD/AppData/photos/manticore/binlog-defect

sudo mv /media/ZimaOS-HD/AppData/photos/manticore/binlog/binlog.* /media/ZimaOS-HD/AppData/photos/manticore/binlog-defect/

sudo systemctl start zimaos-photos.service

Empty gallery, and a “Welcome to ZimaOS Photos / Start” screen in a private window: Photos had lost its setup record (the table photo_schema_meta was empty). First make a backup of AppData/photos/manticore (stop Photos, then cp -a --reflink=auto), then click Start. It reused the existing index (“Importing · 97,488”) and started rebuilding.

Findings so far

  • RAM in use went from 13 GB + 5.1 GB full swap to ~6 GB. Pi-hole answers in ~3 ms again.
  • Previews now land on disk (8,000+ files and growing), and /tmp no longer fills up.
  • After the recovery, Photos re-checks everything: 97,488 photo_ai_repair jobs queued, plus a gallery rebuild. The Month view refills gradually; the Year view and the map (14,966 of 21,687 geotagged photos) are still catching up. Expect this to take a while.

To IceWhale

Please store previews on persistent storage instead of /tmp, cap the cache, and add pause/throttle controls for indexing. A feature that can take down a whole NAS within days should be safe by default, or clearly marked as beta, before it ships to everyone.

Update: two more problems found, and the corrected settings

1. The AI worker needs more memory than I gave it

The AI slice (zimaos-photos.slice/zimaos-photos-ai.slice) had MemoryHigh=1536M. Overnight it was throttled 2.48 million times: its working set is ~1.3–1.5 GB, and it ended up with 1.8 GB in swap. Inference became so slow that the worker crashed:

photos ai-worker failed: AI worker recovery required: ncnn native inference did not drain within 5s after request cancellation

Use at least 2.5 GB for the AI slice, and raise the parent slice so it fits:

mkdir -p /etc/systemd/system/zimaos-photos-ai.slice.d

cat > /etc/systemd/system/zimaos-photos-ai.slice.d/limits.conf <<EOF

[Slice]

CPUWeight=10

CPUQuota=100%

MemoryHigh=2560M

MemoryMax=3G

EOF

mkdir -p /etc/systemd/system/zimaos-photos.slice.d

cat > /etc/systemd/system/zimaos-photos.slice.d/limits.conf <<EOF

[Slice]

CPUWeight=10

CPUQuota=150%

MemoryHigh=3584M

MemoryMax=4G

EOF

systemctl daemon-reload

Check with grep high /sys/fs/cgroup/zimaos.slice/zimaos-photos.slice/zimaos-photos-ai.slice/memory.events.local: the counter should stop growing.

2. Bug: an AI worker restart silently stops all of Photos

This one is not caused by my limits and can hit anyone. zimaos-photos-ai-worker.service has RuntimeDirectory=zimaos-photos, so systemd owns /run/zimaos-photos on behalf of the worker. The main service puts its Manticore socket in that same directory (/run/zimaos-photos/manticore/bulk/http.sock). When the worker restarted after the crash above, systemd wiped the directory, and with it the socket. The main service kept running, but could no longer write anything:

  • no new thumbnails for 12 hours
  • no error in the UI, and nothing obvious in the logs

Check: ls /run/zimaos-photos/. If manticore is missing, Photos is stalled. Fix: keep the directory on worker restarts, then restart Photos once:

mkdir -p /etc/systemd/system/zimaos-photos-ai-worker.service.d

cat >> /etc/systemd/system/zimaos-photos-ai-worker.service.d/limits.conf <<EOF

[Service]

RuntimeDirectoryPreserve=yes

EOF

systemctl daemon-reload

systemctl restart zimaos-photos.service

3. Every restart shows “Welcome to ZimaOS Photos / Start” again

After every restart of zimaos-photos.service, a fresh browser session shows the welcome screen with a Start button, and the gallery and map stay empty or show “Loading” for a while. Clicking Start re-initialises the AI runtime ([bootstrap] NCNN runtime artifacts ready in the journal). Nothing is lost: the index still had all 97,509 items. There are no errors in the journal, and the photo_schema_meta table stays empty. IceWhale: is this expected behaviour?

Current status

  • RAM is stable, /tmp no longer fills up, previews land on disk, and Pi-hole on the same box answers in a few ms.
  • The gallery rebuild (gallery_group_rebuild) completed. The AI pipeline is running (OCR stage) and making progress.
  • Not yet clear: photo_ai_repair has had 97,488 jobs pending since the recovery, and photo_ai_failure grew to 4,944 during the night of the crash.

I’ll post again if I run into anything else.

Thank you again for documenting this issue in detail, sharing your workaround, and following up with additional findings. We understand that this also affected Pi-hole on the same device and disrupted DNS for your home network.

We haven’t been able to reproduce the preview cache filling up /tmp on our own devices so far. Earlier builds did store previews in /tmp, but we expected this to have been fixed in the official ZimaOS 1.8.0 release. To help us check whether an older, manually installed Photos build might still be loading, could you confirm the following?

  • Does /var/lib/extensions/icewhale_photos.raw exist on your device?
  • Have you previously downloaded or installed an older or pre-release Photos build manually?
  • What format or file extensions did the files in /tmp/zimaos-mds-gallery-preview have? A few example filenames would also be helpful, if available.

As part of the improvements we’re planning, we intend to store these files under /media/ZimaOS-HD/AppData/photos/.cache instead of /tmp, so they no longer consume the RAM used by /tmp.

We’ve also noted your suggestion to let users choose where the database is stored, and we’ll pass it along to our product manager for consideration.

We appreciate the time you’ve spent investigating and sharing your findings. Once an updated build is ready, would you be willing to help us test it?

Thanks for the quick and detailed reply! Answers to your questions:

1. Does /var/lib/extensions/icewhale_photos.raw exist? No. /var/lib/extensions/ only contains doconverter.raw, icewhale_files.raw, icewhale_peerdrop.raw and zimaos_zvm.raw (the last three are symlinks to /usr/lib/zimaos/modules/). systemd-sysext status doesn’t list a Photos extension either, so Photos runs from the base image (/usr/bin/zimaos-photos).

2. Older or pre-release Photos build installed manually? No. I never installed Photos manually, and I never used a beta or pre-release build. Photos arrived with a one-time upgrade to ZimaOS 1.8.0. I added my photo source on 16 October, right after the upgrade.

Version details:

  • /etc/os-release: PRETTY_NAME="ZimaOS v1.8.0"
  • /usr/bin/zimaos-photos: 57,729,488 bytes, dated 2026-04-21 23:05

That binary date is about six months before the 1.8.0 release. Could the release image contain an older Photos build than intended, one from before the /tmp fix? I can send a checksum if that helps.

3. Format and file names in /tmp/zimaos-mds-gallery-preview I deleted the original 75,069 files during the recovery. However, the build that is running now still writes to exactly that path: I bind-mounted /tmp/zimaos-mds-gallery-preview onto an SSD folder, and since 19 October 12:06 it has received 8,117 new files (459 MB). Without the bind mount, these would be in RAM again.

  • All files sit in one flat folder, with no subfolders.
  • Names follow the pattern image-<asset id>-<16 hex chars>.<ext>, for example:
    • image-369395397455515648-8218beb03b3d764c.png (159 KB)
    • image-369395398357291008-7505edc6cf359754.png (173 KB)
  • 8,075 files are .jpg (about 57 KB on average) and 42 are .png (about 216 KB on average).

So as far as I can tell, the official 1.8.0 build still uses /tmp for these previews.

Moving them to /media/ZimaOS-HD/AppData/photos/.cache sounds good. Please also cap or evict that cache: .cache already holds about 117,000 photo-thumb-v8-card-*.bin thumbnails for my library.

Two other issues, not related to /tmp, from my follow-up post. They are probably the most important ones:

  1. An AI worker restart silently stops Photos. zimaos-photos-ai-worker.service owns RuntimeDirectory=zimaos-photos, and the main service keeps its Manticore socket there too (/run/zimaos-photos/manticore/bulk/http.sock). When the worker crashed and restarted (ncnn native inference did not drain within 5s), systemd wiped /run/zimaos-photos. The main service kept running but could no longer write: no new thumbnails for 12 hours, and no error in the UI. Suggested fix: give the main service its own RuntimeDirectory, or set RuntimeDirectoryPreserve=yes on the worker.
  2. “Welcome to ZimaOS Photos / Start” after every restart of zimaos-photos.service. The gallery and map stay empty until someone clicks Start in a fresh browser session. After that, gallery_group_rebuild runs again and 97,488 photo_ai_repair jobs are queued. The index itself was intact the whole time.

A third, smaller one: after the memory crash, Manticore refused to start because of a missing binlog file (binlog.0000: No such file or directory). The UI only showed “This feature is temporarily unavailable”. Automatic recovery, for example with --replay-flags=ignore-open-errors, would help.

Some feature suggestions from my testing:

  • “Unknown date” instead of 1970 or 2038. 67 of my photos came from a camera whose clock had been reset. Photos put them in 1970 or in 2038, and the 2038 ones always stayed at the top of the timeline, above my newest photos. Plenty of people will have photos like that. A separate “Date unknown” section would help, ideally with a way to correct the date (or shift it for a whole selection) from the UI.
  • Choose the cover photo of the Collection tiles (Map, Favorites, Memory), as mentioned in my follow-up.
  • Share albums through a link, with an optional password (and ideally an expiry date). An album you can’t share is only half an album.
  • More actions in the photo view. Next to Info and Favorite, I’m missing:
    • Delete
    • Share, as a link or through the device’s share menu
    • Set as cover
  • Fully disable or uninstall built-in apps such as Photos. It runs services and AI work in the background even if you never use it.

For context: I’m mostly doing this to learn. Photos isn’t critical for me, because Synology Photos does the job for my library today. I don’t see a compelling reason to switch yet, and I may well remove Photos after my testing. That is exactly why the option to switch it off or uninstall it completely matters to me.

That said, I’m happy to test an updated build while I’m still evaluating. Just let me know when it’s ready.

Wow, it almost feels like you’re part of our team—you’ve picked up on so many of the features we’re planning and the issues we’re working through! Thank you for spending so much time testing Photos and documenting your findings so thoroughly.

The version details and newly generated preview files also clarify the situation: based on what you’ve provided, this doesn’t appear to be caused by a manually installed older Photos build. We previously expected the release build to have already addressed this cache path, so we need to revisit that assumption. Thank you for helping us catch this.

The test build we’re preparing includes fixes for some of the issues you raised, and we’ll continue following up on the others:

  1. Preview files in /tmp
    We plan to store these files under /media/ZimaOS-HD/AppData/photos/.cache. Any files left behind in /tmp will be cleared automatically when the system reboots. Your suggestion to limit the cache size and add an eviction policy is valuable too, and we’ll evaluate it alongside this change.

  2. An AI worker restart silently stops Photos from working
    Thank you for tracing this to the shared runtime directory and the removal of the Manticore socket. We’ll address this, although the fix may not make it into the upcoming test build.

  3. “Welcome to ZimaOS Photos / Start” appearing after every restart
    We previously fixed an issue with this behavior. The expected behavior is that the welcome screen should not reappear when a configured photo source contains data. What you’re seeing doesn’t match that expectation, so we’ll investigate further using the restart and recovery details you’ve shared.

  4. A missing binlog preventing the service from starting
    We’ve identified this issue as well. The recovery logic after an unexpected power loss was too strict: a missing binlog could prevent the entire service from starting normally. We’ve fixed this for the next version.

Regarding your feature suggestions:

  1. Unknown dates and date editing
    The team is already planning improvements in this area, including handling unknown dates and allowing users to correct photo dates.

  2. Custom cover photos
    We’ve passed along the request to choose cover photos for Collection tiles such as Map, Favorites, and Memory. We understand that “Set as cover” in the photo viewer would be a shortcut for setting the current photo as one of those covers. If you also have another use in mind, please let us know.

  3. Sharing
    This is also being planned, but it will take some time. The details you mentioned—shareable links, optional passwords, and expiry dates—are helpful input.

  4. Deleting from the photo viewer
    The photo viewer already has a delete action. If it isn’t visible in your current version, we’d like to understand why.

  5. Disabling Photos
    We’re planning to support this too. The Photos service and AI worker are designed to reduce their background activity when idle, with relatively low resource usage in that state. We also understand why you want an explicit way to turn the feature off, and we plan to add an option in Settings to stop the entire Photos service.

Thank you as well for being willing to test another build while you’re still evaluating Photos. Once it’s ready, we’ll post the download details and changes here so you can verify them.

Thank you for the kind words and for such a thorough answer! It’s great to see that most of this is already on your list. A few follow-ups and some new findings since my last message:

Refreshing a source

  • The refresh button is hard to find. It’s a small, faint circle at the end of the source line. A casual user won’t notice it, so a visible label or tooltip (“Check for changes”) would help. The progress display it starts (“Importing · 16,900 / 97,515”) is very nice, by the way.
  • Photos doesn’t seem to check for changes by itself. “Last checked” stayed at 20 Oct 10:32 for about 23 hours, and corrected files on my NAS were only picked up after I clicked refresh. My source is a network share (CIFS), where file-change notifications don’t work. A periodic re-check would help, for example every few hours or on a schedule the user can set.

The ✕ next to a source

I haven’t dared to test it: what exactly happens when you click it? Does Photos only stop using the path, or does it also drop the index, AI results, albums and favorites for those photos? Either way, a confirmation dialog that explains the consequences would be welcome. It should also say clearly that the original files are not deleted, if that’s the case.

Files Photos can’t process

Since the recovery, the only AI failures are decode image: unavailable: 1,919 in six hours, and 5,708 in total. The thumbnail prefill log shows the same kind of thing:

Error Count (12 h)
image: unknown format 1,599
unexpected EOF 343
tiff: invalid format: tags are not sorted in ascending order 343
exit status 183 341
context deadline exceeded 136

Two suggestions:

  1. A “problem files” view that lists the files that couldn’t be processed, with the reason, and offers retry, ignore/hide or move to a quarantine folder. Some of these errors (unexpected EOF, context deadline exceeded) may well be read timeouts on a network share rather than truly broken files, so a retry option matters. Deleting should stay the user’s decision.
  2. RAW support, or a way to exclude it. My library contains 8,455 Nikon .NEF files, mostly in raw subfolders next to their JPEGs. As far as I can tell, these account for most of the decode failures. The TIFF error is typical for NEF, which is TIFF-based, and a more lenient decoder (libvips/libtiff) might accept it. Is RAW support planned? Showing the embedded JPEG preview would already be enough for browsing. Otherwise, an option to exclude extensions or subfolders (such as raw/) from a source would avoid thousands of failed jobs.

Dates: what I found after fixing them myself

101 photos had wrong dates, from two HP PhotoSmart cameras:

  • A clock reset after a battery change: EXIF 1970:01:01 00:00:33, 00:09:50 and so on, with a file date of 19-01-2038 04:14:07 (the 32-bit maximum). Photos rejected the EXIF date and fell back to the file date, so these photos always sat at the top of the timeline.
  • A camera whose clock was never set, with dates in January 2003.
  • Inconsistent handling: Photos rejected 1970:01:01 … as invalid, but accepted 1970:01:15 21:54:08 and put that photo in 1970.

I corrected them by shifting the dates per camera, which keeps the order and the spacing between photos. That’s exactly what a “shift date/time for selection” feature would need. Treating any EXIF date before, say, 1995, and any file date in the future, as “unknown date” would catch most of these cases automatically.

Your questions

  • Delete in the viewer: I checked again, and there is no delete action in my version (ZimaOS 1.8.0, web UI in a desktop browser).

    • At the top right, the photo viewer shows Info, Favorite, a download icon, Fullscreen and Close.
    • The bottom bar has previous/next, zoom in/out, rotate left/right and reset.
    • The Info panel shows details only, with no actions.
    • There is no “⋯” menu.
    • Right-clicking gives the standard browser menu.

    Could it be hidden because my source is a network share (CIFS), or because of my user’s permissions?

  • Set as cover: yes, mainly as a shortcut for the Collection tiles. It would also be useful for albums once those get covers.

  • The welcome screen: when it reappeared, the index held all 97,509 items and the source path was unchanged, so it should have been considered “configured with data”.

Current status

With the workarounds in place, Photos has been stable since 20 Oct 10:27:

  • no AI worker restarts;
  • the Manticore socket is intact;
  • no memory throttling. The AI slice needs about 1.5 GB; 2.5 GB is fine.

The AI pipeline is progressing steadily.

Thanks again. I’m looking forward to the test build!

adding …A short update while you work on the test build: the AI processing of my library has finished.

How long it took

  • About 12.5 days for ~97,500 photos on the N150: from 16 Oct, when I added the source, to 28 Oct 22:53. That includes the crash and the days when Photos was throttled.
  • The stages ran one after another, one photo at a time (about one CPU core):
Stage Photos per hour
ocr_detect 300–750
ocr_recognize 1,000–1,200
object_detect ~1,235
  • Object detection was hardly faster than OCR. That suggests most of the time goes into opening and decoding each full-size photo from the network share, not into the models themselves.
  • The UI doesn’t show which stages exist, which one is running, or how long it will take. For CPU-only boxes, a line in the docs (“expect about two weeks per 100,000 photos”) and an ETA per stage would set the right expectations.

Final failure counts

Error Count Likely cause
decode image: unavailable 25,506 RAW files (.NEF)
rapidocr ncnn runtime unavailable, ncnn_model_class_not_loaded, ncnn worker unavailable … ~2,590 The crash on 19–20 Oct
exit status 254, exit status 183, no picture 112 Unknown, possibly videos
image: unknown format 2 Two JPEGs that contain only zero bytes
Other messages 201 Not checked
Total (photo_ai_failure, terminal) 28,413

Two observations:

  1. Each RAW file seems to fail three times. 25,506 is exactly 3 × 8,502, and my library has 8,455 NEF files plus a few dozen other files Photos can’t decode. So every such file apparently fails once in each stage. Remembering that a file can’t be decoded, or skipping unsupported formats up front, would save that work.
  2. Photos found two broken files, but only the database knows. Two JPEGs on my NAS contain nothing but zero bytes (3.3 MB and 2.6 MB), probably damaged during a copy years ago. Photos correctly reports image: unknown format, but I only found out by querying the job table. This is exactly what the “problem files” view from my previous message would show.

Since the AI finished: an endless thumbnail loop

Even with the AI done, searchd keeps peaking at about 70 % of a core. Two things are going on:

  • The same 24 thumbnails are retried forever. In 24 hours, gallery-thumbnail-prefill logged 4,320 failures for only 24 files, each exactly 180 times (about every 8 minutes, day and night). Every attempt starts vips or ffmpeg and writes to the database. The 24 files:
    • 9 are genuinely broken: empty, truncated, zero-filled or overwritten;
    • 1 is misnamed: a Corel Photo-Paint file with a .gif extension;
    • 14 are valid files Photos can’t handle: 6 QuickTime .MOV clips from a Panasonic camera (reported as image: unknown format, so they seem to go through the image decoder), 3 MP4s (exit status 183), a 4.5 GB AVI (context deadline exceeded), 3 TIFFs (tags are not sorted in ascending order) and a PSD.
  • Photos stops and starts its Manticore about every 10 minutes. The service itself has been running since 20 Oct without a restart, but since the AI finished (28 Oct 22:53), searchd was started 70 times between 23:00 and 09:25, each time after a [manticore] shutdown flush tables=…. If this is an idle mode, the periodic work defeats it: each start reloads the tables (about 1.9 GB on disk), and that’s where the CPU peaks come from. The hourly gallery-cache refresh, which takes minutes, adds to it.

Suggestion: remember failed thumbnails. Retry with an increasing delay, or only when the file’s size or modification time changes, and list them in the problem-files view.

Three questions

  • Will updates keep the AI results? Twelve and a half days of OCR and object detection is a lot to lose. I noticed versioned pipeline keys in processing_kv (cleanroom-v17, gallery-fact-v3, photo-search-ir-v3) and an empty photo_schema_meta since the crash. Could a new build decide to reprocess everything? Ideally only stages whose model or version actually changed are re-run, and the release notes say so (“this update re-runs OCR for all photos”). An empty photo_schema_meta should not make the test build throw away the existing index either.
  • The photo_ai_repair queue on a CPU-only box. The job list shows 97,509 pending photo_ai_repair jobs. According to the journal these are VLM jobs (VLM repair audit accepted=0 deduped=97488), and the VLM is skipped here (vlm preload skipped: image backend yolo). Will these jobs ever run or be cleared? Right now the number makes it look as if Photos is far behind, while all the CPU work is done. Hiding VLM-only work in CPU mode, or labelling it as such, would make the status clearer.
  • GPU support. The journal mentions no GPU, Vulkan or CUDA, so the N150’s integrated graphics sits idle. Is Intel iGPU acceleration planned (for example through ncnn’s Vulkan backend)? And if I added an NVIDIA card to the ZimaBoard 2’s PCIe slot (with external power), would Photos use it, only for the VLM or also for OCR and object detection? Is there a minimum card or amount of VRAM, and a list of tested cards and power setups?

Current status

Apart from the loop above, Photos is idle now. With the workarounds in place, it has been stable since 20 Oct. Before installing the test build, I’ll back up the Photos database, and I’ll report back how it goes.

Thank you for following up and for providing such detailed measurements. It’s good to hear that the AI processing has finally finished. After spending 12.5 days on it, your concerns about continued background activity and preserving the results through an update are completely understandable.

We’ve completed an initial investigation based on your findings and are working through the fixes and validation. Here’s where things stand:

  1. Repeated thumbnail attempts and background resource usage
    We’ve confirmed that persistent thumbnail failures aren’t being remembered properly, causing the same files to be processed repeatedly. This is a priority for us to fix. We’re also investigating the repeated database wake-ups, with the goal of keeping Photos’ resource usage low once processing is complete.

  2. Format compatibility and failed-file handling
    There have been improvements for some formats, but we still need to validate the cases you reported before considering them resolved. We’re also improving failure handling to avoid repeatedly attempting to decode the same unsupported file across different stages. The “problem files” view is also on our list, so users can see what failed, understand why, and decide what to do.

  3. Pending tasks in CPU mode
    VLM tasks that don’t apply to the current mode won’t block normal library processing. However, the leftover pending status is confusing, and we’ll continue improving how those tasks are managed and displayed.

  4. Preserving existing results during updates
    Our goal is to reuse existing, valid index data and AI results and avoid unnecessary full reprocessing. However, we still need to validate the upgrade against a database that has been through the recovery you described, so we can’t yet promise that no additional processing will be needed. If an update requires any stages to run again, we’ll make that clear in the release notes. Your plan to back up the database before upgrading is helpful.

  5. Progress reporting and GPU support
    There have been improvements to stage and progress reporting. Remaining-time estimates and performance with network shares still need more work. OCR and object detection currently run on the CPU, and we don’t yet have a verified Intel iGPU or NVIDIA acceleration compatibility list to share. We therefore wouldn’t recommend purchasing a GPU specifically to accelerate Photos at this point.

Sorry for the wait. We’ll complete the key fixes and upgrade validation before posting the test build here, together with a clear description of what it addresses, so you can decide whether to continue testing. Thank you again for putting so much time into helping us understand these issues.

Thanks for the detailed answer, and for the advice on the GPU: I’ll hold off on buying one.

Two new findings that may help with point 1, and one correction to my last post: the CPU load I described there came mostly from somewhere else.

searchd stays busy with nothing to do

Today searchd ran at 210–240 % CPU for hours (load average 6.6 on the 4-core N150). SHOW THREADS showed all four worker threads busy with the same query from Photos itself, 4.2 million jobs between 09:25 and 14:17 (about 240 per second):

SELECT id,natural_key,artifact_json FROM photo_asset_artifacts WHERE id IN (…)

All that time, I had Photos open in a browser tab, on the default page it opens with. I wasn’t using it. With the tab closed, searchd still ran at 120–140 % CPU (measured twice, at 14:56 and 15:15), with 40–90 searches per second and a load average of 3.5–4.2. Nothing in the journal explains it, apart from periodic RAM chunk saves. So the open page adds roughly one core, but something in the background keeps more than one core busy, with nothing left to process.

With the tab closed, I sampled SHOW THREADS 40 times over 30 seconds. 16 of them caught the same query (shown as api-search … index=photo_asset_artifacts); the only other thing was a plain select, twice. So it’s the same query in both cases. The open page just makes it run more often.

The database restarts every ~10 minutes from my last post stopped at 09:25, and they haven’t come back since, with or without the tab (checked at 14:56). That may be useful for your look into the wake-ups.

The broken files

I moved the 10 broken and misnamed files out of the library and refreshed the source.

  • The rescan took over an hour. The UI said “The request took too long, please try again”, while “Importing” kept going in the background. A timeout message while the work continues is confusing.
  • Afterwards, Photos removed the 10 files from its database and stopped retrying them.
  • The 14 valid files it can’t handle (MOV, MP4, AVI, TIFF, PSD) are still retried, about every 6 minutes each.

For point 4: if it helps your upgrade validation, I’m happy to run checks on my recovered database before you post the test build.

Since Photos had nothing left to do, I’ve stopped it for now and made a backup of the database. I’ll start it again for the test build.

Sorry for the wait. We’ve prepared a new test build that fixes most of the issues you reported, along with some additional issues we found during testing. We’ve also added the Photos toggles you requested and optimized memory and CPU usage.

Please run the following command as root. The device will reboot automatically once the installation succeeds:

rauc install https://casaos.oss-cn-shanghai.aliyuncs.com/IceWhaleTech/zimaos-rauc/extenal-package/zimaos-x86_64-1.8.1-alpha1.raucb && reboot

Please give it another try after updating. If any issues remain or you notice anything unexpected, please let us know. Thanks for your patience and help!

Thanks for the new build, and for adding the Photos toggles. That’s great news.

I’m preparing an exhibition and have several photo assignments in the coming weeks, so I can’t test it properly until early December. I’d rather do it then and report back carefully than rush it now. Photos has been stopped on my box since the end of October, and I backed up its database while it was stopped.

A few questions, so I can prepare:

  1. Release notes. Is there a list of what 1.8.1-alpha1 changes? Then I know what to test, and I can check each issue I reported.
  2. My workarounds. I limited Photos with systemd drop-ins (memory and CPU limits on the service and its slices, and RuntimeDirectoryPreserve=yes on the AI worker), and I bind-mounted the preview folder in /tmp to the SSD. Should I remove all of these before installing, so I’m testing your build as shipped?
  3. The existing index. Will alpha1 reuse my AI results, or re-run some stages? You mentioned validating the upgrade against a recovered database like mine.
  4. Rolling back. If something goes wrong, can I return to 1.8.0, and would it still read the Photos database after alpha1 has used it? If not, I’ll restore my backup first.

I’ll report back in December. Thanks again for following up so thoroughly.

Christian

Hi Christian,
Thanks for the update, and best wishes for your exhibition and upcoming photo assignments! Testing in early December is absolutely fine—there’s no rush. Thank you as well for stopping Photos and backing up the database beforehand.
Here are more specific details about the changes in alpha1 and how to prepare for testing.
This update addresses the following issues you reported:

  1. Preview caching and recovery
    We’ve fixed the memory-usage issue caused by storing previews in /tmp and improved cache management. We’ve also fixed the issue where an AI worker restart could leave Photos unable to continue processing, and the startup failure caused by a missing binlog after an abnormal shutdown.
  2. Thumbnail generation and format compatibility
    We’ve made fixes and improvements to the relevant processing paths. When you have time to test, could you check whether the files that previously failed—particularly the MOV, MP4, AVI, TIFF, PSD, and NEF files—now generate thumbnails successfully? If any still fail, would you be comfortable sharing a few sample files with us for testing? That would help us reproduce and address the remaining cases.
  3. AI processing efficiency and CPU usage
    We’ve optimized the AI processing pipeline and CPU usage, including the sustained high CPU activity you reported. It would be helpful to hear how processing speed and resource usage compare, including after processing has finished, with the Photos browser tab both open and closed. This version also includes the Photos toggles, so you can turn it off when it isn’t needed and avoid unnecessary resource usage.
  4. Source updates
    We’ve addressed the issue where changes to a source weren’t being picked up promptly. Please also check whether changes to files on your CIFS source are now detected and reflected correctly.
  5. Visibility of failed files
    You can now see which files failed processing and the corresponding failure reasons, without having to query the database.
  6. Incorrect dates
    Our product team is still following up on the date-handling issues you reported.
    Regarding your workarounds, we suggest backing up your custom Photos configuration, then removing the CPU/memory overrides and the RuntimeDirectoryPreserve=yes override, and undoing the preview-directory bind mount before testing. This will let you evaluate the new build with its default configuration and check whether our fixes resolve the original problems.
    We still need to confirm the details of reusing your existing index and AI results, and database compatibility when rolling back to 1.8.0, before giving you a definitive answer on those two points. In the meantime, please keep your current pre-upgrade database backup.
    Thank you again for putting so much time into testing Photos. Whenever you have time in December, we’d appreciate hearing how these improvements work with your library.

Hi, thank you for the detailed list and the kind wishes, and for all the work you’ve put into these fixes.

I plan to test in the first week of December. To make sure I start well prepared, three quick questions:

  1. Which build? Will 1.8.1-alpha1 still be the build you’d like me to test at that time, or could there be a newer one by then? If so, could you share the install command when it’s ready?
  2. The two open points. Do you expect to have confirmed them by then: whether the update will reuse my existing index and AI results, and whether 1.8.0 can still read the Photos database after a rollback?
  3. Anything else? Is there anything else I should know or do before installing, apart from the steps in your last message?

Thanks again, and I’ll report back in December.

Hi Christian,

Thanks for checking these details in advance. Testing in the first week of December is absolutely fine.

  1. Which build to test
    We’re currently preparing 1.8.1-alpha2. When a new test build is ready, we’ll post it here along with the installation command. We’ll also confirm which build we recommend when you’re ready to start testing.

  2. Existing results and rollback compatibility
    Both questions come down to database compatibility. This update does not change the Photos database schema, so your existing index and AI results can be reused, and the database remains compatible with 1.8.0 if you roll back. The update itself does not require your entire library to be processed again. Please note that removing a photo source from Photos also removes its associated index and processing results, so keep your existing sources configured to preserve those results.

  3. Anything else to prepare
    There’s nothing else that needs special attention—just follow the update steps we provide. Our previous advice to keep a database backup and back up and remove your temporary workarounds still applies.

Thanks again for your help. We look forward to your feedback in December, and best wishes for your exhibition and photo assignments!