[SOLVED] Regular CPU spikes likely caused by Dockerd Containerd.sock process

Insight & Feedback
1

Hello everyone,

UPDATE :megaphone:

It took me another 2 hours to figure out, that the Issue #2 was caused by the Dockhand application. The second I started the container, the every 5 minutes spikes returned.
Now, I need to look into this container a bit closer, but I am not going to delete this thread, as it might be helpful for those affected by the same behavior.

Original Post

In short:
I have been running ZimaOS for at least 6 months now, currently on v.1.6.2 on Beelink ME Pro with N150 CPU and integrated Intel GPU. Just recently I started noticing regular, ever 1 and 5 minutes, CPU spikes for the very first time. Those spikes were also causing periodical increase of power consumption and rise of temperature.

Issue #1

Now, I have noticed regular CPU spikes using Beszel application at first:

Beszel

To investigate it further, I used btop to help me find the problematic processes. It quickly turned out it was caused by the dockerd that was running the following command:

/usr/bin/dockerd --host=fd:// --add-runtime=nvidia=/usr/bin/nvidia-container-runtime

I do not own or use nVidia GPU, yet I have managed to find and rename the override.conf file under /etc/systemd/system/docker.service.d/ location that was responsible for triggering the CPU spikes.

It contains three lines:

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --host=fd:// --add-runtime=nvidia=/usr/bin/nvidia-container-runtime

By renaming it, for example, to override.conf.bak and running systemctl daemon-reload and systemctl restart docker I have managed to solve this issue.

Issue #2

However, yet another dockerd process was causing similar spikes, this time every exactly 5 minutes.

Thanks to btop I was able to gather some more information and take a couple of screenshots.
I have not managed to solve this one yet.

The process:

dockerd /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Before a spike

CPU Spike

After the spike

Another spike, with Docker stats running

From the last screenshot, it does seem like none of the running containers are at fault. It’s just one screenshot, but I have been observing both, btop and docker stats, running for over an hour, and I am quite certain the issue is not related to any of the installed applications.

—edited typos—

1 Like
2

Interesting, i do not have nvidia card as well, yet nvidia runtime is in multiple processes running, have you noticed if after renaming override.config, was the file regenerated or simply system will not use it anymore, if regenerated is the nvidia runtime still loading.?

There should be an option really, do you have nvidia? No, ok i shall not load this runtime then :slight_smile:

3

Just checked and the renamed override.conf.bak file is still there. No new file has been created since. The dockhand app is currently disabled.

On another note, once I have removed my server (the environment) from the Dockhand app, the spikes were gone. Not exactly surprising, but still. Currently switched to Arcane, but will continue investigating the Dockhand further, as I prefer it over any other tool.

4

Yeah, i do like using dockhand, gives me all the info and capabilities that i need, i have played a bit with settings, limit system jobs etc in dockhand, have a look in yours, if you had it set to poll every 300s that would explain the spike,

It is interesting with that nvidia though.

We would need someone from zima team to maybe explain if we need it and what, if any issues we can have if disabled, removed

5

Turns out, I’ve been using the default Stream setting, which, as the tooltip says, “Continuous event stream from Docker, instant notifications, higher CPU usage”.
I have changed it to Poll every 300s, which is the lowest I can choose.
Not much has changed, really. The every-5-seconds spikes are clearly present, no matter which collection mode I pick. Also, the Metric collection interval can go as low as 120s.

It is bizarre to me, that Arcane, which serves the exact same purpose, not only doesn’t suffer from similar issues, but it also uses less resources. Oh well.