Optimizing cAdvisor for Lower CPU Usage

📆 · ⏳ 3 min read · ·

Introduction

I have been running couple of services in my homelab via docker containers. For monitoring purposes I use Grafana and Prometheus. I have been using cAdvisor ↗️ to monitor my docker containers.

Ever since I got the orange pi server, I started to see a noticable difference in the CPU usage being reported by cAdvisor container, it was always hovering around 15-20% CPU usage (which is way higher than any of the containers that I was running in idle state).

I started to investigate this and found a way to reduce the CPU usage of cAdvisor. Let’s see how I did it.

Solution

I won’t go into the details of what is cAdvisor since I believe if you are here you already know what it is. So let’s jump into the solution.

Here’s what my (and probably your’s as well) docker-compose file looked like:

services:
cadvisor:
image: "gcr.io/cadvisor/cadvisor-arm64:v0.49.1"
container_name: "cadvisor"
ports:
- "8080:8080"
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
restart: unless-stopped

The above configuration is the default configuration that is provided in the cAdvisor documentation ↗️. The problem is with the default configuration and how cAdvisor collects the metrics.

Here are the configurations that I changed from the default configuration:

  • -housekeeping_interval from 1s to 10s
  • -docker_only from false to true

Here’s how my updated docker-compose file looks like:

services:
cadvisor:
image: 'gcr.io/cadvisor/cadvisor-arm64:v0.49.1'
container_name: 'cadvisor'
command:
- '-housekeeping_interval=10s'
- '-docker_only=true'
ports:
- '8080:8080'
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
restart: unless-stopped

The major improvement came from the -docker_only=true flag. The -docker_only flag tells cAdvisor to only monitor the docker containers and not any other types of container runtime like Kubernetes. If you are also running only docker containers then setting this flag to true will help in reducing the overhead of collecting unnecessary metrics.

Another improvement came from the -housekeeping_interval=10s flag. This flag tells cAdvisor to collect the metrics every 10 seconds instead of the default 1 second. You can tweak this number based on your requirements but after doing some trail and error I found that 10 seconds was the sweet spot for me.

With this two changes I was able to reduce the CPU usage of cAdvisor from 15-16% to 5-6% which is a huge improvement.

CPU Usage of cAdvisor before and after the changes
CPU Usage of cAdvisor before and after the changes

Conclusion

If you are seeing high CPU usage from cAdvisor then you can try the above changes to reduce the CPU usage. We explore the two flags -housekeeping_interval and -docker_only which helped in reducing the CPU usage of cAdvisor.

If you have any questions or suggestions, feel free to reach out to me on Twitter ↗️ / Reddit ↗️ or leave a comment below.

Happy monitoring!

You may also like

  • # homelab# selfhosted# networking

    Setup Caddy with automatic SSL certificates with Cloudflare

    Recently I migrated my homelab from using Nginx with local domain certificates to using Caddy with automatic SSL certificates from Cloudflare. This post will go over the steps I took to set up Caddy with Cloudflare.

  • # homelab# selfhosted

    PairDrop — Transfer files between devices seamlessly

    PairDrop is a self-hosted file transfer service that allows you to transfer files between devices seamlessly. It is a great alternative to services like Airdrop, Snapdrop, and ShareDrop.

  • # linux# homelab# selfhosted

    Setup Jellyfin with Hardware Acceleration on Orange Pi 5 (Rockchip RK3558)

    Recently I moved my Jellyfin to an Orange Pi 5 Plus server. The Orange Pi 5 has a Rockchip RK3558 SoC with integrated ARM Mali-G610. This guide will show you how to set up Jellyfin with hardware acceleration on the Orange Pi 5.