Optimizing cAdvisor for Lower CPU Usage

📆 · ⏳ 3 min read · ·

Introduction

I have been running couple of services in my homelab via docker containers. For monitoring purposes I use Grafana and Prometheus. I have been using cAdvisor ↗️ to monitor my docker containers.

Ever since I got the orange pi server, I started to see a noticable difference in the CPU usage being reported by cAdvisor container, it was always hovering around 15-20% CPU usage (which is way higher than any of the containers that I was running in idle state).

I started to investigate this and found a way to reduce the CPU usage of cAdvisor. Let’s see how I did it.

Solution

I won’t go into the details of what is cAdvisor since I believe if you are here you already know what it is. So let’s jump into the solution.

Here’s what my (and probably your’s as well) docker-compose file looked like:

services:
cadvisor:
image: "gcr.io/cadvisor/cadvisor-arm64:v0.49.1"
container_name: "cadvisor"
ports:
- "8080:8080"
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
restart: unless-stopped

The above configuration is the default configuration that is provided in the cAdvisor documentation ↗️. The problem is with the default configuration and how cAdvisor collects the metrics.

Here are the configurations that I changed from the default configuration:

  • -housekeeping_interval from 1s to 10s
  • -docker_only from false to true

Here’s how my updated docker-compose file looks like:

services:
cadvisor:
image: 'gcr.io/cadvisor/cadvisor-arm64:v0.49.1'
container_name: 'cadvisor'
command:
- '-housekeeping_interval=10s'
- '-docker_only=true'
ports:
- '8080:8080'
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
restart: unless-stopped

The major improvement came from the -docker_only=true flag. The -docker_only flag tells cAdvisor to only monitor the docker containers and not any other types of container runtime like Kubernetes. If you are also running only docker containers then setting this flag to true will help in reducing the overhead of collecting unnecessary metrics.

Another improvement came from the -housekeeping_interval=10s flag. This flag tells cAdvisor to collect the metrics every 10 seconds instead of the default 1 second. You can tweak this number based on your requirements but after doing some trail and error I found that 10 seconds was the sweet spot for me.

With this two changes I was able to reduce the CPU usage of cAdvisor from 15-16% to 5-6% which is a huge improvement.

CPU Usage of cAdvisor before and after the changes
CPU Usage of cAdvisor before and after the changes

Conclusion

If you are seeing high CPU usage from cAdvisor then you can try the above changes to reduce the CPU usage. We explore the two flags -housekeeping_interval and -docker_only which helped in reducing the CPU usage of cAdvisor.

If you have any questions or suggestions, feel free to reach out to me on Twitter ↗️ / Reddit ↗️ or leave a comment below.

Happy monitoring!

You may also like

  • Jellyfin + arr stack — Self-hosted media streaming in my Homelab

    Since ages, I have been collecting lots of movies, TV shows, and music. Ever since I got into self hosting, I have been looking for a way to stream my media collection to my devices. Jellyfin is the perfect solution for this.

  • Syncing made easy with Syncthing

    Syncthing is one of those tools which have made my life easier. It is a decentralized file synchronization tool which is open source and free to use. Learn how to set it up and use it.

  • Welcoming a new member to my Homelab — A Orange Pi 5 Plus

    Happy to share that I have added a new member to my homelab, a Orange Pi 5 Plus. In this post, I will share the specs and the initial setup process.