Optimizing cAdvisor for Lower CPU Usage

📆 · ⏳ 3 min read · ·

Introduction

I have been running couple of services in my homelab via docker containers. For monitoring purposes I use Grafana and Prometheus. I have been using cAdvisor ↗️ to monitor my docker containers.

Ever since I got the orange pi server, I started to see a noticable difference in the CPU usage being reported by cAdvisor container, it was always hovering around 15-20% CPU usage (which is way higher than any of the containers that I was running in idle state).

I started to investigate this and found a way to reduce the CPU usage of cAdvisor. Let’s see how I did it.

Solution

I won’t go into the details of what is cAdvisor since I believe if you are here you already know what it is. So let’s jump into the solution.

Here’s what my (and probably your’s as well) docker-compose file looked like:

services:
cadvisor:
image: "gcr.io/cadvisor/cadvisor-arm64:v0.49.1"
container_name: "cadvisor"
ports:
- "8080:8080"
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
restart: unless-stopped

The above configuration is the default configuration that is provided in the cAdvisor documentation ↗️. The problem is with the default configuration and how cAdvisor collects the metrics.

Here are the configurations that I changed from the default configuration:

  • -housekeeping_interval from 1s to 10s
  • -docker_only from false to true

Here’s how my updated docker-compose file looks like:

services:
cadvisor:
image: 'gcr.io/cadvisor/cadvisor-arm64:v0.49.1'
container_name: 'cadvisor'
command:
- '-housekeeping_interval=10s'
- '-docker_only=true'
ports:
- '8080:8080'
devices:
- /dev/kmsg:/dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
- /dev/disk/:/dev/disk:ro
restart: unless-stopped

The major improvement came from the -docker_only=true flag. The -docker_only flag tells cAdvisor to only monitor the docker containers and not any other types of container runtime like Kubernetes. If you are also running only docker containers then setting this flag to true will help in reducing the overhead of collecting unnecessary metrics.

Another improvement came from the -housekeeping_interval=10s flag. This flag tells cAdvisor to collect the metrics every 10 seconds instead of the default 1 second. You can tweak this number based on your requirements but after doing some trail and error I found that 10 seconds was the sweet spot for me.

With this two changes I was able to reduce the CPU usage of cAdvisor from 15-16% to 5-6% which is a huge improvement.

CPU Usage of cAdvisor before and after the changes
CPU Usage of cAdvisor before and after the changes

Conclusion

If you are seeing high CPU usage from cAdvisor then you can try the above changes to reduce the CPU usage. We explore the two flags -housekeeping_interval and -docker_only which helped in reducing the CPU usage of cAdvisor.

If you have any questions or suggestions, feel free to reach out to me on Twitter ↗️ / Reddit ↗️ or leave a comment below.

Happy monitoring!

You may also like

  • # docker# raspberrypi

    Resolving Missing Memory Stats in Docker Stats on Raspberry Pi

    Running Docker containers on your Raspberry Pi can be a great way to explore different software and services. However, encountering issues with retrieving memory usage information using docker stats can be frustrating. In this blog post will dive into the common reasons behind missing memory stats and guide you through the troubleshooting steps to resolve them.

  • # postgres# docker

    Safely Backup PostgreSQL Database in Docker / Podman Containers

    In this article, we will learn how to safely backup PostgreSQL database in Docker/Podman containers safely. We will also learn how to restore the database from the backup.

  • # docker

    Reduce Docker Image Size Like a Pro

    Dive into the realm of Docker image optimization and learn how to sculpt sleek and efficient images that won't weigh down your containers. Join me on this journey as we unravel practical techniques and step-by-step methods to trim the excess fat, from choosing the right base image to mastering multi-stage builds. Let's embark on a quest to craft Docker images that are both agile and powerful.