# Monitor cAdvisor with Nightingale Scrapes the cAdvisor built into kubelet or a standalone one for per-container CPU, memory, filesystem and network metrics; in kubelet mode it can attach Pod labels and annotations. Last reviewed 2026-10-13 The cAdvisor input collects the `container_*` metrics that cAdvisor exposes: per-container CPU usage and throttling, memory working set and OOM events, filesystem reads and writes, network traffic and drops. Two ways to connect: `type = "kubelet"` scrapes the kubelet's `/metrics/cadvisor` (port 10250, authenticated) and can attach Pod labels and annotations to the metrics; `type = "cadvisor"` scrapes a standalone cAdvisor (port 8080 by default). The dashboard shows resource usage per container and the metric reference explains each `container_*` metric. The rules cover CPU throttling, memory near the limit, OOM kills and network drops. ## What's in the template[​](#whats-in-the-template "Direct link to What's in the template") * **1 dashboard**, 5 panels in total * **6 alert rules**, each with runbook steps * **45 metric descriptions** * **Categraf collector config**: `input.cadvisor` ## How to connect[​](#how-to-connect "Direct link to How to connect") 1. [Install Categraf](https://n9e.github.io/docs/data/categraf/install.md), put the config below in `conf/input.cadvisor/cadvisor.toml`, fill in your own address and credentials, and restart Categraf. The startup log prints `input: local.cadvisor started` once the plugin is loaded. ``` # # collect interval # interval = 15 [[instances]] # url = "https://1.2.3.4:10250" # type = "kubelet" ## url = "http://1.2.3.4:8080/metrics" ## type = "cadvisor" # url_label_key = "instance" # url_label_value = "{{.Host}}" # bearer_token_string = "eyJlonglongxxxx.eyJlonglongyyyy.oQsXlonglongZZZ" ## bearer_token_file = "/path/to/token/file" # ignore_label_keys = ["id","name", "container_label*"] ## choose_label_keys = ["id"] # timeout = "3s" # use_tls = true ## tls_min_version = "1.2" ## tls_ca = "/etc/categraf/ca.pem" ## tls_cert = "/etc/categraf/cert.pem" ## tls_key = "/etc/categraf/key.pem" ## Use TLS but skip chain & host verification ## insecure_skip_verify = true ``` 2. Open the integration center in Nightingale, find cAdvisor, and [import the dashboards and alert rules](https://n9e.github.io/docs/data/integrations/import.md) into a business group. 3. Imported rules start disabled. [Check that the metrics are arriving](https://n9e.github.io/docs/data/integrations/verify.md) first, then enable the rules one by one with thresholds that fit your environment. ## Bundled dashboards[​](#bundled-dashboards "Direct link to Bundled dashboards") | Dashboard | Panels | | --------- | ------ | | cAdvisor | 5 | ## Bundled alert rules[​](#bundled-alert-rules "Direct link to Bundled alert rules") 6 rules. Each lists its trigger condition (PromQL) and the steps to take when it fires. Thresholds are meant to be adjusted after import. ### Container memory approaching its limit[​](#container-memory-approaching-its-limit "Direct link to Container memory approaching its limit") S1 Critical · for 5 min · Trigger: ``` container_memory_working_set_bytes{name!=""} / (container_spec_memory_limit_bytes{name!=""} > 0) * 100 > 90 ``` Runbook: 1. Take name from the alert labels and run docker stats \ or kubectl top pod to confirm live usage; 2. Use the application's own memory analysis inside the container (jmap -histo:live for Java, pprof heap for Go) to separate normal growth from a leak; 3. If the limit is simply too small, raise it and redeploy, adjusting the application's own heap ceiling to match (-Xmx must leave room for off-heap memory); 4. If it is a leak, estimate the remaining runway from the growth rate and add an automatic restart as a stopgap while you investigate the code. ### Container OOM killed[​](#container-oom-killed "Direct link to Container OOM killed") S1 Critical · for 1 min · Trigger: ``` increase(container_oom_events_total{name!=""}[10m]) > 0 ``` Runbook: 1. Take name from the alert labels and confirm the OOMKilled state via the State.OOMKilled field in docker inspect \ output or with kubectl describe pod; 2. Run dmesg -T | grep -i 'killed process' to find the killed process and its memory usage at the time; 3. Compare that usage against the limit and raise the limit if it is clearly too small; 4. If usage looks abnormal, investigate it as a memory leak; for Java, -Xmx must be smaller than the container limit with room for off-heap memory or an OOM is guaranteed. ### Container CPU heavily throttled[​](#container-cpu-heavily-throttled "Direct link to Container CPU heavily throttled") S2 Warning · for 5 min · Trigger: ``` rate(container_cpu_cfs_throttled_periods_total{name!=""}[5m]) / (rate(container_cpu_cfs_periods_total{name!=""}[5m]) > 0) * 100 > 25 ``` Runbook: 1. Take name from the alert labels and confirm the growth of nr\_throttled and throttled\_usec with cat /sys/fs/cgroup/cpu.stat (cgroup v2); 2. Compare actual CPU usage against the limit; being throttled while below the limit usually means bursts are being sliced up by the 100 ms period, so raise the limit slightly or lower in-period concurrency; 3. A common trigger is a JVM or Go runtime sizing thread pools from the host core count; set GOMAXPROCS or -XX:ActiveProcessorCount to match the limit; 4. If more compute is genuinely needed, raise the CPU limit and watch the throttling ratio fall. ### Container filesystem usage too high[​](#container-filesystem-usage-too-high "Direct link to Container filesystem usage too high") S2 Warning · for 5 min · Trigger: ``` container_fs_usage_bytes{name!=""} / (container_fs_limit_bytes{name!=""} > 0) * 100 > 85 ``` Runbook: 1. Take name and device from the alert labels and run docker exec \ du -sh /\* 2>/dev/null | sort -rh | head to find the biggest consumers; 2. In the vast majority of cases application logs are not being rotated: configure logrotate or switch to writing to stdout and let the runtime rotate them; 3. If the writable layer is filling up, mount a volume for directories that need persistence rather than writing into the layer; 4. As an emergency measure run docker system prune to reclaim host space from unused images and stopped containers. ### Container restarting repeatedly[​](#container-restarting-repeatedly "Direct link to Container restarting repeatedly") S2 Warning · for 5 min · Trigger: ``` changes(container_start_time_seconds{name!=""}[1h]) > 3 ``` Runbook: 1. Take name from the alert labels and run docker logs --tail 200 \ or kubectl logs --previous to read the logs from before the last exit; 2. Run docker inspect \ and check State.ExitCode and State.OOMKilled in the output to distinguish an OOM, a deliberate application exit, and a failing health check; 3. Handle OOMKilled as a memory alert; for a non-zero exit code, find the start-up failure in the application logs (missing configuration, dependency not ready); 4. If a health check is the cause, confirm the probe's initial delay is longer than the real start-up time and raise the start period if needed. ### Container network receive packet drops[​](#container-network-receive-packet-drops "Direct link to Container network receive packet drops") S3 Info · for 5 min · Trigger: ``` rate(container_network_receive_packets_dropped_total{name!=""}[5m]) > 1 ``` Runbook: 1. Take name from the alert labels and run docker exec \ cat /proc/net/dev plus ss -s to check drops and queue state; 2. If the receive queue is overflowing, raise the kernel's net.core.netdev\_max\_backlog and the application's accept queue (somaxconn); 3. Check whether the host NIC is also dropping packets (ethtool -S \ | grep -i drop); if the problem is at the host level, do not chase it inside the container; 4. If traffic genuinely exceeds the container's networking capacity, evaluate switching to host networking or raising the bandwidth allocation. ## Metric reference[​](#metric-reference "Direct link to Metric reference") 45 metrics, the ones the dashboards and rules above are built on. | Metric | Meaning | Unit | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------- | | `irate(container_cpu_system_seconds_total{image!="", image!~".*pause.*"}[3m]) * 100` | Container CPU utilization (system) | % | | `irate(container_cpu_user_seconds_total{image!="", image!~".*pause.*"}[3m]) * 100` | Container CPU utilization (user) | % | | `sum( irate(container_cpu_usage_seconds_total{image!="", image!~".*pause.*"}[3m]) ) by (pod,namespace,container,image) / sum( container_spec_cpu_quota/container_spec_cpu_period ) by (pod,namespace,container,image)` | Container CPU utilization (overall, the value will not be greater than 100) — Only containers with limit set can calculate this utilization | % | | `irate(container_cpu_usage_seconds_total{image!="", image!~".*pause.*"}[3m]) * 100` | Container CPU utilization (overall, value may be greater than 100) — If 200%, it means that 2 cores are occupied | % | | `irate(container_cpu_cfs_periods_total{}[3m])` | How many periods does the container CPU have per second | | | `irate(container_cpu_cfs_throttled_periods_total{}[3m])` | The amount of periods that the container CPU is throttle per second — If the container limits the CPU and the app requires too many algorithms, it will be suppressed. container \_ CPU \_ cfs \_ throttled \_ periods \_ total counts how many periods have been suppressed in total. If suppression occurs recently, it needs attention. Some delay-sensitive apps are particularly affected. If it is suppressed, there is a high probability that it needs to be upgraded. | | | `irate(container_cpu_cfs_throttled_periods_total{}[3m]) / irate(container_cpu_cfs_periods_total{}[3m]) * 100` | The proportion of container CPU being throttle — If this value is greater than 0, pay attention | % | | `container_fs_usage_bytes / container_fs_limit_bytes * 100` | Container filesystem usage | % | | `container_fs_usage_bytes` | Container filesystem usage | bytes | | `container_fs_io_current` | Container filesystem Current IO times | | | `container_fs_limit_bytes` | Container filesystem Total | bytes | | `container_fs_inodes_free` | Container inode free amount | count | | `container_fs_inodes_total` | Container inode total | count | | `100 - container_fs_inodes_free / container_fs_inodes_total * 100` | Container inode usage | % | | `sum(irate(container_fs_writes_bytes_total[3m])) by (namespace, pod)` | Container IO writes bytes per second | bytes/s | | `sum(irate(container_fs_reads_bytes_total[3m])) by (namespace, pod)` | Container IO reads bytes per second | bytes/s | | `container_memory_cache{image!="", image!~".*pause.*"}` | Container memory cache amount | bytes | | `100 * container_memory_usage_bytes/container_spec_memory_limit_bytes and container_spec_memory_limit_bytes != 0` | Container memory Usage (Usage) — If there is a large number of file IO and a large number of container \_ memory \_ cache, the size of container \_ memory \_ usage \_ bytes and container \_ memory \_ working \_ set \_ bytes will be different | % | | `100 * container_memory_working_set_bytes/container_spec_memory_limit_bytes and container_spec_memory_limit_bytes != 0` | Container memory usage rate (Working Set) — If there is a large number of file IO and a large number of container \_ memory \_ cache, the size of container \_ memory \_ usage \_ bytes and container \_ memory \_ working \_ set \_ bytes will be different | % | | `container_memory_mapped_file{image!="", image!~".*pause.*"}` | Container memory usage (mapped \_ file) | bytes | | `container_memory_rss{image!="", image!~".*pause.*"}` | Container memory usage (RSS) | bytes | | `container_memory_swap{image!="", image!~".*pause.*"}` | Container memory usage (Swap) | bytes | | `container_memory_usage_bytes{image!="", image!~".*pause.*"}` | Container memory Usage — If there is a large number of file IO and a large number of container \_ memory \_ cache, the size of container \_ memory \_ usage \_ bytes and container \_ memory \_ working \_ set \_ bytes will be different | bytes | | `container_memory_working_set_bytes{image!="", image!~".*pause.*"}` | Container memory usage (Working Set) — If there is a large number of file IO and a large number of container \_ memory \_ cache, the size of container \_ memory \_ usage \_ bytes and container \_ memory \_ working \_ set \_ bytes will be different | bytes | | `rate(container_memory_failures_total{}[3m])` | Container memory allocation failures (per second) | | | `container_spec_memory_limit_bytes{image!="", image!~".*pause.*"}` | Container memory limit | bytes | | `sum(irate(container_network_transmit_bytes_total[3m])) by (namespace, pod) * 8` | Container net sends bits per second | bit/s | | `sum(irate(container_network_transmit_bytes_total[3m])) by (namespace, pod)` | Container net sends bytes per second | bytes/s | | `irate(container_network_transmit_packets_total[3m])` | Number of packets sent per second by container net | count | | `irate(container_network_transmit_packets_dropped_total[3m])` | Number of drop packets sent by container net per second | count | | `irate(container_network_transmit_errors_total[3m])` | Number of wrong packets sent by container net per second | count | | `sum(irate(container_network_receive_bytes_total[3m])) by (namespace, pod) * 8` | The amount of bits received by the container net per second | bit/s | | `sum(irate(container_network_receive_bytes_total[3m])) by (namespace, pod)` | Container net receives bytes per second | bytes/s | | `irate(container_network_receive_packets_total[3m])` | Number of packets received per second by container net | count | | `irate(container_network_receive_packets_dropped_total[3m])` | Number of drop packets received by container net per second | count | | `irate(container_network_receive_errors_total[3m])` | Number of wrong packets received by container net per second | count | | `container_threads_max{image!="", image!~".*pause.*"}` | The maximum number of threads the container is allowed to run | | | `container_ulimits_soft{image!="", image!~".*pause.*"}` | Process No. 1 soft ulimit value in container — Soft ulimit value for process # 1 inside the container. If-1, there is no limit. | | | `container_start_time_seconds{image!="", image!~".*pause.*"}` | How long the container has been running | s | | `container_sockets{image!="", image!~".*pause.*"}` | Number of currently open sockets in the container | | | `container_file_descriptors{image!="", image!~".*pause.*"}` | Container Number of currently open file handles | | | `container_threads{image!="", image!~".*pause.*"}` | Number of threads currently running in the container | | | `container_processes{image!="", image!~".*pause.*"}` | Number of processes currently running in the container | | | `container_accelerator_memory_total_bytes` | Container Total GPU Accelerator Available Memory | bytes | | `container_accelerator_memory_used_bytes` | The amount of GPU accelerator card memory the container is using | bytes | ## Related[​](#related "Direct link to Related") * [cAdvisor template source](https://github.com/ccfos/nightingale/tree/main/integrations/cAdvisor): the raw JSON for dashboards, rules and collector config. Pull requests welcome. * [Categraf `cadvisor` plugin docs](https://github.com/flashcatcloud/categraf/tree/main/inputs/cadvisor): every config option and the full metric list. * [Import dashboards and alert rules](https://n9e.github.io/docs/data/integrations/import.md) * [Verify that metrics are stored](https://n9e.github.io/docs/data/integrations/verify.md) * [All integrations](https://n9e.github.io/integrations.md)