Skip to main content

Monitor Docker with Nightingale

Collects per-container CPU, memory, block and network IO plus daemon state through the Docker API; the dashboard is written against docker_* metrics and verified on real hosts.

Last reviewed

The Docker input is based on Telegraf's docker plugin. It connects to the Docker daemon socket (unix:///var/run/docker.sock) or a TCP endpoint and collects engine-level container counts and states, image count and storage driver information, plus CPU, memory, blkio and network IO for every container. Swarm services and extended memory statistics are optional.

The bundled Docker Dashboard is written against these docker_* metrics and verified on real hosts, so this input must be enabled for it. cAdvisor is also a recommended way to collect containers, but it emits container_* metrics and needs the dashboards from the cAdvisor integration; the two do not mix. The rules cover container OOM kills, abnormal exits, CPU and memory over limits and the daemon being unreachable.

What's in the template​

  • 1 dashboard, 16 panels in total
  • 6 alert rules, each with runbook steps
  • Categraf collector config: input.docker

How to connect​

  1. Install Categraf, put the config below in conf/input.docker/docker.toml, fill in your own address and credentials, and restart Categraf. The startup log prints input: local.docker started once the plugin is loaded.

    # # collect interval
    # interval = 15

    [[instances]]
    # # append some labels for series
    # labels = { region="cloud", product="n9e" }

    # # interval = global.interval * interval_times
    # interval_times = 1

    ## Docker Endpoint
    ##   To use TCP, set endpoint = "tcp://[ip]:[port]"
    ##   To use environment variables (ie, docker-machine), set endpoint = "ENV"
    # endpoint = "unix:///var/run/docker.sock"
    endpoint = ""

    ## Set to true to collect Swarm metrics(desired_replicas, running_replicas)
    gather_services = false
    gather_extend_memstats = false

    container_id_label_enable = true
    container_id_label_short_style = false

    ## Containers to include and exclude. Globs accepted.
    ## Note that an empty array for both will include all containers
    container_name_include = []
    container_name_exclude = []

    ## Container states to include and exclude. Globs accepted.
    ## When empty only containers in the "running" state will be captured.
    ## example: container_state_include = ["created", "restarting", "running", "removing", "paused", "exited", "dead"]
    ## example: container_state_exclude = ["created", "restarting", "running", "removing", "paused", "exited", "dead"]
    # container_state_include = []
    # container_state_exclude = []

    ## Timeout for docker list, info, and stats commands
    timeout = "5s"

    ## Specifies for which classes a per-device metric should be issued
    ## Possible values are 'cpu' (cpu0, cpu1, ...), 'blkio' (8:0, 8:1, ...) and 'network' (eth0, eth1, ...)
    ## Please note that this setting has no effect if 'perdevice' is set to 'true'
    perdevice_include = []

    ## Specifies for which classes a total metric should be issued. Total is an aggregated of the 'perdevice' values.
    ## Possible values are 'cpu', 'blkio' and 'network'
    ## Total 'cpu' is reported directly by Docker daemon, and 'network' and 'blkio' totals are aggregated by this plugin.
    ## Please note that this setting has no effect if 'total' is set to 'false'
    total_include = ["cpu", "blkio", "network"]

    ## Which environment variables should we use as a tag
    ##tag_env = ["JAVA_HOME", "HEAP_SIZE"]

    ## docker labels to include and exclude as tags. Globs accepted.
    ## Note that an empty array for both will include all labels as tags
    docker_label_include = []
    docker_label_exclude = ["annotation*", "io.kubernetes*", "*description*", "*maintainer*", "*hash", "*author*", "*org_*", "*date*", "*url*", "*docker_compose*"]

    ## Optional TLS Config
    # use_tls = false
    # tls_ca = "/etc/telegraf/ca.pem"
    # tls_cert = "/etc/telegraf/cert.pem"
    # tls_key = "/etc/telegraf/key.pem"
    ## Use TLS but skip chain & host verification
    # insecure_skip_verify = false

  2. Open the integration center in Nightingale, find Docker, and import the dashboards and alert rules into a business group.

  3. Imported rules start disabled. Check that the metrics are arriving first, then enable the rules one by one with thresholds that fit your environment.

Bundled dashboards​

DashboardPanels
Docker Dashboard16

Bundled alert rules​

6 rules. Each lists its trigger condition (PromQL) and the steps to take when it fires. Thresholds are meant to be adjusted after import.

Docker daemon unreachable​

S1 Critical · for 1 min · Trigger:

docker_up == 0

Runbook:

  1. Log in to the host and run systemctl status docker to confirm the daemon is alive;
  2. Run docker ps to verify by hand — if it hangs without returning, the daemon is stuck, use journalctl -u docker -n 200 to locate the cause;
  3. Confirm the permissions on /var/run/docker.sock match the collector account;
  4. Confirm the disk (especially the partition holding /var/lib/docker) is not full, a full disk is a common cause of a stuck daemon.

Docker container health check failing​

S1 Critical · for 3 min · Trigger:

docker_container_health_status == 2

Runbook:

  1. Take container_name from the alert labels and run docker inspect <container> and check the State.Health field in the output for the last few health check results;
  2. Run docker logs --tail 200 <container> to locate the application error;
  3. Reproduce the health check by hand (docker exec <container> <health check command>) to tell an application failure apart from a wrongly written check;
  4. If the application is broken and cannot be fixed quickly, first remove the instance from the upstream load balancer, then investigate.

Docker container OOM killed​

S1 Critical · Trigger:

docker_container_status_oomkilled == 1

Runbook:

  1. Take container_name from the alert labels and run docker inspect <container> | grep -i memory to confirm the current limit;
  2. Compare with the historical peak of docker_container_mem_max_usage to tell a too-small limit apart from an application memory leak;
  3. If the limit is too small, raise --memory and recreate the container;
  4. If a leak is suspected, capture a heap profile on the application side (jmap for Java, pprof for Go) instead of just raising the limit to hide the problem.

Docker container restart loop​

S1 Critical · Trigger:

increase(docker_container_status_restart_count[10m]) > 3

Runbook:

  1. Take container_name from the alert labels and run docker logs --tail 200 <container> to see the error before each exit;
  2. Run docker inspect <container> and read the State.ExitCode exit code in the output (137 usually means OOM or killed; 1 or 2 usually means the application failed to start);
  3. For configuration or dependency problems (cannot reach the database, missing config file), fix the dependency before starting it again;
  4. To keep the restart loop from filling the disk with logs, docker stop the container and remove it from the load balancer while you investigate.

Docker container memory approaching its limit​

S2 Warning · for 5 min · Trigger:

docker_container_mem_usage_percent > 90

Runbook:

  1. Take container_name from the alert labels and run docker stats --no-stream <container> to confirm current usage against the limit;
  2. Check the history curve to tell a steady climb (likely a leak) apart from a normal traffic peak;
  3. For normal peaks, raise --memory leaving 20% headroom and recreate the container;
  4. For a steady climb, capture a heap profile on the application side to find the leak, and raise the limit meanwhile to buy investigation time.

Docker container CPU throttled​

S2 Warning · for 5 min · Trigger:

rate(docker_container_cpu_throttling_throttled_periods[5m]) / rate(docker_container_cpu_throttling_periods[5m]) > 0.25

Runbook:

  1. Take container_name from the alert labels and run docker inspect <container> | grep -iE 'cpuquota|cpuperiod|nanocpus' to confirm the current quota;
  2. Compare with docker_container_cpu_usage_percent to tell a traffic increase apart from a busy loop in the application;
  3. If traffic grew, raise --cpus and recreate the container, or scale out to several instances;
  4. If usage looks abnormal, capture a CPU profile on the application side to find the hot functions.

On this page