Skip to main content

KubeletDown alert: what it means and how to fix it

What the KubeletDown Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

Prometheus has no kubelet target reporting up == 1. Because the rule wraps the whole selector in absent(), it only fires when every kubelet is missing or failing to scrape. One broken node does not trigger it. When it fires you lose kubelet and cAdvisor metrics (container CPU and memory, volume usage) for the whole cluster, and any rule built on them goes quiet.

The rule​

absent(up{job="kubelet", metrics_path="/metrics"} == 1)

Alert message: Kubelet has disappeared from Prometheus target discovery.

Common causes​

  • The scrape configuration changed: the ServiceMonitor for the kubelet was deleted, or its selector, port name or path no longer matches.
  • Prometheus cannot authenticate to the kubelet on port 10250: an expired or missing bearer token, a TLS verification error, or RBAC that no longer grants nodes/metrics.
  • A network policy or firewall now blocks Prometheus from reaching the nodes.
  • Less often, the kubelets really are down across the cluster, for example after a bad configuration rollout.

How to fix it​

  1. On the node run systemctl status kubelet to check the process;
  2. Use journalctl -u kubelet -n 200 --no-pager to find the exit or error reason;
  3. Common causes are expired certificates, config errors and an unavailable container runtime;
  4. After recovery, confirm the node is Ready again and check whether Pods were wrongly evicted meanwhile.

Notes​

  • A single dead kubelet shows up as KubeNodeNotReady, not as this alert. Treat KubeletDown first as a monitoring problem: open the Prometheus /targets page and read the scrape error for the kubelet job.
  • The expression selects job="kubelet". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • up (Prometheus's own metrics)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Kubernetes rule at once as kubernetes.yml.

- alert: KubeletDown
  expr: |-
    absent(up{job="kubelet", metrics_path="/metrics"} == 1)
  for: 15m
  labels:
    severity: critical
  annotations:
    summary: "Kubelet has disappeared from Prometheus target discovery."
    runbook_url: "https://n9e.github.io/alert-rules/kubernetes/kubeletdown"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kubernetes-mixin (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page