Skip to main content

KubePersistentVolumeFillingUp alert: what it means and how to fix it

What the KubePersistentVolumeFillingUp Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

A PersistentVolume has less than 3% of its space left, as reported by the kubelet on the node that mounts it. Databases, queues and other stateful services usually stop accepting writes or crash once the volume is full, so this one is critical.

The rule​

(
  kubelet_volume_stats_available_bytes{job="kubelet", metrics_path="/metrics"}
    /
  kubelet_volume_stats_capacity_bytes{job="kubelet", metrics_path="/metrics"}
) < 0.03
and
kubelet_volume_stats_used_bytes{job="kubelet", metrics_path="/metrics"} > 0
unless on(namespace, persistentvolumeclaim)
kube_persistentvolumeclaim_access_mode{ access_mode="ReadOnlyMany"} == 1
unless on(namespace, persistentvolumeclaim)
kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1

Alert message: The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free.

Common causes​

  • Data has simply grown: tables, indexes, topic logs or object counts outpaced the size chosen at creation.
  • Retention or cleanup stopped working: log rotation, compaction, TTL jobs or backups that delete old copies.
  • Temporary files, dumps or core files written to the data volume and never removed.
  • The volume was sized for a test environment and reused in production.

How to fix it​

  1. Run kubectl get pvc -n <ns> to find the PVC and PV;
  2. kubectl exec into a Pod using the volume and use du to find the largest directories;
  3. Clean up logs and temporary data;
  4. If the StorageClass supports online expansion, kubectl edit pvc and raise spec.resources.requests.storage; otherwise migrate the data.

Notes​

  • Volumes mounted read-only by many Pods (ReadOnlyMany) are skipped. You can also silence one PVC by labelling it excluded_from_alerts=true, but kube-state-metrics only exports PVC labels that are listed in its --metric-labels-allowlist flag, so add that label there first.
  • The metrics come from the kubelet (kubelet_volume_stats_*). Volumes whose driver does not report usage statistics never produce them, so the rule stays silent for those volumes.
  • The expression selects job="kubelet". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • kubelet_volume_stats_available_bytes (kubelet)
  • kubelet_volume_stats_capacity_bytes (kubelet)
  • kubelet_volume_stats_used_bytes (kubelet)
  • kube_persistentvolumeclaim_access_mode (kube-state-metrics)
  • kube_persistentvolumeclaim_labels (kube-state-metrics)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Kubernetes rule at once as kubernetes.yml.

- alert: KubePersistentVolumeFillingUp
  expr: |-
    (
      kubelet_volume_stats_available_bytes{job="kubelet", metrics_path="/metrics"}
        /
      kubelet_volume_stats_capacity_bytes{job="kubelet", metrics_path="/metrics"}
    ) < 0.03
    and
    kubelet_volume_stats_used_bytes{job="kubelet", metrics_path="/metrics"} > 0
    unless on(namespace, persistentvolumeclaim)
    kube_persistentvolumeclaim_access_mode{ access_mode="ReadOnlyMany"} == 1
    unless on(namespace, persistentvolumeclaim)
    kube_persistentvolumeclaim_labels{label_excluded_from_alerts="true"} == 1
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "The PersistentVolume claimed by {{ $labels.persistentvolumeclaim }} in Namespace {{ $labels.namespace }} is only {{ $value | humanizePercentage }} free."
    runbook_url: "https://n9e.github.io/alert-rules/kubernetes/kubepersistentvolumefillingup"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kubernetes-mixin (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page