Skip to main content

Prometheus alert rules with runbooks

238 Prometheus alert rules for 8 services, including Kubernetes, Linux, MySQL and Redis. Every rule has a PromQL condition, a severity, a duration and runbook steps. Free to download as Prometheus rules files.

Last reviewed

An alert rule is only half the job: when it fires at 3 a.m., someone needs to know what to check first. This library collects 238 PromQL alert rules from Nightingale's integration templates, and every one of them comes with runbook steps written for the person on call. The rules use standard exporter metric names (node_exporter, kube-state-metrics, mysqld_exporter and so on), so they work with plain Prometheus as well as with Nightingale.

ServiceRulesMetrics fromDownload
Kubernetes77kube-state-metrics, cAdvisor (kubelet), apiserverkubernetes.yml
Prometheus30Prometheus's own /metrics, Prometheus Operatorprometheus.yml
Linux76node_exporter, Categraf, Telegraflinux.yml
MySQL11mysqld_exporter, Categrafmysql.yml
Redis12redis_exporter, Categrafredis.yml
Elasticsearch11elasticsearch_exporter, Categrafelasticsearch.yml
MongoDB8mongodb_exportermongodb.yml
Windows13windows_exporter, Categrafwindows.yml

Runbooks by alert name​

These standard alert names, used by kubernetes-mixin and kube-prometheus, each have their own page with the meaning, common causes, the rule and the runbook.

Rule format​

  • Each service page lists every rule with its PromQL condition, severity, for duration and runbook steps, grouped by the collector that produces the metrics. Standard alert names with their own page appear there as a short entry with a link.
  • Each .yml file is a standard Prometheus rules file and passes promtool check rules. Severity is a severity label (critical / warning / info), the summary is in annotations.summary, and annotations.runbook_url links to the rule page.
  • Thresholds are starting points, not recommendations for every environment. Review them before you enable a rule.

Why the runbooks matter​

A rule without a runbook moves the investigation to whoever is on call. The steps here say what to look at first (the log before the last restart, the exit code, the slow query list, the replication thread state) and what usually fixes it, so the alert carries its own first response.

Sources and license​

Rule expressions are partly adapted from community collections: kubernetes-mixin (Apache-2.0), kube-prometheus (Apache-2.0), awesome-prometheus-alerts (CC BY 4.0). Runbook steps and the remaining rules come from Nightingale's integration templates (Apache-2.0).