Prometheus alert rules with runbooks
238 Prometheus alert rules for 8 services, including Kubernetes, Linux, MySQL and Redis. Every rule has a PromQL condition, a severity, a duration and runbook steps. Free to download as Prometheus rules files.
Last reviewed
An alert rule is only half the job: when it fires at 3 a.m., someone needs to know what to check first. This library collects 238 PromQL alert rules from Nightingale's integration templates, and every one of them comes with runbook steps written for the person on call. The rules use standard exporter metric names (node_exporter, kube-state-metrics, mysqld_exporter and so on), so they work with plain Prometheus as well as with Nightingale.
| Service | Rules | Metrics from | Download |
|---|---|---|---|
| Kubernetes | 77 | kube-state-metrics, cAdvisor (kubelet), apiserver | kubernetes.yml |
| Prometheus | 30 | Prometheus's own /metrics, Prometheus Operator | prometheus.yml |
| Linux | 76 | node_exporter, Categraf, Telegraf | linux.yml |
| MySQL | 11 | mysqld_exporter, Categraf | mysql.yml |
| Redis | 12 | redis_exporter, Categraf | redis.yml |
| Elasticsearch | 11 | elasticsearch_exporter, Categraf | elasticsearch.yml |
| MongoDB | 8 | mongodb_exporter | mongodb.yml |
| Windows | 13 | windows_exporter, Categraf | windows.yml |
Runbooks by alert name
These standard alert names, used by kubernetes-mixin and kube-prometheus, each have their own page with the meaning, common causes, the rule and the runbook.
- Kubernetes: KubeClientCertificateExpiration, KubeAPIErrorBudgetBurn, KubeAggregatedAPIDown, KubeContainerWaiting, KubeCPUOvercommit, KubeDaemonSetMisScheduled, KubeDeploymentReplicasMismatch, KubeDeploymentRolloutStuck, KubeHpaMaxedOut, KubeJobFailed, KubeletDown, KubeletTooManyPods, KubeMemoryOvercommit, KubeNodeNotReady, KubePersistentVolumeFillingUp, KubePodCrashLooping, KubePodNotReady, KubeQuotaAlmostFull, KubeStatefulSetReplicasMismatch
- Prometheus: PrometheusOperatorRejectedResources, PrometheusBadConfig, PrometheusNotConnectedToAlertmanagers, PrometheusRemoteWriteBehind, PrometheusRemoteWriteDesiredShards, PrometheusRuleFailures, PrometheusTargetLimitHit
Rule format
- Each service page lists every rule with its PromQL condition, severity,
forduration and runbook steps, grouped by the collector that produces the metrics. Standard alert names with their own page appear there as a short entry with a link. - Each
.ymlfile is a standard Prometheus rules file and passespromtool check rules. Severity is aseveritylabel (critical / warning / info), the summary is inannotations.summary, andannotations.runbook_urllinks to the rule page. - Thresholds are starting points, not recommendations for every environment. Review them before you enable a rule.
Why the runbooks matter
A rule without a runbook moves the investigation to whoever is on call. The steps here say what to look at first (the log before the last restart, the exit code, the slow query list, the replication thread state) and what usually fixes it, so the alert carries its own first response.
Sources and license
Rule expressions are partly adapted from community collections: kubernetes-mixin (Apache-2.0), kube-prometheus (Apache-2.0), awesome-prometheus-alerts (CC BY 4.0). Runbook steps and the remaining rules come from Nightingale's integration templates (Apache-2.0).
Related
- Integrations: collector config and dashboards for each service
- Alert rules in Nightingale
- Rule design practices