Every alert rule answers two questions: what counts as bad, and how long it has to stay bad. Most review effort goes into the first one, the expression and the threshold. The second is the for duration, and it often gets whatever value the tool filled in.

We wanted to know how often that happens, so we counted. Nightingale ships alert rule templates for 49 integrations. 692 rules, and 304 of them (44%) have a for duration of exactly 60 seconds, which is the default in Nightingale's rule editor. This post walks through what the numbers show and what for value suits which kind of alert.

What for does​

An alert rule is evaluated on a schedule. When the expression first returns a series, the alert is pending. It becomes firing only once the same series has kept matching on every evaluation for at least the for duration. If the series disappears for even one evaluation, the clock starts over.

Three consequences follow:

• Detection time is about for plus one evaluation interval. A rule with for: 5m evaluated every minute fires five to six minutes after the problem starts.
• A series that flaps never fires. If the condition comes and goes faster than for, no alert is created, and no recovery either. That is the point of for, and also a risk.
• The defaults differ by tool. In Prometheus, leaving out for means 0, so the alert fires on the first evaluation that matches. In Nightingale's rule editor, a new rule starts at a 60-second for and a 60-second evaluation interval. The bundled templates set their own interval, almost always 15 or 30 seconds.

How we counted​

The data is the alert rule templates bundled with Nightingale's integrations: Kubernetes, Linux, MySQL, Redis, Elasticsearch and 44 more. They are published in the alert rule library and on each integration page.

• 705 rules in total, 692 after removing exact duplicates (same integration, same name, same expression).
• Some checks exist twice on purpose, once for Categraf metrics and once for a Prometheus exporter, because the metric names differ. We count those separately.
• These are templates, not production configs. They show what template authors chose. They say nothing about what users changed after importing them.
• Counted on 2026-11-01, after the two changes described in Finding 4.

The distribution​

Three rules use 15s or 90s and are left out of the table.

Finding 1: the default wins, unless someone chose deliberately​

The 60-second spike is the obvious feature. We can't tell from the data whether 304 authors thought about it and picked 60s, or simply kept what the form offered. What we can see is a clear difference between where the rules came from:

• 86 rules in the Kubernetes template keep the standard alert names from kubernetes-mixin and the Prometheus monitoring mixins, the community rule sets behind most Kubernetes monitoring stacks. 46 of them use 15 minutes, and only 2 use 60 seconds. The mixin authors set for alert by alert.
• Rules written for Categraf and for Prometheus exporters have a median of 60 seconds. Only 4 of the 352 use 15 minutes.

Neither is wrong as a whole. The point is that a default is a choice someone else made for you, before they knew what the rule would watch. With the templates' 15- or 30-second evaluation intervals, a 60-second for means "matched on two to four evaluations in a row". That is reasonable for CPU, memory and latency. It is too short for things that recover on their own, and too long for events that are over in seconds.

Finding 2: for: 0 is right for events, if the window is wide enough​

51 rules fire on the first match (four of them are host heartbeat checks with no PromQL). Most of the rest watch something that has already happened, so waiting adds nothing:

• A counter went up: increase(node_vmstat_oom_kill[1m]) > 0. The OOM kill happened; it will not un-happen in five minutes.
• A deadline is close: kubelet_certificate_manager_client_ttl_seconds < 604800. Under seven days is under seven days.
• A process restarted: mysql_global_status_uptime < 60.

The restart check hides a trap. uptime < 60 is only true for the first 60 seconds after a restart. Any for of 60 seconds or more means the alert can never fire, because uptime passes 60 before the duration is met. And the window has to fit several evaluations: copy uptime < 60 into a rule evaluated every 60 seconds, and it gets one chance per restart, which scrape delay can easily push out of the window.

The templates get this right in different ways. The MySQL and MongoDB rules use uptime < 60 with for: 0, and they evaluate every 15 seconds, so each restart gets about four chances. The Redis rules widen the window to uptime_in_seconds < 600. The Apache, EMQX and OpenLDAP rules use uptime < 300 with for: 60s. Make the window several evaluation intervals wide, keep for well under it, and keep both in mind if you change the evaluation interval.

Finding 3: some rules move the waiting into the expression​

Of the 47 for: 0 rules that have a PromQL expression, 22 wrap their metric in a range function, so the persistence check happens inside PromQL:

# GitLab's ServiceDown template, for: 0
avg_over_time(up[5m]) * 100 < 50

This is not the same as up == 0 with for: 5m:

Pick the one that matches what you mean. A target that answers one scrape in four is broken, and only the second form will say so.

Finding 4: a few instant checks deserved a longer wait​

Counting also turned up for: 0 rules that compare a sampled value against a threshold with no window at all. Two of them we changed while writing this post:

• elasticsearch_cluster_health_status{color="yellow"} == 1. A cluster goes yellow on every rolling restart while replicas are reallocated. It now waits 10 minutes.
• (node_nf_conntrack_entries / node_nf_conntrack_entries_limit) > 0.75. A single busy scrape was enough to trigger it. It now waits 5 minutes, the same as the sibling rule at 80%.

The tables in this post already include both changes. The general rule: give sampled gauges a few evaluation intervals, or a range function as in Finding 3. Keep for: 0 for the checks that really are binary: mysql_up == 0, a failed systemd unit, a replication thread that stopped.

Finding 5: long waits are for slow problems​

26 rules wait longer than 15 minutes. Most of them watch something that develops slowly:

• The node filesystem predictions from kubernetes-mixin use one hour. A predict_linear forecast swings a lot on short windows; an hour of consistent prediction means the trend is real.
• Prometheus TSDB compaction failures wait four hours. One failed compaction is noise; a run of them is a problem.
• A kernel version mismatch across nodes waits six hours. It is a hygiene check, not an incident.
• The slow windows of the Kubernetes API error budget burn alerts wait one and three hours. They are designed to catch slow, steady burns.

Long for values pair naturally with a lower severity. Nobody should be paged at 3 a.m. for something that took six hours to become true.

The severity side​

The same templates carry a severity. Nightingale has three levels: S1 (critical), S2 (warning) and S3 (info).

Five rules have no rule-level severity. When seven rules in ten share a level, the level stops helping anyone route or prioritize. A workable split:

• S1: someone must act now. Users are affected or data is about to be lost.
• S2: someone should look today. Capacity trends, single replicas down with redundancy left.
• S3: record it, don't notify. Useful context during an incident.

24 templates put several severities on one rule. The Linux CPU rule, for example, uses cpu_usage_active > 75 as S3, > 85 as S2 and > 95 as S1. One rule with three tiers keeps the thresholds side by side, which is easier to review than three separate rules that drift apart.

Rules of thumb​

for only controls when an alert starts. If an alert fires and recovers over and over, for is not the fix. Use a recovery delay instead: keep_firing_for in Prometheus, or the recover duration in Nightingale. See Evaluation interval and recovery.

Where to look next​

Every rule in the alert rule library shows its severity and its for value ("Fires after"), and links to a runbook. You can download a whole rule set as a Prometheus rules file, or import it into Nightingale from the matching integration.

Kubernetes metrics come from four places, and their names look alike. kube_pod_container_resource_limits, container_memory_working_set_bytes, node_memory_MemAvailable_bytes and the numbers in kubectl top all sound like they describe the same thing. They are made by different components and they answer different questions. Only three of the four are meant for monitoring.

This post explains what each component measures and how to tell which one a metric came from. It also covers the renames that make a query copied from a 2019 blog post return nothing.

The short answer​

You will also meet two more sources. The kubelet's own /metrics endpoint provides things like kubelet_volume_stats_available_bytes, which tells you how full a PersistentVolume is. The API server exposes apiserver_* metrics about itself. Neither belongs to the four above, and alert rules use both.

kube-state-metrics: what the API server believes​

kube-state-metrics listens to the API server and turns objects into metrics. Its README calls this "the health of the various objects inside" the cluster, as opposed to the health of the cluster components themselves. It exposes the values as the API returns them. It does not apply the heuristics kubectl uses, which is why its numbers sometimes differ from what kubectl get prints.

That makes it the right source for questions about intent:

• Does the Deployment have the replicas it asked for? kube_deployment_spec_replicas vs kube_deployment_status_replicas_available.
• Is a container stuck in CrashLoopBackOff? kube_pod_container_status_waiting_reason.
• Why did the last run end? kube_pod_container_status_last_terminated_reason, where reason="OOMKilled" is the one you usually want.
• Is a node NotReady? kube_node_status_condition.

It tells you nothing about usage. A pod at 100% of its CPU limit and an idle pod look the same to kube-state-metrics. For usage you need cAdvisor.

The rename that breaks old queries. kube-state-metrics 2.0 removed the per-resource metrics such as kube_pod_container_resource_limits_cpu_cores and kube_pod_container_resource_requests_memory_bytes. Their replacements are kube_pod_container_resource_limits and kube_pod_container_resource_requests, with resource and unit labels:

# kube-state-metrics 1.x (removed in 2.0)
kube_pod_container_resource_limits_cpu_cores

# kube-state-metrics 2.x
kube_pod_container_resource_limits{resource="cpu", unit="core"}

A query that uses the old name does not fail. It returns an empty result, and an alert rule built on it never fires. We found exactly this in our own Kubernetes template while writing this post: the container CPU rule still used the 1.x metric name and the old pod_name label. It now uses the current names.

cAdvisor: what containers actually use​

cAdvisor is compiled into the kubelet, so there is nothing extra to deploy. Prometheus scrapes it from each node at /metrics/cadvisor. These are the metrics you reach for when a pod is slow or keeps getting killed:

• container_cpu_usage_seconds_total: CPU time consumed. Take a rate() to get cores in use.
• container_memory_working_set_bytes: memory the kernel cannot easily reclaim. The kubelet uses the working set for eviction decisions, and kubectl top shows the same number.
• container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total: how often the container hit its CPU limit and was paused. The CPUThrottlingHigh rule is the ratio of the two.

Two things trip people up:

1. Extra series. cAdvisor also reports cgroups that are not your application container: a pod-level series with container="" and, on older Docker-based nodes, the pause container as container="POD". Summing without a filter counts the same memory twice. Add container!="" (and container!="POD" on old nodes) when you aggregate.
2. Label names changed. Before Kubernetes 1.16, cAdvisor labelled series pod_name and container_name. They are now pod and container. A rule that groups by (pod_name) returns nothing on a current cluster.

Usage becomes useful when you compare it with a limit, and the limit lives in kube-state-metrics. Both sides share the namespace, pod and container labels, so the join is direct:

sum by (namespace, pod, container) (container_memory_working_set_bytes{container!=""})
/
sum by (namespace, pod, container) (kube_pod_container_resource_limits{resource="memory"})
> 0.9

node_exporter: the machine underneath​

node_exporter runs as a DaemonSet with access to the host's /proc and /sys. It knows nothing about pods. It reports the operating system: node_cpu_seconds_total by mode, node_memory_MemAvailable_bytes, node_filesystem_avail_bytes, disk I/O and network errors.

Use it for the questions Kubernetes does not answer. Is the node's root filesystem filling up? Is the clock drifting? Is a NIC dropping packets? Is the kernel OOM killer firing (node_vmstat_oom_kill)? Node-level memory includes page cache and every process on the host, so it will not add up to the sum of your containers. That is expected.

The Linux rule set covers the common host checks, including disk-fill prediction with predict_linear.

metrics-server: for autoscaling, not alerting​

metrics-server collects CPU and memory every 15 seconds from each kubelet's /metrics/resource endpoint and serves them through the Metrics API. The Horizontal Pod Autoscaler reads it, the Vertical Pod Autoscaler reads it, and kubectl top reads it.

It keeps only the latest value, in memory. There is no history to query and no Prometheus endpoint to scrape. The project's README says it plainly: metrics-server is meant for autoscaling, and you should not use it to forward metrics to monitoring solutions. If you want those numbers in your monitoring system, scrape the kubelet's /metrics/resource endpoint directly.

So when kubectl top and your dashboard disagree by a few percent, neither is wrong. They read different endpoints at different moments.

Which component answers your question​

Ready-made rules, sorted by source​

Most of the standard Kubernetes alerts read kube-state-metrics. Each of these pages explains what the alert means, its common causes and the steps to fix it, and lists the metrics it uses:

• kube-state-metrics: KubePodCrashLooping, KubePodNotReady, KubeDeploymentReplicasMismatch, KubeNodeNotReady, KubeJobFailed, KubeHpaMaxedOut
• kubelet: KubePersistentVolumeFillingUp
• cAdvisor: CPUThrottlingHigh
• API server: KubeAPIErrorBudgetBurn, KubeClientCertificateExpiration

The full Kubernetes rule set is available as a Prometheus rules file you can download and load with rule_files.

With Nightingale​

Nightingale does not replace any of these four components. It queries the store where their metrics already land, such as Prometheus or VictoriaMetrics, and runs alert rules against it. The Kubernetes integration ships the rules above, each with its runbook attached, and you can import them into a business group in one step.

Since v9, the Nightingale process is also an MCP server. Claude Code, Cursor or any other MCP client can connect to /mcp and ask which alerts are firing, which rules are noisiest, or what a metric did over the last hour.

The tool list was the easy part. The hard questions were about trust. Whose permissions does the AI client have? What can it change? How do you find out afterwards what it did? This post walks through the six decisions we made, and one caveat we haven't solved.

What this is not: an autonomous agent that resolves incidents on its own. It is a way for a model to read, and with explicit opt-in to change, an alerting system with the same permissions as the person who connected it.

1. No separate server: the endpoint lives in the process​

There is nothing extra to deploy. POST /mcp is served by the same n9e process that serves the UI and the API, and it is on by default.

It uses MCP's Streamable HTTP transport in stateless mode. Every POST is self-contained. The server returns an Mcp-Session-Id on initialize but never validates it, and GET /mcp returns 405 because there is no standalone event stream. The practical benefit: several Nightingale instances behind a load balancer need no sticky sessions. Any instance can answer any request.

2. A tool call is an internal API call​

When a client calls list_alert_rules, the MCP server does not read the database. It dispatches the call, in-process, onto Nightingale's own HTTP route (/api/n9e/busi-group/{gid}/alert-rules), carrying the caller's token through the full middleware chain: authenticate, resolve the user, check the permission point, check the business group.

So the answer to "can the AI get around RBAC?" is structural: no. It walks the same path as a click in the UI. There is no separate permission model for AI to configure, audit and keep in sync.

The consequence is that a client has exactly the permissions of its token's owner. To limit what the AI can see, you give it a limited account: a dedicated user with the Guest or Standard role, in a team that can read only the business groups you choose. list_busi_groups returns only groups that user can reach, so the model can only ever pick from that set. Do not give the AI account the Admin role. Admin bypasses the business-group layer.

OAuth tokens get one more restriction. A token issued through OAuth works on /mcp and /a2a only; the rest of the API rejects it.

3. Write tools are not registered, not just refused​

There are 74 tools. 42 read and 32 write. By default only the 42 read tools exist. The write tools sit in a separate registration list that is skipped unless you set:

[HTTP.A2A]
MCPEnableWriteTools = true

We chose "not registered" over "registered and refused" on purpose. A model plans with the tools it can see. If write tools appear in tools/list and fail at call time, the model keeps trying them, or tries to work around the error. If they are absent, the model never considers them.

This gate sits outside RBAC. Even if a client is handed root's token by mistake, under the default config it cannot change anything. Every tool also carries MCP's readOnlyHint annotation for clients that use it, and it always agrees with the registration list.

4. No delete tools at all​

None of the 74 tools deletes anything. "Write" means create and update. Deleting a rule, a user or a business group is not reachable through MCP.

This changes the worst case from "gone" to "changed wrongly", which is easier to notice and to fix. It does not make write tools harmless. Some of them are among the most powerful operations in the product:

5. Toolsets are an allowlist, and typos don't fall back​

The 74 tools are grouped into 13 toolsets: alerts, targets, datasource, mutes, busi_groups, notify_rules, alert_subscribes, event_pipelines, users, metrics, logs, dashboards and roles. An empty list exposes all of them; a list exposes only those named:

[HTTP.A2A]
MCPToolsets = ["alerts", "busi_groups", "targets", "metrics"]

A misspelled name is dropped with a warning in the log. It does not fall back to "everything". A list made entirely of typos exposes zero tools, not 74. Failing closed is less convenient and much safer.

Work backwards from what you want the AI to do:

6. Every call is attributable​

Two trails answer "who changed this, and was it the AI?":

• The object itself. A rule, mute or dashboard changed over MCP records the token owner's username in update_by, exactly as if they had edited it in the UI.

• The request log. Every /mcp request writes a start line with the full request body (tool name and arguments) and a done line with the user and status code, joined by a trace_id:

  grep '"method":"tools/call"' n9e.log | grep -oE '"name":"[a-z_]+"' | sort | uniq -c | sort -rn
  

  Oversized bodies are truncated, and the log line says so.

A dedicated account per client makes both trails useful. If every client shares one token, update_by tells you nothing.

The caveat: query_logs and SQL data sources​

One read-only tool deserves care. query_logs in the logs toolset passes the query body to Nightingale's log query endpoint, which dispatches on the data source type. If you have SQL data sources registered (MySQL, PostgreSQL, ClickHouse, Doris or TDengine), the SQL runs as written.

There is a read-only check on that path. It is a keyword blocklist that rejects statements containing INSERT, UPDATE, DELETE, DROP, ALTER, CREATE, TRUNCATE, REPLACE, GRANT or REVOKE. That stops the obvious cases. It is not a SQL parser, so treat it as a second line of defence rather than a guarantee. If you have SQL data sources, leave logs out of MCPToolsets, or give the AI account a role that cannot reach those data sources. Registering them with read-only database credentials is a sensible extra layer.

MCP or the built-in assistant?​

Nightingale also has a built-in assistant in the UI. For configuration changes it is the safer of the two: its update operations are two-phase. The first call computes the change and shows a confirmation card, and nothing is written until a person confirms. MCP has no such step. A write tool writes on its first call.

So a reasonable split is: use an MCP client for reading and triage, where its own model and tools shine, and use the built-in assistant when something needs to change.

Try it​

Create a token for a dedicated, least-privileged user under Profile → Token management, then add Nightingale to your MCP client:

{
  "mcpServers": {
    "nightingale": {
      "type": "http",
      "url": "http://127.0.0.1:17000/mcp",
      "headers": { "X-User-Token": "<token of the AI account>" }
    }
  }
}

Then ask something you can check: "Which alerts are firing right now? Group them by severity." or "List every business group I can see." The second one is also a quick test that the account is scoped the way you intended.

More detail is in the docs: Enable the MCP endpoint, Permission inheritance and RBAC, Read and write toolsets and Enable write tools safely.

Ask how to make Prometheus highly available and the usual answer is "run two of them". That is correct, and it is only half the job. Two replicas give you two copies of every alert and two slightly different copies of every time series. Each needs its own fix, and the fixes don't share a configuration.

This post separates the two problems: keeping alerts flowing exactly once, and keeping the data queryable without gaps. It ends with a checklist.

How Prometheus does HA​

Prometheus has no clustering. High availability means running two (or more) identical servers. Each one scrapes the same targets, evaluates the same rules and keeps its own local copy of the data. They don't talk to each other.

The copies are never identical. Each replica scrapes at its own moment, so the timestamps differ. When one replica restarts, it has a gap that the other doesn't. That is fine for alerting, and a problem for dashboards.

The usual convention is to give each replica an external label that says which one it is:

# prometheus-a.yml (prometheus-b.yml says replica: b)
global:
  external_labels:
    cluster: prod-eu
    replica: a

Problem 1: alerts should arrive once, even when a replica dies​

Both replicas evaluate every rule, so both send every alert. Deduplication happens in Alertmanager.

Cluster the Alertmanagers. Alertmanager instances gossip with each other. You start each one with --cluster.peer pointing at the others, and they share notification state and silences. When two replicas send the same alert, the cluster sends one notification.

Point every Prometheus at every Alertmanager. The Alertmanager README is explicit: do not load-balance traffic between Prometheus and its Alertmanagers. Give each Prometheus the full list, because every alert has to reach every instance:

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['am-0:9093', 'am-1:9093', 'am-2:9093']

Drop the replica label from alerts. Alerts carry the external labels. If replica A sends {alertname="X", replica="a"} and replica B sends {alertname="X", replica="b"}, Alertmanager sees two different alerts and notifies twice. Strip the label on the way out:

alerting:
  alert_relabel_configs:
    - action: labeldrop
      regex: replica

The Prometheus Operator does this for you with its own replica label. Hand-built setups often miss it, and the symptom is that every page arrives twice.

With these three in place, alerting HA works well. If replica A dies, replica B was evaluating the same rules all along, so there is no gap in detection. If an Alertmanager dies, Prometheus is still sending to the others.

Problem 2: dashboards and queries should not show gaps​

This is the half that "run two" does not solve. Point Grafana at replica A, and you see replica A's gaps. Put a load balancer in front of both, and consecutive refreshes can hit different replicas and draw slightly different lines.

The fix is a layer that knows the two replicas hold the same data and merges them. The three common options handle the replica label in different ways, and that difference matters:

So a setup moved from Thanos to VictoriaMetrics needs its replica label removed, and a setup moved the other way needs one added. Mixing the conventions gives you either doubled series or a store that silently keeps one replica.

Long-term retention is a separate decision from HA. Thanos, Mimir and VictoriaMetrics also keep data longer than a local Prometheus. But you can have deduplicated HA with short retention, or long retention with a single replica. Decide each one on its own.

Scaling is not HA​

Federation, functional sharding (one Prometheus per team or per cluster) and hashmod sharding spread load. They don't add redundancy: each shard is still a single point of failure until you run it as a pair. If you shard, every shard needs the replica treatment above.

Where a separate alerting engine changes the picture​

Everything above assumes Prometheus evaluates the rules. Some setups move evaluation out of Prometheus: Grafana Alerting, vmalert, Thanos Ruler, or Nightingale. Then storage HA and alerting HA become two different systems.

Nightingale takes a different approach from a Prometheus pair, with a different trade-off:

• It queries one deduplicated data source. Prometheus behind Thanos Querier, VictoriaMetrics or Mimir. The storage-side HA from Problem 2 is still needed, and Nightingale doesn't replace it.
• Rules are shared out, not duplicated. Several n9e instances share one MySQL and one Redis. Each live instance sends a heartbeat every second. The live instances form a consistent hash ring per data source, and each rule runs on exactly one instance. One evaluation means one event, so no Alertmanager-style deduplication is needed.
• The trade-off is a failover gap. An instance counts as dead after 30 seconds without a heartbeat, and only then does its share of rules move to another instance. A freshly started instance also waits 30 seconds before evaluating. A Prometheus pair has no such gap, because the other replica was evaluating all along.
• Shared dependencies need their own HA. If MySQL is down, everything is down. If Redis is unreachable at startup, the process exits. Running several n9e instances protects against losing an n9e process, and nothing else.

Neither design is better in general. Duplicate evaluation with downstream deduplication gives zero-gap failover and costs you a deduplication layer that must be configured exactly right. Single evaluation with a hash ring avoids duplicates by construction and costs you up to 30 seconds of blindness for the rules on a failed instance. The details are in High availability and failure domains.

Checklist​

Alerting

• Two or more Prometheus replicas with identical rule files.
• Alertmanager clustered with --cluster.peer; the cluster port reachable between instances.
• Every Prometheus lists every Alertmanager. No load balancer in between.
• The replica label dropped in alert_relabel_configs.
• Tested: stop one replica and one Alertmanager, fire a test alert, and get exactly one notification.

Storage and queries

• A deduplicating layer: Thanos Querier, Mimir's HA tracker or VictoriaMetrics deduplication.
• Replica labels set the way that layer expects. Thanos and Mimir need them to differ; VictoriaMetrics needs them identical.
• Dashboards and alert rules query the deduplicated endpoint, not a single replica.
• Tested: restart one replica and check that a dashboard over that window shows no gap.

Everything

• The monitoring system is monitored from outside itself. A dead-man's-switch alert that always fires, and pages when it stops arriving, catches the case where the whole alerting path is down.

The last item is the one most often skipped. Every check above runs inside the system it is checking.

At 3 a.m. an alert says CPU is at 92% on db-3. The on-call engineer opens the dashboards. Checkout is fast, errors are flat, and nobody is affected. The alert was true and useless.

A week later, checkout fails for 20 minutes because a dependency returns errors. CPU is fine, memory is fine, every host looks healthy. No alert fires.

Both failures have the same root cause: the rules watch causes, not symptoms. This post explains the difference, shows what symptom rules look like in PromQL, and explains where cause alerts still earn their place.

Symptoms and causes​

A symptom is what users experience: requests failing, pages loading slowly, a service unreachable, data arriving late. A cause is why: a saturated CPU, a full disk, a crashed pod, a slow query.

The idea comes from Google's SRE book. Chapter 6, "Monitoring Distributed Systems", argues that paging should be driven by symptoms, and proposes four golden signals to watch on every user-facing system: latency, traffic, errors and saturation. Rob Ewaschuk's essay "My Philosophy on Alerting", written while he was an SRE at Google, makes the same case in more detail.

Cause-based paging fails in two directions:

• Causes without symptoms create noise. High CPU, a pod restart, one replica out of three down. Systems absorb these all the time. Each page that turns out to be harmless teaches people to ignore the next one.
• Symptoms without a known cause create silence. You can only write cause rules for failures you have imagined. A new failure mode, a dependency outage or a bad deploy that leaves every resource metric normal goes unnoticed.

A symptom rule catches both the failures you predicted and the ones you didn't, because it watches the outcome instead of the mechanism.

What symptom rules look like​

The metric names below follow common conventions. Yours will depend on your instrumentation library, ingress controller or service mesh.

Errors: the share of requests that fail.

sum(rate(http_requests_total{job="checkout", code=~"5.."}[5m]))
  /
sum(rate(http_requests_total{job="checkout"}[5m]))
> 0.02

Use a ratio, not a count. 50 errors a minute is a disaster at 100 requests and noise at 100,000.

Latency: how slow the slow requests are.

histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket{job="checkout"}[5m]))
) > 1

Alert on a high percentile, not the average. An average of 200 ms can hide one request in fifty taking ten seconds.

Availability from outside. A probe that calls the service the way a user would catches DNS, TLS, load balancer and network failures that internal metrics never see. With the Prometheus blackbox exporter that is probe_success == 0. Our Linux rule set includes an "HTTP probe failed" rule for Categraf's HTTP checks.

Freshness and throughput for things that are not request-driven. Did the nightly job run? Is the queue draining?

time() - batch_job_last_success_timestamp_seconds{job="nightly-export"} > 26 * 3600

Three details that make symptom rules work​

1. Measure where users are. The ingress or load balancer sees every request, including the ones that never reached a healthy pod. Per-pod error rates miss requests that were dropped before reaching any pod.

2. Aggregate across instances. sum(...) over the whole service, not one alert per pod. Users don't care which replica failed them.

3. Guard against low traffic. At three requests a minute, one failure is a 33% error ratio. Add a minimum volume:

   (
     sum(rate(http_requests_total{job="checkout", code=~"5.."}[5m]))
       / sum(rate(http_requests_total{job="checkout"}[5m]))
   ) > 0.02
   and
   sum(rate(http_requests_total{job="checkout"}[5m])) > 1
   

Where cause alerts still belong​

"Alert on symptoms" doesn't mean "delete every cause alert". Some causes deserve a notification. They just shouldn't wake anyone up.

Deadlines. A disk that will be full in 24 hours, or a certificate that expires in 7 days, is a symptom waiting to happen, with a known date. Alert on these during working hours, at warning severity. The Linux library has a predict_linear rule for disks; the Kubernetes library has certificate expiry rules.

Lost redundancy. One replica of three is down. Users are fine, but the next failure will hurt them. That is worth a ticket today, not a page tonight.

The monitoring itself. If a scrape target or collector stops reporting, every rule that depends on it looks healthy. up == 0, or a dead-man's-switch alert that always fires, is a cause alert you need from day one.

Explanation. Everything else, such as CPU, memory, GC pauses and connection counts, belongs on a dashboard or as a low-severity alert that someone looks at while investigating a symptom. The Kubernetes rules in our library follow this: "Node CPU usage exceeds 90%" ships at info severity, while the symptom-shaped rules such as KubeAPIErrorBudgetBurn go up to critical.

Connecting symptoms to causes​

A symptom alert tells you that something is wrong. The responder then needs to know where to look. Two ways to close that gap:

• Put the causes in the runbook. Every symptom alert should link to a list of likely causes and the commands that confirm each one. Each page in our alert rule library follows that shape: what the alert means, common causes, then the steps to fix it. KubePodCrashLooping is a typical example.
• Mute symptoms while a known cause is active. If a whole site loses its uplink, every service there reports errors. Alertmanager's inhibit_rules, or an equivalent, can mute the dependent alerts while the upstream alert is firing. That needs a shared label such as site on both.

A worked example​

Here is a rule set for a checkout service built on these ideas:

Three rules can page, and all three describe something a customer would notice.

Auditing a rule set you already have​

For each rule that can page, ask: if this fires and users are fine, would anyone do anything? If the answer is no, lower its severity or move it to a dashboard.

Then look at the data. List the rules that fired most often in the last month and how many of those firings led to an action. Also list the rules that never fired in a year. They are usually broken rather than lucky: a renamed metric, a label that changed, a threshold that cannot be reached. Nightingale's built-in assistant can answer both questions directly ("which rules fired most often in the last 7 days, and which have never fired?"); see Common investigation workflows.

For the rules that stay, the rule design best practices page covers naming, labels and how much one rule should cover.

Moving from Zabbix to Prometheus is not a data migration. There is no export file that turns into rule files, and no tool that converts triggers into PromQL. The two systems model monitoring differently. Most of the work is translating concepts, then rebuilding a smaller set of rules on top of them.

This post maps each Zabbix concept to its Prometheus counterpart, lists the parts that have no equivalent, and suggests an order of work that keeps Zabbix running until the end.

The difference that explains everything else​

Zabbix is host-centric. You create a host, attach templates, and the templates define items (what to collect) and triggers (when to alert) for that host. 40 database servers with one template give you 40 copies of each trigger.

Prometheus is query-centric. Exporters expose metrics, Prometheus scrapes them, and an alert rule is a query. A rule matches every series the query returns. One rule covers all 40 servers, and the 41st server is covered as soon as it is scraped.

Most of the translation follows from this. You rarely port triggers one by one; you merge them.

Concept map​

Collection and storage​

Hosts, templates and discovery​

Triggers and alerting​

Notification​

The parts with no equivalent​

Plan for these before you start. They are where migrations stall.

1. Escalations. Zabbix can notify the on-call engineer, then the team lead after 30 minutes, then the manager. Alertmanager cannot. If you rely on escalation, you need an on-call tool behind Alertmanager.

2. Per-host thresholds. In Zabbix you override {$DISK_PFREE_MIN} on one host. In Prometheus a threshold is part of the query. The common workaround is a small metric that carries thresholds, joined in the rule:

   (node_filesystem_avail_bytes / node_filesystem_size_bytes) * 100
     < on(instance) group_left()
   disk_free_threshold_percent
   

   Or keep a default rule and split the exceptions out by label: disk_profile="large" gets its own rule.

3. SNMP traps. The SNMP exporter polls; it does not receive traps. Traps need a separate receiver that turns them into metrics or logs.

4. Text and log items. If Zabbix watches log files or string values, those move to a log system, not Prometheus.

5. Trigger dependencies, partly. inhibit_rules cover the common case of "switch down mutes everything behind it". They need a label that links the two alerts, such as a shared site or rack. Zabbix's dependency tree has no labels to rely on, so you add them first.

An order that keeps Zabbix running​

Zabbix stays up and keeps paging people until the very last step. Each step can be checked on its own.

1. Deploy exporters next to the Zabbix agents. Start with one group of hosts. Leave the agents installed. Check: the targets are up in Prometheus.
2. Pick the storage you will keep. A single Prometheus for a start; a pair and a long-term store if you need HA or longer retention. Decide now, because changing stores later is a project of its own.
3. Import existing rule sets instead of porting templates. The community rules for node_exporter, MySQL, Redis, Kubernetes and others cover most of what stock Zabbix templates do. Our alert rule library has them as downloadable rule files, each with a runbook.
4. Port the custom triggers, merged. List your custom triggers, group them by item and threshold, and write one rule per group. Use labels for the exceptions.
5. Send Prometheus alerts somewhere nobody is paged. A dedicated channel or mailbox. Compare it with Zabbix every day: fired only in Zabbix (missed rule or metric), fired only in Prometheus (threshold typo, or a Zabbix dependency or maintenance window you forgot), fired in both at different severity.
6. Move notification one channel at a time. Chat first, then email, then phone. Disable the matching Zabbix action as each channel moves, but keep Zabbix evaluating so the daily comparison continues.
7. Stop Zabbix last. After a week with no differences, stop the agents. Keep the database read-only for as long as you need the history. History does not migrate.

Zabbix or Prometheus at all?​

Not every Zabbix setup should move. Zabbix suits fleets of long-lived hosts and network devices, teams that want everything configured in one UI, and environments that need agents to push through firewalls. Prometheus suits environments where targets come and go (containers, autoscaling), where configuration lives in Git, and where software already exposes /metrics. Many organisations run both for years: Zabbix for network gear and legacy hosts, Prometheus for Kubernetes and new services.

With Nightingale​

Nightingale sits on top of Prometheus or VictoriaMetrics as the alerting layer. It manages rules in a UI with business groups and permissions, and handles notification, muting and subscriptions, so step 6 does not require learning Alertmanager's routing tree. It does not import Zabbix templates or triggers; the rebuild described above still applies.

Two Nightingale pages go deeper:

• From Zabbix: every Zabbix object mapped to a Nightingale one, including the ones with no equivalent.
• Coexistence and rollback plan: the daily comparison and channel-by-channel cutover in detail, with a rollback for each stage.

If you are still choosing, see Nightingale vs Zabbix.