How long should a Prometheus alert wait? Data from 692 rules
· 9 min read
Every alert rule answers two questions: what counts as bad, and how long it has to stay bad. Most review effort goes into the first one, the expression and the threshold. The second is the for duration, and it often gets whatever value the tool filled in.
We wanted to know how often that happens, so we counted. Nightingale ships alert rule templates for 49 integrations. 692 rules, and 304 of them (44%) have a for duration of exactly 60 seconds, which is the default in Nightingale's rule editor. This post walks through what the numbers show and what for value suits which kind of alert.