Alert rules
Rules are configured in the UI, scoped by business group, graded S1 / S2 / S3, and can span several data sources. A library of common rules ships with the install.
Create the first rule
A rule from scratch: pick the source, write the query, set severity, test fire it, and watch the event appear.
Metric rules
Metric rules put the threshold inside the PromQL expression; one rule can carry several severities, and the labels the query returns land on the event for routing.
Log rules
Alert on log counts or matched patterns from ElasticSearch, OpenSearch, Loki and VictoriaLogs — all four alert, but only three can be previewed in the Log explorer.
SQL-backed rules
Rules that run a SQL statement against MySQL, PostgreSQL, ClickHouse, Doris, TDengine or IoTDB on a schedule.
Scope by business group and data source
A rule has two scopes: the business group decides who may edit it and owns its events; the data source list decides which instances it queries, and can span several.
Labels, annotations and severity
Attach labels for routing, write annotations that read well in a notification, and choose S1 / S2 / S3 deliberately.
Evaluation interval and recovery
Four timing fields on a rule — interval, for-duration, observation window, recovery — decide when an event is created, when it recovers and when it re-notifies.
Test fire a rule
Fire a rule on demand to check the query, labels and the rendered notification before it goes live.
Inspect evaluation execution records
Every evaluation leaves a record: see what the query returned and why an event did or did not fire.
Rule templates and built-in rules
Nightingale ships rule templates for MySQL, Redis, Kubernetes and sixty other components; import them into any business group, and save your own rules as templates.
Import, export and reuse
Move rules between environments as JSON, and keep them in version control.
Rule design best practices
Opinions on rule design: alert only on what needs a person, leave trends to dashboards, size a rule to one class of object, and review so the rule set does not rot.