Skip to main content

PrometheusBadConfig alert: what it means and how to fix it

What the PrometheusBadConfig Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

The last attempt to reload the Prometheus configuration failed. Prometheus keeps running with the previous configuration, so nothing breaks at once. But every change made since then (new scrape targets, new rule files, new Alertmanager settings) is not in effect, and the next restart may not come back up if the file stays broken.

The rule​

# Without max_over_time, failed scrapes could create false negatives, see
# https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.
max_over_time(prometheus_config_last_reload_successful{job="prometheus-k8s",namespace="monitoring"}[5m]) == 0

Alert message: Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed to reload its configuration.

Common causes​

  • A YAML syntax error or an unknown field in prometheus.yml.
  • A rule file listed in rule_files that fails to parse.
  • A referenced file (TLS certificate, password file, file_sd target list) that is missing or unreadable.
  • On Kubernetes, a ServiceMonitor, PodMonitor or PrometheusRule that the Operator turned into an invalid configuration.

How to fix it​

  1. Find the config parse error line in kubectl logs or the instance log;
  2. Validate locally with promtool check config prometheus.yml;
  3. On Kubernetes the config is usually generated by the Operator — check ServiceMonitor/PrometheusRule for invalid fields;
  4. After fixing, confirm /-/reload succeeded and the /config page shows the new configuration.

Notes​

  • Run promtool check config prometheus.yml before each reload. With the Operator the generated configuration lives in a Secret; the config-reloader sidecar logs show the reload error.
  • The expression selects job="prometheus-k8s", namespace="monitoring". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • prometheus_config_last_reload_successful (Prometheus's own metrics)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Prometheus rule at once as prometheus.yml.

- alert: PrometheusBadConfig
  expr: |-
    # Without max_over_time, failed scrapes could create false negatives, see
    # https://www.robustperception.io/alerting-on-gauges-in-prometheus-2-0 for details.
    max_over_time(prometheus_config_last_reload_successful{job="prometheus-k8s",namespace="monitoring"}[5m]) == 0
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus {{$labels.namespace}}/{{$labels.pod}} has failed to reload its configuration."
    runbook_url: "https://n9e.github.io/alert-rules/prometheus/prometheusbadconfig"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kube-prometheus (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page