Skip to main content

PrometheusOperatorRejectedResources alert: what it means and how to fix it

What the PrometheusOperatorRejectedResources Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

The Prometheus Operator rejected one or more monitoring resources (ServiceMonitor, PodMonitor, Probe, PrometheusRule, AlertmanagerConfig) and left them out of the generated configuration. The resources still exist in the cluster and kubectl apply succeeded, but the targets they describe are not scraped and the rules they contain are not loaded. Nothing else tells you this has happened.

The rule​

min_over_time(prometheus_operator_managed_resources{state="rejected",job="prometheus-operator",namespace="monitoring"}[5m]) > 0

Alert message: Prometheus operator in {{ $labels.namespace }} namespace rejected {{ printf "%0.0f" $value }} {{ $labels.controller }}/{{ $labels.resource }} resources.

Common causes​

  • A PrometheusRule contains a rule that does not parse, such as invalid PromQL or a bad duration.
  • A ServiceMonitor or PodMonitor reads a file from the Prometheus filesystem (bearerTokenFile, TLS file paths) while the Operator is configured to forbid it.
  • A resource references a Secret or ConfigMap that does not exist or is in a namespace the Operator cannot read.
  • An AlertmanagerConfig uses a receiver or field that does not pass the Operator's validation.

How to fix it​

  1. Run kubectl logs -n <ns> deploy/prometheus-operator for the specific error;
  2. For permission errors, verify the Operator's ClusterRole covers the required resources;
  3. For Reconcile/Sync failures, check the corresponding CRD objects (kubectl describe servicemonitor/prometheusrule);
  4. For rejected resources, kubectl describe shows the rejection reason, usually failed field validation.

Notes​

  • The controller and resource labels on the alert tell you which kind of object was rejected. The Operator logs name the object and the reason.
  • The expression selects job="prometheus-operator", namespace="monitoring". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • prometheus_operator_managed_resources (Prometheus Operator)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Prometheus rule at once as prometheus.yml.

- alert: PrometheusOperatorRejectedResources
  expr: |-
    min_over_time(prometheus_operator_managed_resources{state="rejected",job="prometheus-operator",namespace="monitoring"}[5m]) > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus operator in {{ $labels.namespace }} namespace rejected {{ printf \"%0.0f\" $value }} {{ $labels.controller }}/{{ $labels.resource }} resources."
    runbook_url: "https://n9e.github.io/alert-rules/prometheus/prometheusoperatorrejectedresources"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kube-prometheus (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page