Skip to main content

KubeAggregatedAPIDown alert: what it means and how to fix it

What the KubeAggregatedAPIDown Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

An aggregated API, an APIService that the Kubernetes API server proxies to another service, has been available less than 85% of the time over the last 10 minutes. The most common one is v1beta1.metrics.k8s.io from metrics-server; others come from prometheus-adapter, custom metrics adapters and various operators.

The rule​

(1 - max by(name, namespace, cluster)(avg_over_time(aggregator_unavailable_apiservice{job="apiserver"}[10m]))) * 100 < 85

Alert message: Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m.

Common causes​

  • The backing Pods (metrics-server, prometheus-adapter) are crashing, not ready, or scaled to zero.
  • The API server cannot reach the backing service: a network policy, a firewall, or a broken Service or endpoints.
  • TLS problems between the API server and the backend, such as an expired serving certificate or a wrong caBundle on the APIService.
  • The APIService still exists but points to a service that was uninstalled.

How to fix it​

  1. Run kubectl get apiservice to identify the APIService;
  2. Run kubectl -n <ns> get pod to check its backing Pods;
  3. Read that Pod's logs for startup or runtime errors;
  4. After recovery, verify with something like kubectl top node that the functionality is back.

Notes​

  • Effects depend on the API. When the metrics API is down, kubectl top fails and HPAs that scale on CPU or memory stop scaling. A broken APIService can also stall namespace deletion, because the namespace controller cannot list resources from that group.
  • The expression selects job="apiserver". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • aggregator_unavailable_apiservice (kube-apiserver)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Kubernetes rule at once as kubernetes.yml.

- alert: KubeAggregatedAPIDown
  expr: |-
    (1 - max by(name, namespace, cluster)(avg_over_time(aggregator_unavailable_apiservice{job="apiserver"}[10m]))) * 100 < 85
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Kubernetes aggregated API {{ $labels.name }}/{{ $labels.namespace }} has been only {{ $value | humanize }}% available over the last 10m."
    runbook_url: "https://n9e.github.io/alert-rules/kubernetes/kubeaggregatedapidown"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kubernetes-mixin (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page