Skip to main content

KubeClientCertificateExpiration alert: what it means and how to fix it

What the KubeClientCertificateExpiration Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

Of the client certificates presented to the API server over the last 5 minutes, at least 1% expire within 7 days (warning) or within 24 hours (critical). The rules take the 1st percentile of their remaining lifetime. When they expire, those clients get 401 Unauthorized. If they belong to a kubelet, a controller or the scheduler, nodes go NotReady or the control plane stops reconciling.

The rule​

The template splits this alert into 2 rules with different thresholds, windows and severities. Load all of them together.

KubeClientCertificateExpiration-S2​

Warning (warning) · Fires after: —

apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 604800

KubeClientCertificateExpiration-S1​

Critical (critical) · Fires after: —

apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 86400

Alert message: A client certificate used to authenticate to the apiserver is expiring in less than 7.0 days.

Common causes​

  • kubeadm issues control-plane certificates for one year and renews them during kubeadm upgrade. A cluster that has not been upgraded for a year runs into this.
  • Kubelet client certificate rotation is off, or the CSRs it creates are not being approved.
  • Long-lived kubeconfig files issued to people or CI systems with client certificates.
  • Certificates generated by hand or by an external tool that has no renewal job.

How to fix it​

  1. Run kubeadm certs check-expiration (kubeadm clusters) or openssl x509 -noout -dates to confirm the expiry;
  2. On kubeadm clusters run kubeadm certs renew all and restart the control plane components;
  3. kubelet client certificates usually rotate automatically — if not, check the rotateCertificates setting and CSR approval;
  4. After renewal, verify the components loaded the new certificate (restart them) and re-check every node.

Notes​

  • The metric is built from the certificates clients actually present, so it only reflects clients that have connected recently. It does not tell you which client is affected. On kubeadm clusters kubeadm certs check-expiration lists the control plane's own certificates.
  • The expression selects job="apiserver". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • apiserver_client_certificate_expiration_seconds_count (kube-apiserver)
  • apiserver_client_certificate_expiration_seconds_bucket (kube-apiserver)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Kubernetes rule at once as kubernetes.yml.

- alert: KubeClientCertificateExpiration-S2
  expr: |-
    apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 604800
  labels:
    severity: warning
  annotations:
    summary: "A client certificate used to authenticate to the apiserver is expiring in less than 7.0 days."
    runbook_url: "https://n9e.github.io/alert-rules/kubernetes/kubeclientcertificateexpiration"
- alert: KubeClientCertificateExpiration-S1
  expr: |-
    apiserver_client_certificate_expiration_seconds_count{job="apiserver"} > 0 and on(job) histogram_quantile(0.01, sum by (job, le) (rate(apiserver_client_certificate_expiration_seconds_bucket{job="apiserver"}[5m]))) < 86400
  labels:
    severity: critical
  annotations:
    summary: "A client certificate used to authenticate to the apiserver is expiring in less than 24.0 hours."
    runbook_url: "https://n9e.github.io/alert-rules/kubernetes/kubeclientcertificateexpiration"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kubernetes-mixin (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page