Skip to main content

KubeJobFailed alert: what it means and how to fix it

What the KubeJobFailed Prometheus alert means, its common causes, the PromQL rule and the runbook steps to fix it, with a ready-to-use rule in YAML.

Last reviewed

A Job has been marked Failed (kube_job_failed > 0): it used up its backoffLimit (6 retries by default) or ran past activeDeadlineSeconds, and the batch work it was meant to do (a backup, a report, a data sync) did not happen.

The rule​

kube_job_failed{job="kube-state-metrics"}  > 0

Alert message: Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert.

Common causes​

  • The task itself errors out: bad input data, a failed query, a dependency that is down.
  • Pods are killed for exceeding their memory limit, or evicted from a node under pressure.
  • The Job ran longer than activeDeadlineSeconds.
  • The container image or command changed and no longer works with the Job arguments.

How to fix it​

  1. Run kubectl get job <job> -n <ns> for the failure count, then kubectl logs job/<job> for the reason;
  2. Run kubectl describe job for Events and whether it gave up after exhausting backoffLimit;
  3. After fixing the root cause, kubectl delete job and resubmit, or let the CronJob run again on schedule;
  4. Confirm whether the failure has downstream impact (missing output, skipped cleanup) and redo it manually if needed.

Notes​

  • The alert keeps firing for as long as the failed Job object exists, even after a later run succeeds. Delete the Job once you have looked at it. For CronJobs, failedJobsHistoryLimit (1 by default) controls how many failed Jobs are kept.
  • The expression selects job="kube-state-metrics". If your scrape job or namespace is named differently, change these matchers or the rule never matches.

Metrics it uses​

  • kube_job_failed (kube-state-metrics)

Prometheus rule YAML​

Add this to a rules file loaded through rule_files, or download every Kubernetes rule at once as kubernetes.yml.

- alert: KubeJobFailed
  expr: |-
    kube_job_failed{job="kube-state-metrics"}  > 0
  for: 15m
  labels:
    severity: warning
  annotations:
    summary: "Job {{ $labels.namespace }}/{{ $labels.job_name }} failed to complete. Removing failed job after investigation should clear this alert."
    runbook_url: "https://n9e.github.io/alert-rules/kubernetes/kubejobfailed"

With Nightingale: this rule ships in the Kubernetes integration template; import it into a business group and Nightingale attaches the runbook to every alert it sends.

Sources and license​

The rule expression is adapted from kubernetes-mixin (Apache-2.0). The runbook steps come from Nightingale's integration templates (Apache-2.0); the explanation and causes are written for this page.

On this page