Skip to main content

Monitor IPMI with Nightingale

Reads out-of-band temperature, fan, voltage, power and sensor state over IPMI, from the local BMC or remote ones.

Last reviewed

The IPMI input is ported from ipmi_exporter. It runs a set of FreeIPMI commands and turns the output into metrics: temperatures, fan speeds, voltages, power draw, the health of every sensor and the chassis power state. The data comes from the BMC, so it is still available when the operating system is down.

Collect locally on each server, or point one collector at a list of remote BMC addresses and credentials. IPMI itself must already be configured so that commands such as ipmimonitoring return output; otherwise there is nothing to convert.

What's in the template​

  • 3 dashboards, 31 panels in total
  • 4 alert rules, each with runbook steps
  • 19 metric descriptions
  • Categraf collector config: input.ipmi

How to connect​

  1. Install Categraf, put the config below in conf/input.ipmi/conf.toml, fill in your own address and credentials, and restart Categraf. The startup log prints input: local.ipmi started once the plugin is loaded.

    # Read metrics from the bare metal servers via freeipmi
    [[instances]]
    # target指定是本地采集还是远程采集
    #target="localhost"
    # 指定采集的用户名和密码，这里务必保证ipmi命令能获取正确输出，不是网上查到一个用户名 密码就可以。
    #user = "user"
    #pass = "1234"

    # ipmi协议版本，支持1.5 和 2.0
    #driver = "LAN_2_0"

    # 指定特权用户名
    #privilege = "user"

    ## session-timeout, ms
    #timeout = 100000

    # 支持的采集器  bmc, bmc-watchdog, ipmi, chassis, dcmi, sel，sm-lan-mode
    # 默认使用 bmc, ipmi, chassis和dcmi，建议保持下列配置便于仪表盘更好的展示
    collectors = [ "bmc", "ipmi", "chassis", "sel", "dcmi"]

    # 不关注的传感器，指定id 排除掉
    #exclude_sensor_ids = [ 2, 29, 32, 50, 52, 55 ]

    # 如果你想使用定制化的参数覆盖内置的命令，可以修改以下内容； 建议保持注释
    #[instances.collector_cmd]
    #ipmi = "sudo"
    #sel = "sudo"
    #[instances.default_args]
    #ipmi = [ "--bridge-sensors" ]
    #[instances.custom_args]
    #ipmi = [ "--bridge-sensors" ]
    #sel = [ "ipmi-sel" ]

  2. Open the integration center in Nightingale, find IPMI, and import the dashboards and alert rules into a business group.

  3. Imported rules start disabled. Check that the metrics are arriving first, then enable the rules one by one with thresholds that fit your environment.

Bundled dashboards​

DashboardPanels
IPMI12
IPMI by Categraf12
IPMI for Prometheus7

Bundled alert rules​

4 rules. Each lists its trigger condition (PromQL) and the steps to take when it fires. Thresholds are meant to be adjusted after import.

CPU temperature exceeds 90​

S2 Warning · for 1 min · Trigger:

ipmi_cpu1_temp{} > 90 or ipmi_cpu2_temp{} > 90

Runbook:

  1. Run ipmitool sdr type temperature to see the actual readings and thresholds of each temperature sensor;
  2. Check that all chassis fans are spinning (ipmitool sdr type fan) and that filters and heatsinks are not clogged with dust;
  3. Confirm the room air conditioning and cold aisle temperature are normal and whether other machines in the same rack are also hot;
  4. Short term, reduce load on this machine or move critical workloads off to buy time for a dust-clean or fan replacement.

Abnormal CPU voltage​

S2 Warning · for 1 min · Trigger:

count by (agent_hostname, server) ({__name__=~"ipmi_vcpu[0-9]+", status_code=~"lcr|ucr"}) > 0

Runbook:

  1. Run ipmitool sdr type voltage to see each voltage sensor's reading against its upper and lower thresholds and which way it deviates;
  2. Check the power supply status (ipmitool sdr type 'Power Supply'); on dual-PSU machines confirm whether one feed has failed;
  3. Check the power line and PDU output stability, and whether other machines in the same rack are alarming at the same time;
  4. If a PSU is confirmed faulty, schedule a replacement and move critical workloads off meanwhile.

Abnormal fan speed​

S2 Warning · for 1 min · Trigger:

count by (agent_hostname, server) ({__name__=~"ipmi_fan[0-9]+", status_code=~"lcr|lnc|unc|ucr"}) > 0

Runbook:

  1. Run ipmitool sdr type fan to see each fan's speed and thresholds and identify the faulty one;
  2. A fan reading 0 or far below its peers is almost certainly failed — schedule a replacement;
  3. If every fan is running fast, intake temperature or load is too high: check room temperature, dust in the filters and any airflow obstruction;
  4. Before swapping fans, run ipmitool sdr type temperature to see whether temperatures have already risen, and move workloads off if needed.

Motherboard temperature exceeds 90​

S2 Warning · for 1 min · Trigger:

ipmi_system_temp > 90 or ipmi_pch_temp > 90

Runbook:

  1. Run ipmitool sdr type temperature to identify which sensor is hot and what its threshold is;
  2. Check the chassis fans and airflow, especially whether cabling is blocking intake around the motherboard;
  3. Confirm the room air conditioning and in-rack temperature, and whether the machine is wedged into an enclosed spot;
  4. Clean the dust, tidy the cabling and re-measure; if it still does not drop, contact the vendor to inspect the heatsink assembly.

Metric reference​

19 metrics, the ones the dashboards and rules above are built on.

MetricMeaningUnit
ipmi_bmc_infoIPMI BMC Info — A constant metric containing information such as BMC firmware revision, manufacturer ID, and system firmware version.
ipmi_chassis_cooling_fault_stateIPMI Chassis Cooling Fault State — Indicates the chassis cooling/fan fault state, 1 for no fault, 0 for fault.
ipmi_chassis_drive_fault_stateIPMI Chassis Drive Fault State — Indicates the drive fault state, 1 for no fault, 0 for fault (may be reversed in native Go implementation).
ipmi_chassis_power_stateIPMI Chassis Power State — Indicates the chassis power state, 1 for power on, 0 for power off.
ipmi_current_amperesIPMI Current (Amperes) — Current measured by current sensors, in Amperes.A
ipmi_current_stateIPMI Current Sensor State — State of current sensors, 0 for nominal, 1 for warning, 2 for critical, NaN for N/A.
ipmi_dcmi_power_consumption_wattsIPMI DCMI Power Consumption (Watts) — Current power consumption collected via DCMI, in Watts.W
ipmi_fan_speed_rpmIPMI Fan Speed (RPM) — Speed measured by fan speed sensors, in rotations per minute.RPM
ipmi_fan_speed_stateIPMI Fan Speed Sensor State — State of fan speed sensors, 0 for nominal, 1 for warning, 2 for critical, NaN for N/A.
ipmi_power_stateIPMI Power Sensor State — State of power sensors, 0 for nominal, 1 for warning, 2 for critical, NaN for N/A.
ipmi_scrape_duration_secondsIPMI Scrape Duration (Seconds) — The time taken to retrieve IPMI data, in seconds.s
ipmi_sel_free_space_bytesIPMI SEL Free Space (Bytes) — Free space remaining for new entries in the System Event Log (SEL), in bytes.bytes
ipmi_sel_logs_countIPMI SEL Logs Count — Total number of event log entries in the System Event Log (SEL).
ipmi_sensor_stateIPMI Sensor State — State of generic unclassified sensors, 0 for nominal, 1 for warning, 2 for critical, NaN for N/A.
ipmi_temperature_celsiusIPMI Temperature (Celsius) — Temperature measured by temperature sensors, in degrees Celsius.°C
ipmi_temperature_stateIPMI Temperature Sensor State — State of temperature sensors, 0 for nominal, 1 for warning, 2 for critical, NaN for N/A.
ipmi_upIPMI Collector Status — Indicates if the specified collector successfully retrieved data from the remote host, 1 for success, 0 for failure.
ipmi_upIPMI Collection Status — Scrape metadata metric. 1 indicates successful collection, 0 indicates failure.
ipmi_voltage_stateIPMI Voltage Sensor State — State of voltage sensors, 0 for nominal, 1 for warning, 2 for critical, NaN for N/A.

On this page