Prerequisites for using Nightingale
Learn logs, processes, time-series concepts, PromQL and the data path before diagnosing collection, storage, alerting or delivery.
Last reviewed
After this page, you will know the prerequisites for Nightingale and be able to separate missing data, missing alerts and missing notifications into specific checks.
Linux and deployment basics
For binaries, understand working directories, configuration files, stdout / stderr and systemd.
For containers, understand port mappings, volumes, logs and where a container's 127.0.0.1 points.
sudo journalctl -u n9e.service -n 100 --no-pager
sudo journalctl -u categraf.service -n 100 --no-pager
Use the actual service names. Expected result: recent logs for the component under investigation. If no such unit exists, use the log entry point for its real deployment.
Time-series model
- A metric name describes the measurement.
- Labels distinguish hosts, services and instances; the name and labels identify a series.
- A timestamp and value form a sample. Incorrect clocks place data outside the intended range.
- A Gauge is a current value; a Counter accumulates and may reset when a process restarts.
- Histograms and summaries require understanding buckets, counts and sums.
Avoid unique request IDs as labels. See the Prometheus data model.
PromQL
Start with selecting series by labels, using rate() on counters and aggregating instances.
Build thresholds around the meaning of the data; a cumulative count is not a requests-per-second value.
An instant query evaluates a time; a range query returns a series over an interval. Range and step affect the display. Verify an expression in the metrics explorer before putting it into a metric rule. See the Prometheus query documentation.
Pull and push
Prometheus and vmagent scrape with Pull. The scraper knows whether a request succeeded and can generate up.
Categraf sends Remote Write with Push. Its liveness cannot be treated as a Prometheus scrape-status check.
Nightingale v9 can use embedded or external storage. If metrics already exist in Prometheus, register the datasource without moving them. Heartbeat metadata and sample delivery are separate paths; see Linux monitoring.
Troubleshooting and asking for help
Find the failing stage first:
- Does the collector produce samples locally?
- Do they reach the correct store?
- Does Nightingale query that datasource?
- Does rule evaluation produce an event?
- Is the event muted, filtered or processed?
- Is notification delivery successful?
Include versions, deployment method, expected and actual behavior, the incident time range, minimal configuration and relevant logs. Remove credentials and personal information first. Keep enough context to make the failure diagnosable.