Skip to main content

Monitor NSQ with Nightingale

Reads nsqd's /stats endpoint for per-topic and per-channel depth, in-flight, timeouts and client counts; rules watch growing backlog and lost consumers.

Last reviewed

The NSQ input is ported from Telegraf and reads the /stats HTTP endpoint of nsqd: message counts and depth per topic, depth, in-flight, deferred, requeue and timeout counters per channel, and ready and in-flight state per client connection.

The rules cover channel depth growing steadily, rising timeouts and requeues, channels without consumers and nsqd being unreachable.

What's in the template​

  • 5 alert rules, each with runbook steps
  • Categraf collector config: input.nsq

How to connect​

  1. Install Categraf, put the config below in conf/input.nsq/nsq.toml, fill in your own address and credentials, and restart Categraf. The startup log prints input: local.nsq started once the plugin is loaded.

    # # collect interval
    # interval = 15

    # [[instances]]
    ## The Nsq API URI used to collect statistical information.
    # targets = ["http://localhost:4151"]
     
    # headers={Authorization="", X-Forwarded-For="", Host=""}
     
    # timeout="5s"

    # # basic auth
    # username=""
    # password=""

    ## append some labels for series
    # labels = { product="nsq" }

    ## interval = global.interval * interval_times
    # interval_times = 1

  2. Open the integration center in Nightingale, find NSQ, and import the dashboards and alert rules into a business group.

  3. Imported rules start disabled. Check that the metrics are arriving first, then enable the rules one by one with thresholds that fit your environment.

Bundled alert rules​

5 rules. Each lists its trigger condition (PromQL) and the steps to take when it fires. Thresholds are meant to be adjusted after import.

NSQ channel backlog​

S2 Warning · for 5 min · Trigger:

nsq_channel_depth > 10000

Runbook:

  1. Take topic and channel from the alert labels and run curl -s http://<nsqd>:4151/stats?format=json to inspect client_count and in_flight_count for that channel;
  2. If client_count looks normal but in_flight stays at its ceiling, individual messages are slow to process; go to the consumer application logs and tracing to find the slow step;
  3. If the processing logic itself is slow, scale out consumer instances or raise max_in_flight on the consumer;
  4. If the backlog came from a one-off traffic spike, you may run curl -X POST 'http://<nsqd>:4151/channel/empty?topic=..&channel=..' after evaluating the impact, but note this drops messages and must be confirmed with the business first.

NSQ channel overflowing to disk​

S2 Warning · for 5 min · Trigger:

nsq_channel_backend_depth > 0

Runbook:

  1. Run curl -s http://<nsqd>:4151/stats?format=json and compare depth with backend_depth to size the spill;
  2. The root cause is usually still slow consumption, so scale out consumers or optimise consumption logic as for a backlog;
  3. Check the free space on the disk holding the nsqd data directory (--data-path) with df -h so it cannot fill up;
  4. For an expected short spike you can raise --mem-queue-size to buy memory buffering, but re-check the nsqd process memory ceiling at the same time.

NSQ channel has no consumer but has a backlog​

S2 Warning · for 3 min · Trigger:

nsq_channel_client_count == 0 and nsq_channel_depth > 0

Runbook:

  1. Take topic and channel from the alert labels and confirm whether the consuming service has been fully taken offline or has crashed;
  2. If the consumer is running, check its connections to nsqlookupd/nsqd (netstat -anp | grep 4150) and look for connection errors in the consumer logs;
  3. If a release changed the channel name and left the old channel orphaned, confirm it is unused and delete it with curl -X POST 'http://<nsqd>:4151/channel/delete?topic=..&channel=..';
  4. If the consumer does need to come back, start it and watch depth begin to fall.

NSQ message processing timeouts spiking​

S2 Warning · for 5 min · Trigger:

rate(nsq_channel_timeout_count[5m]) > 1

Runbook:

  1. Take topic and channel from the alert labels and look in the consumer application logs for handlers exceeding msg-timeout;
  2. If processing genuinely needs longer, call Touch() periodically on long tasks in the consumer, or raise --msg-timeout on nsqd;
  3. If a slow downstream dependency (database, third-party API) is dragging processing down, fix the dependency first and add timeouts and circuit breaking in the consumer;
  4. Make sure the business logic is idempotent, otherwise redelivery will cause duplicate writes.

NSQ message requeues spiking​

S3 Info · for 5 min · Trigger:

rate(nsq_channel_requeue_count[5m]) > 10

Runbook:

  1. Aggregate the failure reasons from the consumer logs and separate business validation failures (which should not be retried) from dependency flapping (which should);
  2. Validation failures should be FIN-ed and written to a dead-letter store rather than requeued forever;
  3. For dependency flapping, check that retry backoff is configured sensibly, since fixed-interval retries create storms;
  4. Read this together with nsq_channel_timeout_count: if both are high the consumer is simply overloaded and needs to be scaled out first.

On this page