# Backup and restore

Only the metadata database and the etc/ directory are irreplaceable: back both up on a schedule and rehearse the restore; embedded TSDB metrics can be re-collected.

Last reviewed 2026-10-10

Only two things in Nightingale are genuinely irreplaceable: the **metadata database** and the `etc/` directory. Metrics can be collected again and logs can be thrown away, but losing configuration and rules means rebuilding them by hand, one at a time. This page covers what to back up, how, and how to prove the backup actually works.

## What to back up[​](#what-to-back-up "Direct link to What to back up")

| Content                     | Where                                        | Cost of not having it                                                      | Suggested frequency                          |
| --------------------------- | -------------------------------------------- | -------------------------------------------------------------------------- | -------------------------------------------- |
| Metadata database           | MySQL / PostgreSQL, or `n9e.db`              | Users, business groups, rules, dashboards and notification config all gone | Daily, keep 7–30 days                        |
| Configuration               | The `etc/` directory                         | Every setting has to be worked out again                                   | On every change (keep it in version control) |
| Integration templates       | The `integrations/` directory                | Custom dashboard and rule templates lost                                   | Same                                         |
| Embedded TSDB               | `[EmbeddedTSDB] Dir`, `data/tsdb` by default | Historical metrics lost                                                    | Depends how much you rely on it              |
| Logs and evaluation records | `[Log] Dir`                                  | No consequence                                                             | **Do not back up**                           |

Evaluation records are a troubleshooting aid: they have their own retention ceiling, are cleaned up automatically, and are regenerated by every subsequent evaluation. There is nothing to preserve.

## The metadata database[​](#the-metadata-database "Direct link to The metadata database")

Almost all the value is here.

```
# MySQL

mysqldump --single-transaction --routines --triggers \

  -h mysql -u n9e -p n9e_v6 | gzip > n9e_v6-$(date +%F).sql.gz



# PostgreSQL

pg_dump -h postgres -U n9e -Fc n9e_v6 > n9e_v6-$(date +%F).dump
```

`--single-transaction` dumps from a consistent snapshot without locking tables.

**SQLite only turns up in test environments**, and it has a backup trap of its own: the data is not only in `n9e.db` but also in `n9e.db-wal` and `n9e.db-shm` beside it. **All three must be copied and restored together** — taking only `n9e.db` gives you a database paired with a stale WAL. Safer still, stop the process before copying.

## Configuration and integration templates[​](#configuration-and-integration-templates "Direct link to Configuration and integration templates")

```
tar czf n9e-etc-$(date +%F).tar.gz etc/ integrations/
```

`etc/` holds `config.toml`, the edge's `etc/edge/edge.toml`, `metrics.yaml` and the notification scripts. Better yet, **keep `etc/` in version control**: you get a record of what changed, when and by whom, and rolling back becomes one `git checkout`.

`config.toml` contains the database password, so move credentials out before committing it — see [Secret management](https://n9e.github.io/docs/security/secrets.md).

## The embedded TSDB[​](#the-embedded-tsdb "Direct link to The embedded TSDB")

The embedded store has **no online snapshot endpoint**. Keeping a copy means stopping the process and copying the directory:

```
systemctl stop n9e

tar czf n9e-tsdb-$(date +%F).tar.gz data/tsdb/

systemctl start n9e
```

Copying it while running captures half-written blocks, which may not open on restore.

Be pragmatic: if the embedded store holds "the last 15 days for looking at trends", a backup that costs downtime is not worth it — losing it is acceptable. If it is your only metric store and you cannot afford to lose it, the right answer is not to back it up but to **move to an external store** — see [Embedded TSDB single-Center limits](https://n9e.github.io/docs/operations/embedded-tsdb-limits.md).

## The restore procedure[​](#the-restore-procedure "Direct link to The restore procedure")

Follow the order; do not skip steps.

1. **Stop every `n9e` process.** All of them in the cluster, `n9e-edge` included. A surviving instance writes heartbeats and events into the database you are restoring.

2. **Restore the database.**

   ```
   gunzip < n9e_v6-2026-09-01.sql.gz | mysql -h mysql -u n9e -p n9e_v6
   ```

   **Expected result**: `select count(*) from alert_rule;` matches the count at backup time.

3. **Restore `etc/` and `integrations/`.** Confirm `[DB] DSN` points at the database you just restored.

4. **Restore the embedded TSDB** if you backed it up and still use it: copy `data/tsdb` back.

5. **Start one instance.** Not all of them at once. **Expected result**: `curl --noproxy '*' http://n9e:17000/ping` returns `pong`, and after logging in the rule list, dashboards and notification config are all there.

6. **Confirm evaluation resumed.** `curl --noproxy '*' http://n9e:17000/metrics | grep n9e_alert_rule_eval_total`. **Expected result**: the count starts climbing 30 seconds after the process starts — there is a 30-second evaluation delay after boot.

7. **Start the remaining instances.** Check that they all appear under **System → Alerting engines**.

Redis **does not need restoring**. It holds sessions, target heartbeat timestamps and host metadata, all of which expire; starting empty is fine. The cost is that everyone has to log in again, and target heartbeat times stay blank until the collectors next report.

## Rehearsing[​](#rehearsing "Direct link to Rehearsing")

A backup nobody has restored is not a backup. At minimum:

* check the file size after every backup — a sudden drop means the dump failed part way;
* **do a full restore once a quarter** into a separate test database and test instance, verifying through steps 5 and 6 that you can log in, the rules are there, and evaluation is running;
* time the rehearsal. When it actually happens, "how long does a restore take" is a question you will have to answer.

For rebuilding from scratch in a fresh environment, see [Disaster recovery](https://n9e.github.io/docs/operations/disaster-recovery.md).

## Related[​](#related "Direct link to Related")

* [Disaster recovery](https://n9e.github.io/docs/operations/disaster-recovery.md)
* [Rollback and recovery](https://n9e.github.io/docs/operations/rollback.md)
* [Embedded TSDB single-Center limits](https://n9e.github.io/docs/operations/embedded-tsdb-limits.md)
* [Production readiness checklist](https://n9e.github.io/docs/get-started/production-checklist.md)