Skip to main content

Upgrade or database migration fails

Unknown column errors, blank pages or a hung process after an upgrade mean the schema did not fully migrate — usually missing database privileges or an interrupted run.

You swapped in the new binary, the process started, and the UI reports Unknown column; or one new feature's page is blank; or the process hangs entirely after startup. All of these point at one thing: the schema was not fully brought up to date.

Gauge urgency by reach. Errors confined to one new feature's page while everything else works can be fixed calmly. Cannot log in, or the events page will not open means a core table is affected — go straight to the last section and get back to a consistent state.

The process starting is not evidence the migration succeeded​

This is the foundation of the page, and the least intuitive part: a migration failure is non-fatal — the error is only logged, and the process starts anyway.

So "the service came up" says nothing about whether the upgrade worked. The only evidence is the startup log:

grep -n "failed to migrate table" /path/to/n9e.log

Any output means the migration is incomplete. The line looks like this, and the name after the colon is the table that did not get upgraded:

ERROR failed to migrate table:alert_his_event Error 1142 (42000): ALTER command denied to user 'n9e'@'%' for table 'alert_his_event'

Pull these four out together — they mean quite different things:

grep -nE "failed to migrate table|failed to create index|skip index .* not migrated yet|recovered panic during" /path/to/n9e.log
Log lineMeaningAct on it?
failed to migrate table:<table> ...That table was not upgradedYes — see the first root cause below
failed to create index ... please create it manually with an online DDL tool (gh-ost / pt-online-schema-change)An index was not built; the table structure itself is fineYes, but not urgently — see the second
skip index <name> on notification_record: column(...) not migrated yet, will retry on next startIndex building and column adding run in parallel, and this pass missedNo — the next startup picks it up
recovered panic during <scene>A panic during migration was caughtYes — include the stack when you report it

How the migration happens​

There is no separate migration command. Every n9e startup runs GORM's AutoMigrate: it creates missing tables, adds missing columns and builds missing indexes. It is additive only — it never drops anything, and never records what the schema looked like before.

Three consequences you will use repeatedly:

  1. An upgrade needs no hand-written SQL from you, provided the account it connects with may create and alter tables;
  2. Restarting the same version retries the migration, so after one failure, fixing the permission and restarting often completes it;
  3. There is no reverse migration. After rolling the binary back the database holds tables and columns the old version does not know about — it usually ignores them and starts fine, but that is not a guarantee.

docker/sqlite.sql and docker/initsql/a-n9e.sql are fresh-install files and play no part in an upgrade. sqlite.sql in particular is still at v7 and contains none of notify_rule, notify_channel, message_template, event_pipeline or ai_llm_config — which is fine, AutoMigrate creates them. Do not treat either file as the source of truth for the schema.

The database name has been n9e_v6 since v6 and does not change on upgrade.

Common root causes, and how to confirm each​

The database account cannot create or alter tables​

This is the main source of failed to migrate table. Plenty of production databases grant the application account only DML, not DDL.

How to confirm: the error text on that log line says so directly — on MySQL, Error 1142 ... ALTER command denied or CREATE command denied; on PostgreSQL, permission denied for table .... You can also check it head-on:

-- run as the account Nightingale connects with
SHOW GRANTS FOR CURRENT_USER();

How to fix, one of two ways:

  • Grant temporarily and restart — give the account CREATE, ALTER and INDEX, restart n9e so it completes the migration itself, confirm the log is clean, then revoke;
  • Hand it to the DBA — take the sections for your target version out of docker/migratesql/migrate.sql. That file accumulates by version, so applying it section by section makes any failure much easier to locate.

An index will not build, and the log tells you to call a DBA​

The indexes on two large tables are built asynchronously after startup, which can take tens of minutes on a big database. The service is usable throughout.

TableIndex
alert_his_eventidx_group_last_eval_time
notification_recordidx_nr_rule_created_evt, idx_nr_created_at

On MySQL both are issued with an explicit ALGORITHM=INPLACE, LOCK=NONE — where online index creation is unsupported it fails loudly rather than silently degrading into a write-blocking DDL. PostgreSQL uses CREATE INDEX CONCURRENTLY.

How to confirm: the please create it manually with an online DDL tool line, or directly:

SHOW INDEX FROM notification_record;
SHOW INDEX FROM alert_his_event;

How to fix: do as it says and add them with gh-ost or pt-online-schema-change. It works without them, but the events list and notification record queries get noticeably slower.

PostgreSQL has one self-healing behaviour worth recognising: a failed CREATE INDEX CONCURRENTLY leaves an unusable index behind, and the next startup drops it before rebuilding, logging dropping invalid index ... left behind by a failed CREATE INDEX CONCURRENTLY. Ignore that line.

Startup hangs​

The process has not exited and has not errored; the log simply stops in the migration phase.

How to confirm: check the database for DDL waiting on a lock:

SHOW PROCESSLIST;                          -- MySQL: look for "Waiting for table metadata lock"
SELECT * FROM pg_stat_activity WHERE state <> 'idle';   -- PostgreSQL

A repeated CREATE INDEX against a very large alert_his_event waits on the metadata lock forever and hangs startup with it. Several instances starting the new version for the first time simultaneously is the usual way to hit this.

How to fix: roll the upgrade, one instance at a time, waiting for a clean log before starting the next. If you are already stuck, stop the surplus instances and let one finish building the index.

The config file is the old one, so new features stay off​

This is not a migration failure, but it presents very much like "the upgrade did nothing": the feature is there and the page is empty.

How to confirm: diff for the sections the new version added:

diff -u /opt/n9e/etc.bak/config.toml /opt/n9e/etc/config.toml

The classic case is [EmbeddedTSDB], which only exists in the new etc/config.toml — carrying your old config over leaves it off, presenting as "I replaced the binary and the embedded TSDB did nothing".

How to fix: copy the new sections into your config file one at a time. Do not overwrite your file with the new one — that discards your customisations. The integrations/ directory is the opposite: replace it wholesale, do not merge, or you get dashboards and rule templates that no longer match the version.

The UI version does not match the backend​

How to confirm: System → About shows the frontend and backend versions separately; both should be the new one.

How to fix: hard-refresh the browser (the cache-clearing kind). Cached frontend js is the most common false alarm after an upgrade.

Getting back to a consistent state after a partial migration​

In order of increasing cost — try the cheap one first:

  1. Fix the cause and restart once. The migration is retried at every startup, so restoring the permission, freeing the disk or releasing the table lock and restarting n9e completes most partial migrations by itself. This is the first thing to try;
  2. Have the DBA add the missing tables and columns using the matching version sections of docker/migratesql/migrate.sql, then restart and confirm no failed to migrate table remains;
  3. Roll the binary back. The columns the new version added are merely surplus to the old one, which normally ignores them and boots. If the old version reports Unknown column or Error 1364 (field has no default value), this route does not work;
  4. Restore the database from backup. The cost is blunt: everything created since the backup is gone — rules written since, alert events, notification records, dashboard edits. In a cluster, stop every instance first, or a surviving one keeps writing and fights the import.

Option 4 is exactly what the pre-upgrade mysqldump is for. The steps are in Rollback.

Confirming the upgrade succeeded​

Work through these; each has a definite expected result:

  1. A clean log: grep "failed to migrate table" <log> produces no output;

  2. The right version:

    curl -s --noproxy '*' http://127.0.0.1:17000/api/n9e/version

    then check the frontend version under System → About;

  3. Heartbeats ticking: the heartbeat times under Infrastructure → Hosts are updating — if they are not, check Redis connectivity first;

  4. One end-to-end run: pick an existing alert rule and test fire it, which verifies query, evaluation, event and notification in one pass;

  5. One test notification: hit Run test on any notification rule to confirm the media config survived.

Steps 4 and 5 pass perfectly well while indexes are still building in the background — do not restart just because you see index building in the log.

Collect this before you ask​

  1. The versions before and after (./n9e --version or /api/n9e/version; the frontend version is on the About page);
  2. The complete startup log, at least from launch to "serving";
  3. The output of grep -nE "failed to migrate table|failed to create index|recovered panic during" <log>;
  4. The database type and version, and SHOW GRANTS for the account Nightingale uses;
  5. The result of diff -u etc.bak/config.toml etc/config.toml;
  6. The deployment shape: how many instances, whether there is an edge, and whether metadata is in MySQL, PostgreSQL or SQLite.

Redacting: replace the DSN password, channel tokens and LLM API keys. Keep the table and column names in the error text — they are the answer. Host names in SHOW GRANTS can be masked.

Next​

On this page