Backups and disaster recovery

What to protect#

PostgreSQL: configuration, incidents, raw events, jobs, notification attempts, outbox and audit.
NATS stream state is optional: the workers recover every unprocessed raw event from PostgreSQL when the stream is empty. Use supported snapshots or a quiesced backup if you also want the queue files.
ACKNEXA_CONFIG_KEY and other recovery secrets, protected independently from the database copy.
TLS materials, IdP configuration/backups, deployment values, image digests and approved release artifacts.
Customer-created monitoring dashboards and alert configuration. Redis is not authoritative for the current core.

Small-install backup#

bash scripts/onprem_backup.sh \
  /secure/path/acknexa.env \
  /secure/backups/acknexa-NEW

Use a new absolute destination. By default (ACKNEXA_BACKUP_MODE=live) the script takes a consistent database snapshot while ingress, API and workers keep paging, and saves the protected configuration and a checksum; NATS is not copied. ACKNEXA_BACKUP_MODE=quiesced instead pauses the Web, API, workers and NATS, also copies the NATS files, and restarts only previously running services; paging is paused while it runs. Supply the same optional Compose overlays through ACKNEXA_COMPOSE_OVERRIDES when applicable.

Restore safely#

Verify backup checksums and image/database/broker version compatibility.
Provision a new isolated target, not the live source. Restore protected configuration and the encryption key.
Restore the PostgreSQL dump into an empty database. The small-install restore script then starts with an empty NATS stream and the workers recover unprocessed raw events from PostgreSQL. If you restore NATS data or snapshots, use the supported, stopped-target procedure; do not merge files into a live stream.
Start matching application versions without test seeding. Restored pending escalations resume paging as soon as workers start, so point the target at test recipients unless this is a real cut-over. Inspect tenant configuration, audit history, pending work and incident state.
Use dedicated test recipients to verify delivery and ACK before switching traffic. Measure actual recovery time and data loss against agreed targets.

Production recovery#

Use continuous WAL/PITR and encrypted backups to customer-controlled NAS, another site, or an internal object store as appropriate. Cloud S3 is not required. Exercise primary loss, a host/failure-domain loss, broker quorum loss, and restoration of encrypted provider configuration.

Do not replay blindly#

A lost stream may include work whose outbox row was already marked published. Resetting every publication flag can create unwanted replay. Recovery must identify unprocessed work, respect durable idempotency, and account for ambiguous provider acceptance. Backup success alone is not a disaster-recovery pass.