Signals to collect#
- API request errors and latency; health/readiness failures.
- Unprocessed raw events, oldest backlog age, outbox publication delay and quarantined events.
- Escalation delay, overdue jobs and worker-loop progress.
- Notification attempts, failures, unknown outcomes and provider acceptance latency.
- PostgreSQL connections, replication/WAL health, storage and backup freshness.
- NATS stream replica health, pending messages and durable consumer progress.
Local observability stack#
The on-prem Compose observability profile provides Prometheus, Grafana with a provisioned dashboard, and an OTel Collector configuration. Grafana reporting and update checks are disabled in that profile. Keep private monitoring UIs behind customer access controls.
OpenTelemetry boundary#
AckNexa exposes Prometheus metrics and structured JSON logs. The included collector accepts internal OTLP metrics and exposes them for local Prometheus. A customer Collector can also scrape the application's metrics. Native application OTLP distributed traces are not implemented; do not assume trace correlation exists because a collector is deployed.
Keep metrics off public ingress. The Helm API ingress publishes only the API path prefixes, not /metrics; Prometheus scrapes the in-cluster API Service. The API's /metrics can require a bearer token (METRICS_TOKEN), and the bundled Compose nginx answers /metrics on the API host with 404.
Health checks are not delivery checks#
Readiness proves a limited dependency check, not that every provider, schedule or escalation path works. Combine probes with queue/progress metrics and periodic authenticated end-to-end tests using dedicated internal recipients.
Independent alerting#
Monitor AckNexa from another customer-controlled failure domain, and route critical platform alerts through a channel that does not rely only on AckNexa. A self-hosted Alertmanager and an independent dead-man check can run entirely on the private network. A dashboard on the same failed host cannot page about its own disappearance.