Service-level objectives
The bundled alert rules were written against specific latency and error-rate thresholds, but those thresholds were never written down as objectives in their own right. This page is that write-down: four SLOs, the metric each is measured from, the target, and which alert enforces the corresponding error-budget burn. Treat it as the source of truth if you're tuning a threshold for your own environment — the alert should move with the objective, not the other way around.
| # | SLO | SLI (PromQL source) | Target | Budget window |
|---|---|---|---|---|
| 1 | API availability | cluster:http_requests:error_rate5m / error_rate1h (non-5xx share of http_requests_total) | 99.9% of requests non-5xx | 30d rolling |
| 2 | Pull-redirect latency | s3_request_duration_seconds{op="presign"} p99 | < 300ms | 30d rolling |
| 3 | Scan-queue lag | sum(scan_queue_depth{state=~"pending|claimed|running"}) | backlog drains below 1000 within 30m | per-incident |
| 4 | Replication lag | replication_lag_seconds per peer | p99 peer lag < 60s | 30d rolling |
1. API availability
Why this SLI: http_requests_total is labelled by route and status on every chi handler (see Metric reference), so a non-5xx share is a whole-API availability signal without needing a separate synthetic prober.
99.9% over 30 days leaves a budget of about 43 minutes of majority-5xx traffic per month. OrbitalRegHighErrorRate enforces this with a multi-burn-rate check (5m fast window, 1h slow window both have to be hot) so a one-off blip doesn't page but a sustained burn does — see High error rate.
2. Pull-redirect latency
Why this SLI: a pull is a chi handler doing an artifact lookup followed by PresignedGetObject and a 302 — the bytes never transit the API pod (see ARCHITECTURE.md). s3_request_duration_seconds{op="presign"} (see S3 / MinIO) isolates exactly that presign call across all 40+ format adapters and Docker/OCI, which the generic per-route http_request_duration_seconds cannot — that series is split by the ~80 distinct download route patterns across the format adapters and dilutes into the broader OrbitalRegHighLatency (p99 > 1s, any route) alert instead of tracking this SLO on its own.
OrbitalRegPullRedirectLatencyHigh reads s3_request_duration_seconds{op="presign"} directly against the 300ms target above via the s3:presign_duration_seconds:p99_5m recording rule — see Pull-redirect latency high. Unlike OrbitalRegHighLatency (a broader 1s/any-route check that dilutes across ~80 download route patterns), this alert tracks the SLO's own 300ms budget in isolation.
3. Scan-queue lag
Why this SLI: the scan dispatcher does not currently emit a queue-age (oldest-pending-job) gauge, only depth (scan_queue_depth, see Scan dispatcher). Depth staying above 1000 for 30 minutes is used as the lag proxy: the dispatcher claims jobs on a fixed cadence, so a backlog that isn't draining is definitionally a lag problem. OrbitalRegScanQueueDeep enforces this — see Scan queue deep. This SLO is scoped per-incident rather than a 30-day rolling budget because queue depth is bursty by design (importer churn) in a way request-rate and replication lag are not.
4. Replication lag
Why this SLI: replication_lag_seconds (see Geo-Sync replication) is seconds since the last successful push to a peer, sampled per peer. Any peer above 60s for 5 minutes trips OrbitalRegReplicationLagHigh — see Replication lag high. This SLO only applies to commercial-tier installs with geo-sync peers configured; an install with no peers has no series to evaluate and the alert is a no-op.
Error-budget dashboard
docs/grafana/orbitalreg-slo-error-budget.json is a dedicated panel set for the four budgets above: remaining 30-day budget as a percentage for API availability, pull-redirect latency, and replication lag, plus a current-backlog-vs-threshold gauge for scan-queue lag (a per-incident SLO, not a rolling budget — see the row above). Import it alongside the overview and deep-dive dashboards, which cover the underlying RED/USE metrics these SLIs are built from but not the budget math itself.