Redis high availability

Redis ships as part of the chart alongside Postgres and S3, but unlike those two it is not currently in the hot path for any read or write. This page covers what actually depends on Redis today, the two ways the chart can deploy it, and how to alert on its availability in each mode.

What actually depends on Redis today

A grep across internal/ for redis/rdb. at the time this page was written turned up exactly three consumers — notably not sessions, rate limiting, or caching, despite Redis's usual role in that stack:

Consumer	Behaviour on Redis outage
main.go boot (cache.Connect)	Hard dependency — the process calls os.Exit(1) if it can't connect at startup
GET /health/ready	Degrades gracefully (see below) — does not fail readiness
ArtifactHandler's rdb field (oidc_exchange.go)	Unused; reserved for future sharding, not wired to any request path

Sessions are HMAC-signed cookies, not server-side session state (see ARCHITECTURE.md), and the login rate limiter is Postgres-backed (internal/security/loginlimit.go). So today, a Redis outage after a pod has started does not interrupt reads or writes — only a pod that hasn't finished booting yet needs Redis reachable.

Graceful /health/ready degradation

/health/ready pings Postgres, S3, and Redis on every check. Before this changed, any one of the three failing flipped the whole endpoint to 503, which pulled pods out of rotation on a Redis blip even though nothing user-facing was actually broken.

internal/cache/ready.go's ReadyResult now classifies each dependency:

• Postgres and S3 are hard — a failure still returns 503 and takes the pod out of rotation, because both sit on the request path.
• Redis is non-hard — a failure is still reported in the JSON body (so the outage stays visible to monitoring) but does not flip the status code, so pods stay in rotation.

$ curl -s http://localhost:8080/health/ready | jq
{
  "postgres": "ok",
  "s3": "ok",
  "redis": "dial tcp: connection refused"
}

A 200 with a non-"ok" redis field is the expected shape during a Redis outage — that's the signal to page on, not the readiness status code itself.

Deployment modes

Default: in-cluster StatefulSet

redis.ha.enabled: false (the default) deploys the chart's own single-replica redis-statefulset.yaml. This is a SPOF for the boot dependency described above, but per the table it no longer takes already-running pods out of rotation.

HA: externally managed Redis

Set redis.ha.enabled: true plus redis.ha.host / redis.ha.port to point at an externally managed, already-HA'd Redis — AWS ElastiCache, Azure Cache for Redis, GCP Memorystore, or a self-operated Sentinel cluster:

redis:
  enabled: true
  ha:
    enabled: true
    host: my-redis-cluster.abcdef.use1.cache.amazonaws.com
    port: 6379

With ha.enabled: true, the chart skips redis-statefulset.yaml and instead renders redis-ha-service.yaml — an ExternalName Service still named <fullname>-redis, so ORBITALREG_REDIS_ADDR (orbitalreg.redisAddr in _helpers.tpl) doesn't need to change between modes. The NetworkPolicy's api-to-deps rule opens egress to ha.port for this mode, since Redis is no longer an in-cluster pod that the existing in-cluster rule would cover.

No PodDisruptionBudget is rendered in HA mode — there's no in-cluster pod for a PDB to protect. Failover is the managed provider's job.

Alerting

The bundled PrometheusRule (observability/prometheus-rules.yaml) ships an OrbitalRegRedisDown alert scoped to the StatefulSet mode's in-cluster pod (kube_pod_status_ready, matching the pattern already used for the S3 readiness-proxy alert). Severity is warning, not critical, reflecting the graceful-degradation behaviour above — an already-running deployment stays functional, but a pod that's mid-rollout when Redis is down will not become ready.

In HA mode there is no in-cluster Redis pod for this alert (or any kube_pod_status_ready-based rule) to match — alerting on the managed instance's availability and failover is the responsibility of the provider's own monitoring (CloudWatch alarms for ElastiCache, Azure Monitor for Cache for Redis, Sentinel's own +switch-master events, etc.). Wire that provider-side alert to the same on-call route as the rest of this bundle.

Incident runbook: OrbitalRegRedisDown fires

1. Confirm scope first. Query /health/ready — if postgres and s3 both read "ok" and only redis doesn't, this is exactly the graceful-degradation case above: already-running pods keep serving reads and writes. There is no user-facing incident yet.
2. Check for pods mid-rollout. A pod that hasn't finished booting when Redis is down will fail its boot-time cache.Connect and restart-loop (see the hard-dependency row in the table above). If a deploy or autoscale event is in flight, that's the actual symptom to chase, not request-path errors.
3. Restore Redis. StatefulSet mode: kubectl delete pod the Redis pod to force a reschedule, or investigate the underlying node if it's a repeat failure. HA mode: this is the managed provider's failover — check their console/alarms rather than the cluster.
4. Downgrade the page once redis reports "ok" again in /health/ready; no further action is needed since no state was lost (Redis holds no durable data in this deployment — see Disaster recovery).

This sequence is exercised end-to-end (not just asserted from the alert rule) by TestRedis_Kill_CorePathsStay200 (tests/integration/harness/redis_chaos_test.go): the test boots a private per-test Redis container, publishes an artifact, kills the container mid-run, and asserts /health/ready stays 200 with Postgres/S3 still "ok", that the pre-kill artifact is still downloadable, and that a new upload immediately after the kill still succeeds. It lives in the harness package rather than tests/integration because the latter's TestMain shares one Redis container across the whole package — killing it there would take every sibling test down too.

Related docs

• Monitoring (metrics + alerts)
• Postgres on CloudNativePG — the HA story for the other hard dependency