Multi-AZ / multi-node deployment

The chart defaults and the launch/demo runbooks (CUSTOMER-TEST-RUNBOOK.md, DEMO-INSTALL.md) both target a single-node k3s cluster — that's the right shape for an evaluation, and the wrong shape for production. This page is the reference architecture for a real multi-node, multi-AZ install: the 3-node minimum, how each stateful dependency gets its own HA story, a sizing starting point, and what the single-node demo shape deliberately gives up.

3-node minimum

Three worker nodes, one per AZ, is the floor for an install that should survive a single AZ outage:

Component	Why 3
CNPG Postgres	postgres.cnpg.instances: 3 = one primary + two hot standbys; CNPG's own anti-affinity needs 3 distinct nodes to actually place them on different failure domains, not just 3 pods on 2 nodes
API / frontend / scanner	Stateless — topology spread needs ≥3 zones to actually distribute 3-way, not just satisfy maxSkew: 1 on 2
Redis	Only matters if you opt into redis.ha.enabled: true — see below

Below 3 nodes, topologySpread's default whenUnsatisfiable: ScheduleAnyway (see values.yaml) means pods still schedule, they just stack up on whatever nodes exist — the chart won't refuse to install on a 1- or 2-node cluster, it just can't spread further than the topology actually has room for.

AZ distribution per component

Postgres — CNPG replicas, one per AZ

CloudNativePG is the HA path; the chart's legacy single-replica postgres-statefulset.yaml (postgres.cnpg.enabled: false, the default) is single-node by construction and has no AZ story. With CNPG enabled and instances: 3, set a Cluster.spec.affinity.topologyKey: topology.kubernetes.io/zone (CNPG's own affinity field, not this chart's topologySpread) so the operator refuses to co-locate two replicas in the same zone. A primary-AZ outage then promotes a standby in an unaffected zone in under 30 seconds, per the existing failover claim in Postgres on CloudNativePG.

Redis — external HA, not the in-cluster StatefulSet

Per Redis high availability, Redis is a boot-time hard dependency but not on the runtime read/write path — so the chart's default single-replica redis-statefulset.yaml is an acceptable SPOF for many deployments (a Redis-down pod just fails to start; already-running pods keep serving). For a zero-boot-time-risk multi-AZ install, set redis.ha.enabled: true plus redis.ha.host pointed at an externally managed HA Redis (ElastiCache Multi-AZ, Azure Cache for Redis, Memorystore, or a self-run Sentinel cluster) — the chart renders an ExternalName Service either way, so nothing else in the API config changes.

S3 — external by design

Object storage was never in-cluster: storage.* in values.yaml always points at an external S3-compatible endpoint. Multi-AZ durability is whatever the S3 provider gives you (AWS S3 standard storage is multi-AZ by default; a self-hosted MinIO/Ceph RadosGW needs its own multi-node erasure-coded pool to make the same claim). The optional dual-write backup (storage.backup) to a second endpoint is a second-region/second-provider concern, not an AZ one — see Disaster recovery for that scenario.

Topology spread across zones, not just nodes

The chart's topologySpread (roadmap item 213) defaults to topologyKey: kubernetes.io/hostname — it spreads api/frontend/scanner pods across nodes, which is the right default for a single-AZ cluster (or the single-node demo shape, where it's a no-op). For a real multi-AZ cluster, override it to spread across zones instead:

topologySpread:
  enabled: true
  maxSkew: 1
  topologyKey: topology.kubernetes.io/zone
  whenUnsatisfiable: DoNotSchedule

DoNotSchedule is a deliberate tightening for production — it trades "pods might go Pending if a zone is briefly out of capacity" for "pods never silently stack in one zone," which is the failure mode you're trying to buy insurance against in the first place.

Sizing: RPS → replicas → resources

This is a starting point derived from the chart's own configured defaults, not a measured load-test baseline — no benchmark run has yet pinned actual sustained RPS-per-pod for this specific deployment shape (that's the open item, roadmap item 231, "Lasttest-Baseline + Regression-Gate"). Treat the table as where to start dialing in autoscaling.keda.targetRequestRate, then correct it against your own tests/load run or production metrics.

The KEDA ScaledObject path (autoscaling.keda.enabled: true, see values.yaml) defaults targetRequestRate: 50 (requests/sec/pod, averaged over the 30s pollingInterval) for the api component, combined with its existing CPU target. Each api pod requests 250m CPU / 256Mi memory and is capped at 1000m / 512Mi (api.resources in values.yaml):

Sustained RPS	api replicas (at 50 rps/pod)	Aggregate api CPU request	Aggregate api memory request
≤ 100	2 (autoscaling.minReplicas)	500m	512Mi
~250	5	1250m	1.25Gi
~500	10 (autoscaling.maxReplicas)	2500m	2.5Gi
> 500	raise autoscaling.maxReplicas first	—	—

Two inputs feed back into this table and are worth tuning before trusting it at scale:

• targetRequestRate itself is a starting guess, not a measured ceiling — a heavier per-request workload (large artifact uploads, metadata-heavy index rebuilds) will saturate CPU well before 50 rps/ pod, in which case the existing CPU trigger (not the RPS one) will be what actually drives scale-out.
• frontend (static nginx) and scanner (queue worker, polls Postgres rather than serving requests) have no request-rate signal to scale on — their ScaledObjects use the CPU trigger only, so their replica counts track worker/serving load, not RPS directly.

Known single-node trade-offs (demo shape vs. production)

The single-node k3s shape used for demos and customer test installs — real experience, not a hypothetical — makes several deliberate compromises that the multi-node topology above removes:

• No CNPG multi-instance HA. postgres.cnpg.instances set to 1 (or CNPG disabled entirely) means a node loss is a Postgres outage, not a 30-second failover.
• topologySpread is a no-op. maxSkew: 1 across one node trivially satisfies itself; the constraint exists in the manifest but provides zero actual spread.
• PDBs can't protect against involuntary node loss. The rendered PodDisruptionBudgets (api-pdb.yaml, frontend-pdb.yaml, scanner-pdb.yaml; minAvailable: replicas - 1) only guard voluntary disruptions (drains, kubectl evict) — on a single node, a hard node failure takes every pod regardless of the PDB, since there's nowhere else for the scheduler to place them.
• Redis HA is moot. redis.ha.enabled: true still requires an externally managed HA Redis reachable from the cluster; a single-node demo box has no such thing, so it stays on the default in-cluster single replica.
• Ingress has one path in. k3s's bundled ServiceLB binds directly to the one node's ports 80/443 (see CUSTOMER-TEST-RUNBOOK.md) — there's no second node for a failed ingress controller pod to reschedule onto.

None of this makes the single-node shape wrong for its purpose (evaluation, demos, small internal installs that accept the risk) — it just means "it works on the demo cluster" is not evidence toward "it survives an AZ loss," and the two should not be conflated when sizing a production install.