Zero-downtime upgrade

The chart is built for zero-downtime rollouts — maxUnavailable: 0 on the API/frontend/scanner Deployments, a PodDisruptionBudget on the API, and /health/ready-gated readiness — but a chart shape is not the same as a proven rollout. This page is the procedure plus the load-test harness (PRODUCT-ROADMAP item 122) that proves it, and what's still pending an operator's own cluster run.

What the chart already does

Mechanism	Where
RollingUpdate with maxUnavailable: 0, maxSurge: 1	api-deployment.yaml, frontend-deployment.yaml, scanner-deployment.yaml
PodDisruptionBudget (minAvailable: replicas - 1)	api-pdb.yaml (only rendered when api.replicas > 1)
Readiness gate before a pod joins the Service	readinessProbe → GET /health/ready (Postgres + S3 must be reachable; see Redis high availability for why Redis is not a hard dependency here)
Migrations applied before the process serves traffic	db.Migrate runs at boot in api/cmd/server/main.go, before the HTTP listener starts

None of that is new — it's the standard ingredients for a zero-downtime rollout. What item 122 adds is proof: a load-test scenario pair that drives real traffic through an actual rollout and asserts zero 5xx, for both the plain-upgrade case and the upgrade-that-ships-a-migration case.

Procedure

1. Rolling upgrade (no migration)

Run tests/load/scenarios/rolling-upgrade.js against the deployment, then trigger the upgrade in a second terminal while it's running:

export ORBITALREG_BASE_URL=https://orbital.staging.example.com
export ORBITALREG_TOKEN=orbsa_xxxxxxxxxxxxxxxxxx
export ORBITALREG_REPO_ID=00000000-0000-0000-0000-000000000000
export ORBITALREG_ROLLOUT_DURATION=10m   # comfortably longer than the rollout itself

k6 run tests/load/scenarios/rolling-upgrade.js &

# second terminal, once the first is running:
helm upgrade orbitalreg charts/orbitalreg -f values-staging.yaml
kubectl rollout status deployment/orbitalreg-api
wait

A pass is k6 exiting 0 with five_xx_total.....: 0 in the summary. The scenario's own thresholds are deliberately loose on latency (a rolling restart legitimately dips capacity for a few seconds per pod) — the only load-bearing signal is the 5xx counter. See tests/load/README.md for the full scenario catalogue and prerequisites (token scope, k6 install, ORBITALREG_REPO_ID).

2. Rolling upgrade with a migration

tests/load/scripts/migration-under-load.sh orchestrates both halves in one command: it starts the traffic + /health/ready-poll scenario (scenarios/migration-under-load.js), waits WARMUP_SECONDS (default 5s), then applies the pending migration via MIGRATE_CMD, and fails loudly if either the migration command or the k6 thresholds don't come back clean.

export ORBITALREG_BASE_URL=https://orbital.staging.example.com
export ORBITALREG_TOKEN=orbsa_xxxxxxxxxxxxxxxxxx
export ORBITALREG_REPO_ID=00000000-0000-0000-0000-000000000000
export MIGRATE_CMD='kubectl exec deploy/orbitalreg-api -- /app/server -migrate-only'  # operator's real migration path

tests/load/scripts/migration-under-load.sh

MIGRATE_CMD defaults to the local migrate/migrate Docker invocation against the compose stack's Postgres — that default only proves the orchestration script itself; a cluster run needs MIGRATE_CMD pointed at whatever the operator's actual migration Job or kubectl exec invocation is. This script never assumes cluster credentials.

Expected output on a healthy run:

PASS: migration-under-load — 0 five_xx, N health/ready poll(s) non-200

N (the count of non-200 /health/ready polls during the migration window) is informational, not a failure signal — a pod that's mid-migration is supposed to report not-ready (CLAUDE.md §3: "/health/ready blocks until applied"). The only failure signals are a non-zero five_xx_total or the migration command itself exiting non-zero.

What's proven so far, and what isn't yet

Both scenarios and the orchestration script exist and pass locally — see tests/load/README.md's "Migration under load" section for the exact local run performed (make local-reset, API boot, /health/ready polled at 1s intervals through a no-op migrate ... up, stayed at 200 throughout). That local run proves the script's mechanics; it does not prove the actual zero-downtime property, for two reasons:

• A local single-process run has no rolling pod cycle for rolling-upgrade.js to observe — there's nothing to roll.
• make local-reset starts from a fully-migrated DB, so the migration apply in migration-under-load.sh's default MIGRATE_CMD takes the "no change" no-op path — it never actually exercises the /health/ready-gating window a slow, real migration would produce.

Proving the property for real requires a cluster run: multiple API replicas, an actual helm upgrade (Phase A) or a migration that runs long enough to observe the gating window (Phase B), against a real Service with real in-flight traffic. That run is not something this bot can execute or judge (BOT-INSTRUCTIONS §5 — "say so, don't guess") — an operator needs to run both procedures above against a real staging or pre-prod cluster and confirm five_xx_total stayed at 0 before this item's heading is marked done.

Related docs

• tests/load/README.md — full scenario catalogue, prerequisites, cleanup
• Redis high availability — why a Redis blip mid-rollout doesn't fail readiness
• Postgres on CloudNativePG — the HA story for the other hard dependency
• Disaster recovery — the runbook for total-loss scenarios, as opposed to a planned upgrade