Rollout troubleshooting
This page is a consolidated collection of rollout-time findings: the kind of problem that shows up once, while installing or upgrading, not as a steady-state alert. If OrbitalReg is already running and something is wrong now, you want Runbooks (Prometheus alerts) or Operator troubleshooting (reconcile failures) instead — this page is for the install/upgrade window itself.
Every entry below was hit for real during one of the project's own rollouts (trial, demo, or a customer test) — see the source runbook linked at the end of each section for the full walkthrough it came from.
Cluster / networking
| Symptom | Cause | Fix |
|---|---|---|
Pods CrashLoop with "connection refused" to a Running pod (api→postgres, bucket-job→minio) | k3s enforces NetworkPolicies (unlike a local make dev); the chart's default-deny blocks a separately-deployed MinIO | Trial/demo overlays set networkPolicy.enabled: false. On an older chart, add --set networkPolicy.enabled=false |
cannot patch StatefulSet … Forbidden | StatefulSet fields are immutable | Fresh namespace/install, or kubectl -n <ns> delete sts orbitalreg-postgres --cascade=orphan (keeps the PVC) then helm upgrade. Cleanest fix: move to CNPG |
| Rebuilt image not picked up (still the old bug) | k3s/containerd caches by tag; :<tag> is pulled IfNotPresent | After re-pushing the same tag: sudo k3s crictl rmi docker.io/<image>:<tag>, then restart the deployment |
ImagePullBackOff | Image not published to the registry the chart points at | Run the publish workflow, or point at the alternate registry (e.g. --set api.image.repository=ghcr.io/… + imagePullSecrets) |
Source: Customer test runbook, Demo install
Storage / state
| Symptom | Cause | Fix |
|---|---|---|
api: password authentication failed for user "orbitalreg" after a reinstall | helm uninstall does not delete the Postgres PVC — a reinstall generates a new password while Postgres keeps the old one | Clear state first: kubectl delete namespace <ns> (removes pods, PVCs and secrets), then reinstall. A real one-shot customer install never hits this |
MinIO ImagePullBackOff (bitnami/minio, then minio/minio) | Two registry removals in a row: Bitnami dropped its free Docker Hub images (2025-08-28), and MinIO deleted minio/minio and minio/mc outright on 2026-09-11 — the whole repository, so pinned RELEASE.* tags are unpullable too, and quay.io no longer serves them anonymously either | Use cgr.dev/chainguard/minio plus cgr.dev/chainguard/minio-client:latest-dev for the bucket Job, as in charts/orbitalreg/examples/minio-{demo,trial}.yaml. Two things change with it: the image runs as UID 65532, so the pod needs fsGroup: 65532 or it cannot write to a volume created by the old root-owned image (MinIO then reports InsufficientWriteQuorum, which looks like a full disk and is not); and the non--dev client image is distroless with no shell, so a script-based Job needs the -dev variant (its shell is /usr/bin/sh; /bin is a symlink to /usr/bin, so /bin/sh works too) |
MinIO InsufficientWriteQuorum / "Storage resources are insufficient for the write operation" on a disk with plenty of free space | Almost always a permissions problem, not a capacity one — the server cannot write to /data. Most common after switching to an image that runs as a different UID | Check df -h first to rule capacity out, then set fsGroup to the image's UID (65532 for Chainguard) so the kubelet chowns the volume on mount |
Frontend CrashLoop, open() "/run/nginx.pid" … Permission denied | The nginx-unprivileged image (UID 101) can't write /run | Image ships pid /tmp/nginx.pid from 0.1.0+; on an older image, patch a writable /run emptyDir onto the deployment |
Source: Customer test runbook
TLS / DNS
| Symptom | Cause | Fix |
|---|---|---|
certificate stuck READY=False | The A-record doesn't resolve to the ingress IP yet, or ports 80/443 aren't reachable — HTTP-01 fails otherwise | kubectl -n <ns> describe certificate <name>; fix DNS/firewall, then wait for the next HTTP-01 attempt |
admission webhook … risky annotation | An ingress-nginx security-header snippet gets blocked by the admission webhook | Set ingress.securityHeaders.enabled=false |
Source: Customer test runbook, Demo install
SAML / SSO
| Symptom | Cause | Fix |
|---|---|---|
Pod CrashLoop, log fetch idp metadata | idpMetadataURL wrong/placeholder, or the IdP isn't reachable (check egress) | Verify the URL resolves and returns metadata from inside the cluster network |
Pod CrashLoop, log load sp keypair: cert: no PEM block | SAML enabled but the SP keypair secret is empty | Delete the stale *-saml secret and helm upgrade — the chart regenerates it (genSelfSignedCert) |
SAML login: InvalidResponse | IdP cert rolled, or the ACS (reply URL) is wrong | ACS must be exactly https://<domain>/saml/acs (https, no trailing slash); re-fetch IdP metadata after a cert rotation |
| 405 right after the IdP login redirect | IdP's "Service Provider Binding" is Redirect (a common default) — the SAML response arrives as a GET, but crewjam's ACS only accepts POST | Set the provider's binding to Post |
404 on /saml/acs | ACS URL has a trailing slash or a different path — crewjam compares it exactly | Set the ACS URL to exactly https://<domain>/saml/acs |
405 when opening /saml/acs directly in a browser | Expected — that's a GET against a POST-only endpoint | Not an error |
Logged in but not admin / 403 on admin pages | Your principal isn't matching auth.adminGroups: Entra/Azure send App Role values or security-group GUIDs, never a display name | Create an App Role with the exact value used in adminGroups (e.g. admin) and assign it, or put the raw email into adminGroups |
| Group claim empty on the free tier (Entra) | Expected — Entra gates group claims behind P1 | Use the App Role path above instead of group claims |
groups in /auth/whoami is empty (authentik) | The property mapping isn't "Selected" on the provider, the attribute URI is wrong, or the session predates a mapping change | Check the mapping URI + Selected flag; log out and back in — groups are baked into the session at login |
Login works but role is always reader despite a matching group | Older chart joined multiple groups with a comma; env-var splitting only respects whitespace, producing one non-matching string | Fixed — current chart templates join groups with spaces. Verify with kubectl exec … env | grep ADMIN_GROUPS |
Source: Demo install, authentik SAML how-to, Customer test runbook
Where to go next
- Operator handbook — the full install → upgrade → backup → SSO → sizing → troubleshoot walkthrough this page is one stop on
- Runbooks — steady-state Prometheus alert playbooks
- Operator troubleshooting — Kubernetes-operator reconcile-failure reason codes
- Single sign-on (SAML) — SP configuration model and per-IdP setup guides