Rollout troubleshooting
This page is a consolidated collection of rollout-time findings: the kind of problem that shows up once, while installing or upgrading, not as a steady-state alert. If OrbitalReg is already running and something is wrong now, you want Runbooks (Prometheus alerts) or Operator troubleshooting (reconcile failures) instead — this page is for the install/upgrade window itself.
Every entry below was hit for real during one of the project's own rollouts (trial, demo, or a customer test) — see the source runbook linked at the end of each section for the full walkthrough it came from.
Cluster / networking
| Symptom | Cause | Fix |
|---|---|---|
Pods CrashLoop with "connection refused" to a Running pod (api→postgres, bucket-job→minio) | k3s enforces NetworkPolicies (unlike a local make dev); the chart's default-deny blocks a separately-deployed MinIO | Trial/demo overlays set networkPolicy.enabled: false. On an older chart, add --set networkPolicy.enabled=false |
cannot patch StatefulSet … Forbidden | StatefulSet fields are immutable | Fresh namespace/install, or kubectl -n <ns> delete sts orbitalreg-postgres --cascade=orphan (keeps the PVC) then helm upgrade. Cleanest fix: move to CNPG |
| Rebuilt image not picked up (still the old bug) | k3s/containerd caches by tag; :<tag> is pulled IfNotPresent | After re-pushing the same tag: sudo k3s crictl rmi docker.io/<image>:<tag>, then restart the deployment |
ImagePullBackOff | Image not published to the registry the chart points at | Run the publish workflow, or point at the alternate registry (e.g. --set api.image.repository=ghcr.io/… + imagePullSecrets) |
Source: Customer test runbook, Demo install
Storage / state
| Symptom | Cause | Fix |
|---|---|---|
api: password authentication failed for user "orbitalreg" after a reinstall | helm uninstall does not delete the Postgres PVC — a reinstall generates a new password while Postgres keeps the old one | Clear state first: kubectl delete namespace <ns> (removes pods, PVCs and secrets), then reinstall. A real one-shot customer install never hits this |
MinIO ImagePullBackOff (bitnami/minio) | Bitnami removed its free Docker Hub images (2025-08-28) | Use the official minio/minio image plus a one-shot mc bucket Job (chart 0.1.1+) |
Frontend CrashLoop, open() "/run/nginx.pid" … Permission denied | The nginx-unprivileged image (UID 101) can't write /run | Image ships pid /tmp/nginx.pid from 0.1.0+; on an older image, patch a writable /run emptyDir onto the deployment |
Source: Customer test runbook
TLS / DNS
| Symptom | Cause | Fix |
|---|---|---|
certificate stuck READY=False | The A-record doesn't resolve to the ingress IP yet, or ports 80/443 aren't reachable — HTTP-01 fails otherwise | kubectl -n <ns> describe certificate <name>; fix DNS/firewall, then wait for the next HTTP-01 attempt |
admission webhook … risky annotation | An ingress-nginx security-header snippet gets blocked by the admission webhook | Set ingress.securityHeaders.enabled=false |
Source: Customer test runbook, Demo install
SAML / SSO
| Symptom | Cause | Fix |
|---|---|---|
Pod CrashLoop, log fetch idp metadata | idpMetadataURL wrong/placeholder, or the IdP isn't reachable (check egress) | Verify the URL resolves and returns metadata from inside the cluster network |
Pod CrashLoop, log load sp keypair: cert: no PEM block | SAML enabled but the SP keypair secret is empty | Delete the stale *-saml secret and helm upgrade — the chart regenerates it (genSelfSignedCert) |
SAML login: InvalidResponse | IdP cert rolled, or the ACS (reply URL) is wrong | ACS must be exactly https://<domain>/saml/acs (https, no trailing slash); re-fetch IdP metadata after a cert rotation |
| 405 right after the IdP login redirect | IdP's "Service Provider Binding" is Redirect (a common default) — the SAML response arrives as a GET, but crewjam's ACS only accepts POST | Set the provider's binding to Post |
404 on /saml/acs | ACS URL has a trailing slash or a different path — crewjam compares it exactly | Set the ACS URL to exactly https://<domain>/saml/acs |
405 when opening /saml/acs directly in a browser | Expected — that's a GET against a POST-only endpoint | Not an error |
Logged in but not admin / 403 on admin pages | Your principal isn't matching auth.adminGroups: Entra/Azure send App Role values or security-group GUIDs, never a display name | Create an App Role with the exact value used in adminGroups (e.g. admin) and assign it, or put the raw email into adminGroups |
| Group claim empty on the free tier (Entra) | Expected — Entra gates group claims behind P1 | Use the App Role path above instead of group claims |
groups in /auth/whoami is empty (authentik) | The property mapping isn't "Selected" on the provider, the attribute URI is wrong, or the session predates a mapping change | Check the mapping URI + Selected flag; log out and back in — groups are baked into the session at login |
Login works but role is always reader despite a matching group | Older chart joined multiple groups with a comma; env-var splitting only respects whitespace, producing one non-matching string | Fixed — current chart templates join groups with spaces. Verify with kubectl exec … env | grep ADMIN_GROUPS |
Source: Demo install, authentik SAML how-to, Customer test runbook
Where to go next
- Operator handbook — the full install → upgrade → backup → SSO → sizing → troubleshoot walkthrough this page is one stop on
- Runbooks — steady-state Prometheus alert playbooks
- Operator troubleshooting — Kubernetes-operator reconcile-failure reason codes
- Single sign-on (SAML) — SP configuration model and per-IdP setup guides