Zitadel Setup — Operator Runbook

This doc covers everything an operator needs to set up Zitadel for ShiftIt across three deployment shapes: local dev (one command via scripts/zitadel-up.sh), Zitadel Cloud (managed SaaS), and self-hosted Zitadel (the compose service block in deploy/docker/docker-compose.yml:61-76). It also documents the environment-variable contract between Zitadel and ShiftIt, how to onboard a new tenant, and a troubleshooting matrix of the common failure modes.

Story 2.0 ships the local-dev half (the up/down scripts + this doc); Story 2.2 will switch scripts/run-backend.sh and the frontend OIDC client off scripts/mock-jwks.py and onto whatever Zitadel the operator points it at.

1. Local dev quickstart

# Spin up Zitadel + its own Postgres (90 seconds, idempotent):
scripts/zitadel-up.sh

# Tear it down (idempotent):
scripts/zitadel-down.sh
# Or keep generated artifacts for inspection:
scripts/zitadel-down.sh --keep-vars

What the up script produces:

URL What it serves

URL	What it serves
`http://localhost:8090/ui/console`	Zitadel admin console (web UI)
`http://localhost:8090`	OIDC issuer (matches `iss` claim on tokens)
`http://localhost:8090/oauth/v2/keys`	JWKS endpoint the backend’s `JwksCache` polls
`http://localhost:8090/.well-known/openid-configuration`	OIDC discovery doc

http://localhost:8090/ui/console

Zitadel admin console (web UI)

http://localhost:8090

OIDC issuer (matches iss claim on tokens)

http://localhost:8090/oauth/v2/keys

JWKS endpoint the backend’s JwksCache polls

http://localhost:8090/.well-known/openid-configuration

OIDC discovery doc

Bootstrap admin login (created on first boot via ZITADEL_FIRSTINSTANCE_* env vars): admin / Admin123! (login name resolves to admin@<orgdomain> — Zitadel appends the org’s primary domain when you log in via the console).

The script also writes a downstream-consumable env file at dev/zitadel-vars.env:

ZITADEL_ISSUER_URL=http://localhost:8090
ZITADEL_DOMAIN=http://localhost:8090
ZITADEL_AUTH_URL=http://localhost:8090
ZITADEL_JWKS_URL=http://localhost:8090/oauth/v2/keys
ZITADEL_DEMO_ORG_ID=...        # zitadel-generated org UUID
ZITADEL_DEMO_PROJECT_ID=...    # zitadel-generated project UUID
ZITADEL_ADMIN_CLIENT_ID=...
ZITADEL_EMPLOYEE_CLIENT_ID=...
ZITADEL_CUSTOMER_CLIENT_ID=...
ZITADEL_EXPECTED_AUDIENCE=...  # = the project_id; see §3 + the backend TODO

The trailing comment block in the file contains a copy-pasteable UPDATE tenants ... statement that reconciles tenants.zitadel_org_id (placeholder demo-org in scripts/seed-demo.sql:11) against the real Zitadel-generated org UUID. The up script attempts this UPDATE automatically when shiftit-pg is running and reachable.

Port allocation in dev

Port	Service
8080	Frontend Trunk dev server (`frontend/Trunk.toml`)
8090	Zitadel (this story’s container)
5433	shiftit-pg (application Postgres)
5434	shiftit-zitadel-pg (Zitadel’s own Postgres)
3010	Backend dev server (`scripts/run-backend.sh`)

Zitadel was originally specced on :8080 but that collides with Trunk. The deviation is documented in Story 2.0’s Change Log.

Seed data parity

The up script seeds Zitadel to match scripts/seed-demo.sql:11, 20-23 so the backend’s tenant + user FKs resolve cleanly once Story 2.2 wires real JWT validation in:

Username Email Role Password

Username	Email	Role	Password
`demo-admin`	admin@demo.test	admin	`Demo123!`
`demo-employee`	emma@demo.test	employee	`Demo123!`
`demo-customer`	colin@demo.test	customer	`Demo123!`

demo-admin

admin@demo.test

admin

Demo123!

demo-employee

emma@demo.test

employee

Demo123!

demo-customer

colin@demo.test

customer

Demo123!

All seed users have email.isVerified=true and password.changeRequired=false; MFA is not enforced (default org login policy). Every user has a tenant_id metadata key set to the demo tenant UUID (00000000-0000-0000-0000-000000000001), copied into a tenant_id claim on token issuance via a complement-token action (forward-compat for Story 2.2 — see §3).

2. Environment-variable contract

Every variable below appears in dev/zitadel-vars.env. Story 2.2 will pick which subset the backend / frontend read from at runtime vs. build time.

Variable Read by Consumed at Production equivalent

Variable	Read by	Consumed at	Production equivalent
`ZITADEL_ISSUER_URL`	backend	`backend/src/auth/jwks.rs:113` (`set_issuer`)	`${PROD_ZITADEL_URL}` from the operator’s secrets file
`ZITADEL_DOMAIN`	backend	`JwksCache::new(zitadel_domain)` — `backend/src/auth/jwks.rs:38`	same as above (alias)
`ZITADEL_AUTH_URL`	frontend (build-time)	`frontend/src/customer/auth/oauth_client.rs:20` (`option_env!`)	injected at frontend build by CI
`ZITADEL_JWKS_URL`	informational	not directly read; derived from `ZITADEL_DOMAIN` at runtime	derived
`ZITADEL_DEMO_ORG_ID`	one-shot SQL reconcile	`tenants.zitadel_org_id` column (matches `auth/middleware.rs:101-104`)	set per-tenant during tenant onboarding (see §5)
`ZITADEL_DEMO_PROJECT_ID`	backend (audience check)	future `validation.set_audience(...)` — `backend/src/auth/jwks.rs:8`	same key, different value per environment
`ZITADEL_*_CLIENT_ID`	frontend (build-time)	`frontend/src/customer/auth/oauth_client.rs:10` (today hard-coded)	one value per frontend build; Story 2.2 swaps the const
`ZITADEL_EXPECTED_AUDIENCE`	backend	future audience claim assertion — same TODO as above	= project_id

ZITADEL_ISSUER_URL

backend

backend/src/auth/jwks.rs:113 (set_issuer)

${PROD_ZITADEL_URL} from the operator’s secrets file

ZITADEL_DOMAIN

backend

JwksCache::new(zitadel_domain) — backend/src/auth/jwks.rs:38

same as above (alias)

ZITADEL_AUTH_URL

frontend (build-time)

frontend/src/customer/auth/oauth_client.rs:20 (option_env!)

injected at frontend build by CI

ZITADEL_JWKS_URL

informational

not directly read; derived from ZITADEL_DOMAIN at runtime

derived

ZITADEL_DEMO_ORG_ID

one-shot SQL reconcile

tenants.zitadel_org_id column (matches auth/middleware.rs:101-104)

set per-tenant during tenant onboarding (see §5)

ZITADEL_DEMO_PROJECT_ID

backend (audience check)

future validation.set_audience(...) — backend/src/auth/jwks.rs:8

same key, different value per environment

ZITADEL_*_CLIENT_ID

frontend (build-time)

frontend/src/customer/auth/oauth_client.rs:10 (today hard-coded)

one value per frontend build; Story 2.2 swaps the const

ZITADEL_EXPECTED_AUDIENCE

backend

future audience claim assertion — same TODO as above

= project_id

The backend’s JwksCache::validate_token (backend/src/auth/jwks.rs:77-122) is the canonical JWT validation flow today. It sets issuer (set_issuer) but does NOT yet validate audience. Story 2.2 closes that gap by wiring ZITADEL_EXPECTED_AUDIENCE.

3. Setting up Zitadel Cloud

For most production deployments, Zitadel Cloud (the managed SaaS) is the path of least friction. Steps below are precise UI paths; screenshots intentionally omitted (they rot fast).

Create a Zitadel Cloud account at https://zitadel.com/signin if you don’t have one. You get one free instance.
Create an instance. From the Customer Portal: Customer Portal > New Instance > pick a region close to your backend. Note the instance URL — it’s the value of ZITADEL_ISSUER_URL (e.g. https://shiftit-prod-abc123.zitadel.cloud).
Create an organization for the customer. Zitadel Console > Organizations > Default > New. Name = your customer’s legal name (e.g. Acme GmbH). Copy the org’s ID from the URL bar after creation; this is tenants.zitadel_org_id for that tenant.
Create the ShiftIt project. Zitadel Console > [pick the org] > Projects > New. Name ShiftIt. After creation, open the project’s General settings and toggle ON both:
- Assert Roles on Authentication (= projectRoleAssertion)
- Check Roles at Authentication (= projectRoleCheck)
Create the three OIDC applications under the project (matches §1’s table). For each:
- Project > [ShiftIt] > General > New Application > Web > continue
- Name: shiftit-admin / shiftit-employee / shiftit-customer
- Authentication method: PKCE (NOT Basic / Post / None)
- Redirect URIs:
  - admin → https://app.shiftit.example/auth/admin/callback
  - employee → https://app.shiftit.example/auth/employee/callback
  - customer → https://app.shiftit.example/auth/callback
- Post-logout URIs: same hostname, paths /admin, /employee, /
- Save → copy the client ID. These go into ZITADEL_ADMIN_CLIENT_ID etc.
Create the three project roles. Project > [ShiftIt] > Roles > New. Three roles, keys exactly admin, employee, customer.
Onboard the first admin user. Users > New > Human > username + email + password. Mark email verified (skip the verification mail flow for the bootstrap user). Then `Authorizations > New > pick the user, the ShiftIt project, the admin role

Save`.
Copy the values into the operator’s .env (used by your deploy tool of choice — deploy/docker/docker-compose.yml reads ZITADEL_DOMAIN from there).

4. Setting up self-hosted Zitadel

The production compose at deploy/docker/docker-compose.yml:61-76 already includes a zitadel: service block ready to run. Two operator-side prerequisites:

Generate a masterkey. This is the symmetric key Zitadel uses to encrypt secrets at rest in its own database. It is unchangeable post-install; losing it means re-initializing Zitadel and losing every user / org / token.
```
openssl rand -base64 24 | head -c 32
```
Store the output in your secrets manager and export it as ZITADEL_MASTERKEY for the compose stack.
Set ZITADEL_DOMAIN to the external hostname Zitadel will be reached at (e.g. https://auth.shiftit.example). The compose service reads it as ZITADEL_EXTERNALDOMAIN and bakes it into every issuer claim.

Once the compose stack is up, the rest of the setup mirrors §3 (create org → project → 3 apps → 3 roles → first admin user). You can drive that via the console UI, or replay the management-API calls from scripts/zitadel-up.sh — the calls are the same; only the issuer URL and the bootstrap PAT path differ.

5. Onboarding a new tenant

For every new ShiftIt tenant (i.e. every new customer organization on a SaaS deployment, or the only org on a single-tenant self-host):

Create a Zitadel organization for the tenant (UI or API).
Note the org’s ID (UI: bottom of the org’s detail page; API: POST /v2/organizations return value, organizationId).

Insert the tenants row in shiftit-pg with zitadel_org_id set to that UUID:

INSERT INTO tenants (id, zitadel_org_id, name, slug, settings)
VALUES (gen_random_uuid(), '<the-zitadel-org-id>', 'Acme GmbH', 'acme',
    '{"time_tracking_enabled": true, "default_claiming_mode": "open", ...}');

Under the ShiftIt project, the three OIDC apps and three project roles created at install time are shared across all orgs. Per-org admin onboarding is just Users > New > Human under the org, plus an Authorizations grant for the admin role.

After step 3 the backend’s auth/middleware.rs:101-104 will resolve the new tenant from any JWT issued for a user under that Zitadel org.

6. Troubleshooting

Symptom Likely cause Fix

Symptom	Likely cause	Fix
Every API call returns 401 with no further info	JWKS URL is wrong, or the JWKS cache failed to populate.	`curl ${ZITADEL_JWKS_URL}` should return a JSON object with a non-empty `keys` array. If empty, the web-key feature is off (see "JWKS empty" below).
Backend log: `kid not in JWKS even after refresh` (`auth/jwks.rs:100`)	The token was signed by a key the backend doesn’t know about. Either the issuer URL is misconfigured (`ZITADEL_DOMAIN` ≠ the actual issuer) or the JWKS endpoint is unreachable from the backend container.	Verify `ZITADEL_DOMAIN` matches the `iss` claim by base64-decoding any JWT payload (`jwt.io` or `jwt-cli`). Verify the backend container can reach the JWKS URL — in compose, both must share a network.
Backend log: `JWT signature/claim validation failed` (`auth/jwks.rs:117`)	Clock skew, expired token, or a wrong-issuer token.	Check `date` on the host vs. the container (`docker exec shiftit-zitadel date`). Skew > 5s exceeds the leeway set at `auth/jwks.rs:112`.
Backend log: `no tenant for zitadel org_id` (`auth/middleware.rs:109`)	The `tenants.zitadel_org_id` column doesn’t match the `urn:zitadel:iam:org:id` claim.	For local dev: rerun `scripts/zitadel-up.sh` (it issues the SQL UPDATE). For production: apply the SQL UPDATE manually using the org ID from the Zitadel console URL.
Backend log: `rejected unknown role claim` (`auth/middleware.rs:116`)	Token’s `role` claim is empty or unknown. Today the backend reads a flat `role` string; Zitadel emits `urn:zitadel:iam:org:project:roles` (a structured claim).	Story 2.2 reconciles claim shape. Until then, the complement-token action seeded by `zitadel-up.sh` exists to demonstrate forward-compat custom-claim wiring.
Token endpoint returns `invalid_client` on `/oauth/v2/token`	Frontend’s `CLIENT_ID` const is hard-coded `"shiftit-customer"` at `frontend/src/customer/auth/oauth_client.rs:10`, but Zitadel mints an opaque per-app `clientId` (the value captured into `ZITADEL_CUSTOMER_CLIENT_ID` in `dev/zitadel-vars.env`) independent of the app’s display name.	Customer OAuth against this dev Zitadel will return `invalid_client` until Story 2.2 lands — that story switches `oauth_client.rs:10` to read the Zitadel-generated id at build time. There is no workaround at the Zitadel side (the management API does not accept operator-supplied client_ids for new OIDC apps); the fix is purely in the frontend, scope-locked to 2.2. For backend-only smoke work in the meantime, keep `scripts/run-backend.sh`'s mock-jwks path.
Refresh token rejected	`offline_access` scope wasn’t granted, or the app’s grantTypes don’t include `OIDC_GRANT_TYPE_REFRESH_TOKEN`.	Confirm both: (a) every `/oauth/v2/authorize` request includes `scope=...offline_access...` (the customer flow does — see `frontend/src/customer/auth/oauth_client.rs:11`); (b) the app’s grant types include refresh_token.
JWKS empty (`{"keys":[]}`)	The web-key feature is disabled at the instance level (v2.71 default).	`scripts/zitadel-up.sh` enables it automatically via `PUT /v2beta/features/instance {"webKey":true}`. For Cloud / self-host, do this once per instance via the management API or the console under Instance Settings.
`Web key feature disabled (WEBKEY-Ohx6E)`	Trying to manually create a web key without enabling the feature first.	Enable the feature first (see above).

Every API call returns 401 with no further info

JWKS URL is wrong, or the JWKS cache failed to populate.

curl ${ZITADEL_JWKS_URL} should return a JSON object with a non-empty keys array. If empty, the web-key feature is off (see "JWKS empty" below).

Backend log: kid not in JWKS even after refresh (auth/jwks.rs:100)

The token was signed by a key the backend doesn’t know about. Either the issuer URL is misconfigured (ZITADEL_DOMAIN ≠ the actual issuer) or the JWKS endpoint is unreachable from the backend container.

Verify ZITADEL_DOMAIN matches the iss claim by base64-decoding any JWT payload (jwt.io or jwt-cli). Verify the backend container can reach the JWKS URL — in compose, both must share a network.

Backend log: JWT signature/claim validation failed (auth/jwks.rs:117)

Clock skew, expired token, or a wrong-issuer token.

Check date on the host vs. the container (docker exec shiftit-zitadel date). Skew > 5s exceeds the leeway set at auth/jwks.rs:112.

Backend log: no tenant for zitadel org_id (auth/middleware.rs:109)

The tenants.zitadel_org_id column doesn’t match the urn:zitadel:iam:org:id claim.

For local dev: rerun scripts/zitadel-up.sh (it issues the SQL UPDATE). For production: apply the SQL UPDATE manually using the org ID from the Zitadel console URL.

Backend log: rejected unknown role claim (auth/middleware.rs:116)

Token’s role claim is empty or unknown. Today the backend reads a flat role string; Zitadel emits urn:zitadel:iam:org:project:roles (a structured claim).

Story 2.2 reconciles claim shape. Until then, the complement-token action seeded by zitadel-up.sh exists to demonstrate forward-compat custom-claim wiring.

Token endpoint returns invalid_client on /oauth/v2/token

Frontend’s CLIENT_ID const is hard-coded "shiftit-customer" at frontend/src/customer/auth/oauth_client.rs:10, but Zitadel mints an opaque per-app clientId (the value captured into ZITADEL_CUSTOMER_CLIENT_ID in dev/zitadel-vars.env) independent of the app’s display name.

Customer OAuth against this dev Zitadel will return invalid_client until Story 2.2 lands — that story switches oauth_client.rs:10 to read the Zitadel-generated id at build time. There is no workaround at the Zitadel side (the management API does not accept operator-supplied client_ids for new OIDC apps); the fix is purely in the frontend, scope-locked to 2.2. For backend-only smoke work in the meantime, keep scripts/run-backend.sh's mock-jwks path.

Refresh token rejected

offline_access scope wasn’t granted, or the app’s grantTypes don’t include OIDC_GRANT_TYPE_REFRESH_TOKEN.

Confirm both: (a) every /oauth/v2/authorize request includes scope=...offline_access... (the customer flow does — see frontend/src/customer/auth/oauth_client.rs:11); (b) the app’s grant types include refresh_token.

JWKS empty ({"keys":[]})

The web-key feature is disabled at the instance level (v2.71 default).

scripts/zitadel-up.sh enables it automatically via PUT /v2beta/features/instance {"webKey":true}. For Cloud / self-host, do this once per instance via the management API or the console under Instance Settings.

Web key feature disabled (WEBKEY-Ohx6E)

Trying to manually create a web key without enabling the feature first.

Enable the feature first (see above).

TODO 2.2: the jwks.rs log lines aren’t yet pinned

auth/jwks.rs:78-100 already emits the warnings referenced above. Story 2.2’s refactor may shift line numbers; if a log message moves, update this matrix accordingly.

7. Other dev dependencies

ShiftIt’s dev environment has a few moving parts beyond Zitadel:

Mailpit STARTTLS smoke (docker run -d --name shiftit-mailpit -p 1025:1025 -p 8025:8025 axllent/mailpit) — currently an outstanding manual task before Story 3.2 (per-locale email templates). See CLAUDE.md "Outstanding manual task" section.
scripts/mock-jwks.py — still used by scripts/run-backend.sh for backend-only unit work that doesn’t need a real OIDC issuer. Story 2.2 will switch run-backend.sh to point at the real Zitadel from this story.
_bmad-output/implementation-artifacts/2-3-customer-auth-substrate.md — Story 2.3’s "Manual Smoke Plan" describes the customer-side login smoke this dev environment unblocks. Once Story 2.2 lands, that runbook is the end-to-end verification.