Operations
View as MarkdownThis page covers ongoing operations once the Ory stack is deployed and your IdP is connected.
Add additional OAuth2 clients
By default the ory-stack module registers a single OAuth2Client in Hydra for the
Materialize Console. If you have other internal applications that
should authenticate through the same Hydra instance, you can register
additional clients using Hydra Maester’s OAuth2Client CRDs.
Apply a manifest like:
apiVersion: hydra.ory.sh/v1alpha1
kind: OAuth2Client
metadata:
name: my-internal-app
namespace: ory
spec:
clientName: My Internal App
grantTypes: ["authorization_code", "refresh_token"]
responseTypes: ["code", "id_token"]
scope: "openid profile email offline"
redirectUris:
- "https://my-app.example.com/auth/callback"
secretName: my-internal-app-credentials
tokenEndpointAuthMethod: "client_secret_basic"
# Keep skipConsent: false. The consent handler injects the identity's email
# and groups claims; skipping it issues tokens Materialize rejects.
skipConsent: false
Hydra Maester watches for these resources and registers the client with
Hydra. The generated client_id and client_secret are written to the
secretName Kubernetes Secret in the same namespace.
To read the credentials:
kubectl get secret my-internal-app-credentials -n ory \
-o jsonpath='{.data.CLIENT_ID}' | base64 -d
kubectl get secret my-internal-app-credentials -n ory \
-o jsonpath='{.data.CLIENT_SECRET}' | base64 -d
Rotate the license key
When your Materialize license key approaches expiry or you receive a new one with updated entitlements:
- Update
license_keyinterraform.tfvars. - Run
terraform apply.
The imagePullSecret in the ory namespace gets updated with the new
JWT. Pods don’t roll automatically, but the next time they restart (or
the next image pull) they’ll use the new credentials. To force an
immediate roll:
kubectl rollout restart deployment kratos hydra ory-selfservice-ui -n ory
kubectl rollout restart deployment polis -n ory # only if enable_polis
The Materialize side also picks up the new key on the next operator reconcile.
Manage Kratos identities
Kratos stores user identities in its own PostgreSQL database. You can inspect and manage them via Kratos’s admin API.
Get the in-cluster admin URL:
kubectl port-forward -n ory svc/kratos-admin 4434:4434
List identities:
curl -s http://localhost:4434/admin/identities | jq .
Get a specific identity:
curl -s http://localhost:4434/admin/identities/<id> | jq .
Lock a user out (disable login):
curl -X PATCH http://localhost:4434/admin/identities/<id> \
-H "Content-Type: application/json-patch+json" \
-d '[{"op": "replace", "path": "/state", "value": "inactive"}]'
See the Kratos admin API reference for the full set of operations.
Manage Polis SAML connections and SCIM directories
Polis admin operations go through its admin API. The Polis admin web UI
is not exposed by default (the module sets hosted = false).
Get the admin API key:
POLIS_API_KEY=$(kubectl get secret -n ory polis-config \
-o jsonpath='{.data.API_KEYS}' | base64 -d)
List SAML connections:
curl -s -H "Authorization: Api-Key $POLIS_API_KEY" \
"https://<your-polis-hostname>/api/v1/sso?tenant=<customer-name>&product=materialize" | jq .
Update a SAML connection (for example, to refresh the IdP metadata XML after a certificate rotation):
curl -X PATCH "https://<your-polis-hostname>/api/v1/sso?tenant=<customer-name>&product=materialize&name=okta-saml" \
-H "Authorization: Api-Key $POLIS_API_KEY" \
-H "Content-Type: application/x-www-form-urlencoded" \
--data-urlencode "rawMetadata=$(cat new-saml-metadata.xml)"
List SCIM directories:
curl -s -H "Authorization: Api-Key $POLIS_API_KEY" \
"https://<your-polis-hostname>/api/v1/dsync?tenant=<customer-name>&product=materialize" | jq .
Delete a SCIM directory (use cautiously, this disconnects the IdP push target):
curl -X DELETE -H "Authorization: Api-Key $POLIS_API_KEY" \
"https://<your-polis-hostname>/api/v1/dsync?tenant=<customer-name>&product=materialize&directoryId=<id>"
If you want to enable the multi-tenant admin UI for hands-on management instead of the API, set in tfvars:
polis_helm_values = {
polis = { hosted = true }
}
and re-apply. The UI is then available at
https://<your-polis-hostname>/admin/auth/login. You’ll need to
configure a NextAuth provider for login (see
Polis hosted-mode docs).
Unlock the Polis admin UI without SMTP
Once the admin UI is up, prefer it over the direct API calls in Configure identity providers for registering SAML connections and SCIM directories going forward.
The default admin login flow uses an email magic link, which requires
SMTP to be configured. If you don’t want to run SMTP just to access the
admin plane, Polis exposes a built-in reserved tenant
(_jackson_boxyhq / _jackson_admin_portal) whose sole purpose is to
gate the admin UI login on a SAML IdP you already have. Registering
your customer’s IdP there lets operators sign into the admin UI via
their own SSO with no mail server involved.
Reuse the same Okta (or other) SAML app you configured for user login, then register a second connection under the reserved tenant:
curl -sS -H "Authorization: Api-Key $POLIS_API_KEY" \
-X POST "https://<your-polis-hostname>/api/v1/connections" \
--data-urlencode "tenant=_jackson_boxyhq" \
--data-urlencode "product=_jackson_admin_portal" \
--data-urlencode "name=admin-bootstrap" \
--data-urlencode "encodedRawMetadata=$(base64 < okta-metadata.xml | tr -d '\n')" \
--data-urlencode "defaultRedirectUrl=https://<your-polis-hostname>/admin/sso-connection" \
--data-urlencode 'redirectUrl=["https://<your-polis-hostname>/api/auth/callback/boxyhq-saml"]'
Then browse to https://<your-polis-hostname>/admin/auth/login and
click “Sign in with SAML”. The flow bounces through your IdP and
lands you at the Polis admin dashboard.
For a production deployment, register a separate SAML app on the IdP side for admin access (rather than reusing the end-user app), so you can control admin membership independently.
Disable registration in Kratos
By default Kratos allows users to register new identities through the selfservice UI. For production deployments where users come exclusively from your IdP, disable registration via Helm values in tfvars:
kratos_helm_values = {
kratos = {
config = {
selfservice = {
flows = {
registration = {
enabled = false
}
}
}
}
}
}
Apply. The “Sign up” link disappears from the login screen.
Back up the Ory PostgreSQL database
The Ory components (Kratos, Hydra, Polis) store identities, OAuth2 clients, sessions, SAML connections, and SCIM directory state in a PostgreSQL database. Loss of this database means:
- All Kratos identities are gone (users will need to re-register / be re-provisioned via SCIM on next login)
- All issued Hydra tokens are invalidated
- All Polis SAML connections and SCIM directories have to be re-created
Each cloud’s database module enables managed automated backups by
default (backup_retained_backups = 35 on Cloud SQL,
backup_retention_days = 35 on Flexible Server and RDS). For
production deployments, verify the retention period matches your DR
requirements and consider running periodic restore drills to confirm
the backups are usable.
Monitor the stack
If you deployed with enable_observability = true (the default), the
example provisions Prometheus and Grafana scraping Materialize and the
Ory pods. The Ory Helm charts emit standard metrics including request
counts and latencies per endpoint, which you can wire into your
existing alerting.
Key signals to alert on:
- Hydra
/oauth2/token5xx rate (token issuance failing) - Kratos
/sessions/whoami5xx rate (session validation failing) - Polis OIDC callback errors (SAML assertions failing)
- Pod restart counts on any Ory component
- PostgreSQL connection failures from any Ory component (suggests DB saturation or networking issues)