Tenant Authorization and Tenant Generations
Every OctoMesh access token is bound to exactly one tenant. This page explains how the services check that binding, what happens to tokens when a tenant is deleted, recreated or restored, what clients and integrators have to do, and how operators of a self-hosted installation configure and switch on the checks.
The Two Checks
| Check | Question | Where it runs |
|---|---|---|
| Route tenant | Was the token issued for the tenant the request addresses? | Every OctoMesh API service with a tenant route (/{tenantId}/...), including the Identity Service's own API, and CK method calls. Platform services runs no tenant check. |
| Tenant generation | Was the token issued for this tenant, or for an earlier tenant that had the same id? | The same places, plus the Identity Service's token endpoint (refresh tokens, authorization codes, device codes, token exchange, on-behalf-of) |
Both compare against the tenant_id claim of the access token. The allowed_tenants claim only lists the tenants a
user may switch to (for a tenant picker); it never grants access to an API.
Route Tenant Check
A request to /{tenantId}/... is allowed when the route tenant equals the token's tenant_id. Otherwise the service
answers, in Enforce mode, 403 Forbidden with a JSON body {"message": "..."} that names the reason. There are two exceptions:
- Parent-tenant administration. Endpoints that are explicitly marked as tenant administration (for example backup and restore of a child tenant) accept a user token of the parent tenant. Data routes (entities, queries, stream data, and GraphQL with the one exception below) and the identity data of a child tenant (users, roles, groups, clients) are never among them. The exception: on the GraphQL endpoint a request made exclusively of the backup, restore and archive data methods or of reads of their operations qualifies, see Parent-Tenant Administration through CK Methods.
- Allow-listed service clients. A client-credentials client listed in
TenantAuthorization:CrossTenantServiceClientIdsmay address other tenants. The list is meant to stay empty; every entry is a permanent exception.
Client-credentials tokens are checked like user tokens (ServiceTokenEnforcement, Enforce by default; the octo-mesh
chart runs identity in LogOnly unless a value is set, see Configuration): their tenant_id is the tenant the token was requested for with
acr_values=tenant:<tenantId>. A client-credentials login without acr_values gets a token of the system tenant and
is refused on every other tenant's routes (see Client Credentials).
Parent-Tenant Administration through CK Methods
An administrator of a parent tenant backs up, restores and exports or imports the archive data of a child tenant
through the same CK methods the child's own administrator uses, and follows the operations through runtime.operations
— exactly what the REST jobs endpoint {tenantId}/v1/jobs allows. The rule mirrors the REST rule and grants nothing more:
| Allowed with a user token of the parent tenant P on tenant C | |
|---|---|
| Methods | Only System.Bot/TenantRepository.Dump-1, Restore-1, ExportArchiveData-1 and ImportArchiveData-1 (GraphQL runtime { systemBotTenantRepositorys { dump / restore / exportArchiveData / importArchiveData } }). Every other method on C is METHOD_FORBIDDEN with reason = tenant-not-allowed. |
| Operations | runtime.operations on C lists and reads only the operations of these four methods: by handle (operationId) for the caller who started it, every caller's with allOwners: true for an administrator (AdminPanelManagement) of P. Other operations of C are not found. |
| Which tenants | C must be a registered direct child of P. The system tenant reaches every tenant. A grandchild, a sibling, the parent of the caller's tenant, and a deleted or not yet registered tenant are refused. |
| Roles | The role the methods declare, TenantManagement, in the parent tenant (the same role the REST jobs endpoint requires). Without it: METHOD_FORBIDDEN with reason = not-authorized, and no operations are listed. |
| Tokens | User tokens only. A client-credentials token of P is refused on C as before; only clients on TenantAuthorization:CrossTenantServiceClientIds address other tenants. |
| Tenant generation | The token's tenant_gen is checked against the parent tenant P. A token of P issued before P was deleted and created again is refused like on P itself. |
| HTTP gate | A request to /tenants/{C}/graphQl passes the route tenant check only when every field of the request is one of the method calls above or a runtime.operations read: one JSON POST, no batch, no fragments, no query string. A request that mixes an allowed and another field is refused as a whole. Send one plain operation per request. |
Every allowed call is logged at Information level with the subject, the client, both tenants and the method. The file
of a backup is downloaded from the downloadPath of the operation result ({C}/v1/jobs/download?id=… of the bot
service), which accepts the parent's token for the same reason.
The Identity Service Checks the Route Tenant Too
The Identity Service runs the same route tenant check on its own tenant API (users, roles, groups, clients, identity
providers and the other /{tenantId}/... routes) for bearer tokens. Earlier versions stopped a token of another tenant
there only by the role requirements of the individual endpoints.
What integrators have to change: a client that calls the Identity Service API of tenant B with a token issued for tenant A gets 403. Use a token of the tenant you address:
- Users and interactive apps: sign in to the target tenant (the login with
acr_values=tenant:<tenantId>), or exchange the current token for one of the target tenant with the token exchange. - Service clients: request one token per tenant, each with
acr_values=tenant:<tenantId>; the client must be registered in that tenant (for example through client mirroring into child tenants). - Parent-tenant administrators who manage the users or clients of a child tenant do so with a token of the child tenant, not with the parent's token.
Operators can run this check in a log-only mode first, see Configuration.
Tenant Generations (tenant_gen)
A tenant id can be used again: when tenant my-project is deleted and a new tenant my-project is created, a token of
the deleted tenant would still match the tenant_id of the new one until it expires. To tell them apart, every access
token carries the claim tenant_gen, the generation of its tenant: the time the tenant was created, as Unix
milliseconds in a decimal string (for example "1794750381540"). ID tokens and the userinfo response do not carry it.
| Event | Generation | Effect on existing tokens |
|---|---|---|
| Create a tenant | new generation | — |
| Restore a backup over the existing tenant | kept | Access tokens stay valid. Refresh tokens and codes are those of the backup: the ones issued after the backup was taken are gone, and their clients sign in again (invalid_grant). |
| Delete the tenant | the generation ends | Refresh tokens, authorization codes and device codes of the tenant are gone at once (they are stored in the tenant's database). Access tokens are self-contained and stay valid until they expire, but with the check in Enforce the services refuse them as soon as the tenant is recreated. |
| Delete and create again with the same id | new generation | Tokens of the deleted tenant are never accepted by the new tenant (with the check in Enforce). |
| Delete, create again and restore a backup of the deleted tenant | the new generation of the create | Tokens, refresh tokens and codes of the deleted tenant are refused, also those that the restored data brings back (with the check in Enforce). |
The system tenant is never refused by the generation check. Tenants without a recorded creation time (created by older platform versions) carry no generation and are not checked.
Run the generation check in Enforce before you restore a backup of a deleted tenant into a new tenant with the same id.
In LogOnly, a refresh token from that backup that was issued before the Identity Service stored generations is
accepted on its next refresh and from then on carries the generation of the new tenant.
What the Check Refuses
With the check in Enforce:
| Caller | Token of an earlier tenant, or token without tenant_gen | Generation cannot be read |
|---|---|---|
Service API (/{tenantId}/...) | 403 Forbidden with a JSON message | 503 Service Unavailable |
| CK method call | METHOD_FORBIDDEN, reason tenant-not-allowed | METHOD_HANDLER_FAILED |
| Identity token endpoint: refresh token, authorization code, device code, token exchange, on-behalf-of | invalid_grant (a refused refresh token is not rotated) | server_error |
In LogOnly (the default) none of these requests is refused; each one is counted in the tenant authorization metrics,
and a token of an earlier tenant or without tenant_gen is logged as "would be denied" (at most once a minute per client;
a failed lookup is only counted).
Tokens issued before the Identity Service stamped tenant_gen carry no generation. They are refused in Enforce as
well, which is why an environment switches to Enforce only after such tokens have expired (see
Recommended Rollout).
What Clients Have to Do
| Situation | What to do |
|---|---|
Interactive user (Refinery Studio, apps, octo-cli LogIn) gets 403 or invalid_grant after the tenant was recreated | Sign in again. The new sign-in issues tokens of the new tenant. |
A client's refresh fails with invalid_grant | Discard the stored refresh token and run the sign-in again. |
| Client-credentials client gets 403 | Request a new token (acr_values=tenant:<tenantId>). If the tenant was recreated, the client must exist in the new tenant first (registered again or mirrored again). |
| Long-running service that caches tokens | Do not keep a token past a tenant delete. On a 403 for a tenant that was recreated, drop the cached token and request a new one. |
Configuration
Self-hosted operators configure the checks per service with the configuration section TenantAuthorization (environment
variable prefix OCTO_, double underscore between the levels).
| Configuration key | Environment variable | Values | Default | Services |
|---|---|---|---|---|
TenantAuthorization:TenantGenerationEnforcement | OCTO_TENANTAUTHORIZATION__TENANTGENERATIONENFORCEMENT | LogOnly, Enforce, Disabled | LogOnly | Every service that runs the check: identity (its API and its grants), asset repository, bot, communication controller, report services, MCP service, AI services |
TenantAuthorization:UserTokenEnforcement | OCTO_TENANTAUTHORIZATION__USERTOKENENFORCEMENT | LogOnly, Enforce | Enforce in identity, report, MCP and AI services; LogOnly in asset repository and communication controller. The octo-mesh chart renders LogOnly for identity, asset repository and communication controller unless a value is set | Route tenant check of user tokens |
TenantAuthorization:ServiceTokenEnforcement | OCTO_TENANTAUTHORIZATION__SERVICETOKENENFORCEMENT | LogOnly, Enforce, Disabled | Enforce in the service; the octo-mesh chart renders LogOnly for identity unless a value is set | Route tenant check of client-credentials tokens |
TenantAuthorization:CrossTenantServiceClientIds | OCTO_TENANTAUTHORIZATION__CROSSTENANTSERVICECLIENTIDS__0, __1, ... | client ids; a trailing * matches a prefix | empty | Service clients that may address other tenants |
The modes:
LogOnlychanges no outcome. A request thatEnforcewould refuse is logged as a warning that names the client and the tenants (never the token); the generation check logs at most once a minute per client and outcome and counts every check in the tenant authorization metrics.Enforcerefuses those requests as described above.Disabled(generation check and service tokens only) turns the check off without log or metric. Use it only as an emergency exit and return toLogOnlyafterwards, because it hides which clients would be affected.
The user-token modes of the asset repository and the communication controller (LogOnly by default) are switched on
separately, by the same steps as in Recommended Rollout.
The asset repository additionally guards its file routes (REST, and GraphQL operations on System.Files) with its own
switch Files:TenantGuard (OCTO_Files__TenantGuard, Enforce or LogOnly, default Enforce; an unknown value counts
as Enforce). It refuses a user token of another tenant on those routes even while UserTokenEnforcement is LogOnly,
is independent of the switches on this page and is set and observed on its own.
Set the generation check to the same value on every service of an installation. A partial switch is not unsafe, but the protection is only complete when every service enforces.
Helm Values
In the octo-mesh chart:
tenantAuthorization:
tenantGenerationEnforcement: LogOnly # LogOnly | Enforce | Disabled, all services of the chart
services:
identity:
userTokenEnforcement: LogOnly # LogOnly | Enforce
serviceTokenEnforcement: LogOnly # LogOnly | Enforce | Disabled
# tenantGenerationEnforcement: Enforce # optional override for this service only
| Value | Default | Renders |
|---|---|---|
tenantAuthorization.tenantGenerationEnforcement | LogOnly | OCTO_TENANTAUTHORIZATION__TENANTGENERATIONENFORCEMENT on identity, asset repository, bot, communication controller and AI services |
services.<service>.tenantGenerationEnforcement with <service> = identity, assetRepository, bot, communication, aiServices | unset = the global value | the same variable on that service only |
services.identity.userTokenEnforcement | LogOnly | OCTO_TENANTAUTHORIZATION__USERTOKENENFORCEMENT on identity |
services.identity.serviceTokenEnforcement | LogOnly | OCTO_TENANTAUTHORIZATION__SERVICETOKENENFORCEMENT on identity |
services.assetRepository.userTokenEnforcement, services.communication.userTokenEnforcement | LogOnly | OCTO_TENANTAUTHORIZATION__USERTOKENENFORCEMENT on that service |
The octo-mesh-reporting, octo-mesh-mcp and octo-mesh-ai charts have the same value
tenantAuthorization.tenantGenerationEnforcement (default LogOnly); set it to the same value as in octo-mesh.
Any other value than the ones listed fails the chart render. Platform services and Refinery Studio do not run the checks
and get no value.
Recommended Rollout
Switch one environment at a time, test environments first:
-
Run
LogOnly. Deploy the Identity Service release that stampstenant_genand checks its grants, together with services that run the generation check, all inLogOnly. -
Wait for old tokens to expire. Tokens and refresh tokens issued before that Identity Service release carry no generation and would be refused in
Enforce. Wait at least the longest refresh-token lifetime configured on any client (default 30 days), and at least the longest access-token lifetime if that is larger. Instead of waiting for the refresh tokens you can accept that users with older sessions have to sign in again; the longest access-token lifetime must pass in any case. -
Observe. Check the logs for "would be denied" lines for at least one day before the switch:
This would be denied with TenantGenerationEnforcement=Enforce(generation check, all services and the token endpoint),This would be denied with UserTokenEnforcement=EnforceandThis would be denied with ServiceTokenEnforcement=Enforcefrom the Identity Service (route tenant check). The asset repository and the communication controller log the same user-token lines while their ownuserTokenEnforcementisLogOnly; that is a separate switch with its own values.
Each line names the client. A line for a token of a deleted and recreated tenant is exactly what
Enforcemust refuse. Every other client is fixed first: it requests its tokens withacr_values=tenant:<tenantId>, signs in again, or — only when it genuinely works across tenants — is added toCrossTenantServiceClientIds. The tenant authorization metrics show the same outcomes per client without tenant or user data. -
Switch to
Enforce. Setservices.identity.userTokenEnforcementandservices.identity.serviceTokenEnforcement, andtenantAuthorization.tenantGenerationEnforcementinocto-meshand in the reporting, MCP and AI charts, then roll out the deployments. Right after the rollout run a smoke test: a fresh sign-in, a GraphQL query, a CK method call, a token refresh and a client-credentials call. Watch theDenied: ...log lines for a day. -
Rollback. Set the values back to
LogOnlyand roll out again. Requests that were refused work again at once; no data changes.
Troubleshooting
403 after a tenant was deleted and recreated, or recreated and restored
Symptom: API calls to the tenant answer 403 with the message "Access is denied: the token was issued for an earlier tenant '...' that was deleted and created again, or before the tenant generation was issued. Sign in again.", although the user or client has the right roles.
Cause: the token was issued for the deleted tenant with the same id. With the generation check in Enforce the new
tenant never accepts it. A restore of a backup into the recreated tenant does not change that: the restored tenant
keeps the generation of the create.
Fix: sign in again (users), or request a new token (client-credentials clients). A client that must also exist in the new tenant is registered or mirrored there first.
invalid_grant on token refresh after a restore
Symptom: a token refresh, the redemption of an authorization code or device code, or a token exchange fails with
invalid_grant after the tenant was deleted, created again and restored from a backup.
Cause: the refresh token or code belongs to the earlier tenant. The restore brings the stored grants of the backup back, but the Identity Service refuses grants whose tenant generation does not match the current tenant.
Fix: discard the refresh token and sign in again. A restore over the existing tenant (without delete) keeps the
generation, so the refresh tokens in the backup stay valid; a refresh token issued after the backup was taken is not in
the restored database and fails with invalid_grant as well, for that reason.
403 on the Identity Service API with a token of another tenant
Symptom: calls to /{tenantId}/... endpoints of the Identity Service answer 403, and the message names the token tenant and
the route tenant.
Fix: use a token of the route tenant: sign in to that tenant, use the token exchange, or (service clients) request
the token with acr_values=tenant:<tenantId>. See The Identity Service Checks the Route Tenant Too.
503 from a service with the generation check in Enforce
Symptom: the message reads "The tenant of the token cannot be verified right now. Try again later."
Cause: the service could not read the tenant's generation from the system database. In Enforce it answers 503
instead of accepting the token unchecked. The request succeeds again once the database is reachable; check the
connection between the service and MongoDB.
See Also
- Authentication — flows, tokens and claims
- Tenant Lifecycle — create, restore and delete tenants
- Cross-Tenant Authentication — users of one tenant in another tenant