Skip to main content

Tenant Authorization and Tenant Generations

Every OctoMesh access token is bound to exactly one tenant. This page explains how the services check that binding, what happens to tokens when a tenant is deleted, recreated or restored, what clients and integrators have to do, and how operators of a self-hosted installation configure and switch on the checks.

The Two Checks​

CheckQuestionWhere it runs
Route tenantWas the token issued for the tenant the request addresses?Every OctoMesh API service with a tenant route (/{tenantId}/...), including the Identity Service's own API, and CK method calls. Platform services runs no tenant check.
Tenant generationWas the token issued for this tenant, or for an earlier tenant that had the same id?The same places, plus the Identity Service's token endpoint (refresh tokens, authorization codes, device codes, token exchange, on-behalf-of)

Both compare against the tenant_id claim of the access token. The allowed_tenants claim only lists the tenants a user may switch to (for a tenant picker); it never grants access to an API.

Route Tenant Check​

A request to /{tenantId}/... is allowed when the route tenant equals the token's tenant_id. Otherwise the service answers, in Enforce mode, 403 Forbidden with a JSON body {"message": "..."} that names the reason. There are two exceptions:

  • Parent-tenant administration. Endpoints that are explicitly marked as tenant administration (for example backup and restore of a child tenant) accept a user token of the parent tenant. Data routes (entities, queries, stream data, and GraphQL with the one exception below) and the identity data of a child tenant (users, roles, groups, clients) are never among them. The exception: on the GraphQL endpoint a request made exclusively of the backup, restore and archive data methods or of reads of their operations qualifies, see Parent-Tenant Administration through CK Methods.
  • Allow-listed service clients. A client-credentials client listed in TenantAuthorization:CrossTenantServiceClientIds may address other tenants. The list is meant to stay empty; every entry is a permanent exception.

Client-credentials tokens are checked like user tokens (ServiceTokenEnforcement, Enforce by default; the octo-mesh chart runs identity in LogOnly unless a value is set, see Configuration): their tenant_id is the tenant the token was requested for with acr_values=tenant:<tenantId>. A client-credentials login without acr_values gets a token of the system tenant and is refused on every other tenant's routes (see Client Credentials).

Parent-Tenant Administration through CK Methods​

An administrator of a parent tenant backs up, restores and exports or imports the archive data of a child tenant through the same CK methods the child's own administrator uses, and follows the operations through runtime.operations — exactly what the REST jobs endpoint {tenantId}/v1/jobs allows. The rule mirrors the REST rule and grants nothing more:

Allowed with a user token of the parent tenant P on tenant C
MethodsOnly System.Bot/TenantRepository.Dump-1, Restore-1, ExportArchiveData-1 and ImportArchiveData-1 (GraphQL runtime { systemBotTenantRepositorys { dump / restore / exportArchiveData / importArchiveData } }). Every other method on C is METHOD_FORBIDDEN with reason = tenant-not-allowed.
Operationsruntime.operations on C lists and reads only the operations of these four methods: by handle (operationId) for the caller who started it, every caller's with allOwners: true for an administrator (AdminPanelManagement) of P. Other operations of C are not found.
Which tenantsC must be a registered direct child of P. The system tenant reaches every tenant. A grandchild, a sibling, the parent of the caller's tenant, and a deleted or not yet registered tenant are refused.
RolesThe role the methods declare, TenantManagement, in the parent tenant (the same role the REST jobs endpoint requires). Without it: METHOD_FORBIDDEN with reason = not-authorized, and no operations are listed.
TokensUser tokens only. A client-credentials token of P is refused on C as before; only clients on TenantAuthorization:CrossTenantServiceClientIds address other tenants.
Tenant generationThe token's tenant_gen is checked against the parent tenant P. A token of P issued before P was deleted and created again is refused like on P itself.
HTTP gateA request to /tenants/{C}/graphQl passes the route tenant check only when every field of the request is one of the method calls above or a runtime.operations read: one JSON POST, no batch, no fragments, no query string. A request that mixes an allowed and another field is refused as a whole. Send one plain operation per request.

Every allowed call is logged at Information level with the subject, the client, both tenants and the method. The file of a backup is downloaded from the downloadPath of the operation result ({C}/v1/jobs/download?id=… of the bot service), which accepts the parent's token for the same reason.

The Identity Service Checks the Route Tenant Too​

The Identity Service runs the same route tenant check on its own tenant API (users, roles, groups, clients, identity providers and the other /{tenantId}/... routes) for bearer tokens. Earlier versions stopped a token of another tenant there only by the role requirements of the individual endpoints.

What integrators have to change: a client that calls the Identity Service API of tenant B with a token issued for tenant A gets 403. Use a token of the tenant you address:

  • Users and interactive apps: sign in to the target tenant (the login with acr_values=tenant:<tenantId>), or exchange the current token for one of the target tenant with the token exchange.
  • Service clients: request one token per tenant, each with acr_values=tenant:<tenantId>; the client must be registered in that tenant (for example through client mirroring into child tenants).
  • Parent-tenant administrators who manage the users or clients of a child tenant do so with a token of the child tenant, not with the parent's token.

Operators can run this check in a log-only mode first, see Configuration.

Tenant Generations (tenant_gen)​

A tenant id can be used again: when tenant my-project is deleted and a new tenant my-project is created, a token of the deleted tenant would still match the tenant_id of the new one until it expires. To tell them apart, every access token carries the claim tenant_gen, the generation of its tenant: the time the tenant was created, as Unix milliseconds in a decimal string (for example "1794750381540"). ID tokens and the userinfo response do not carry it.

EventGenerationEffect on existing tokens
Create a tenantnew generation—
Restore a backup over the existing tenantkeptAccess tokens stay valid. Refresh tokens and codes are those of the backup: the ones issued after the backup was taken are gone, and their clients sign in again (invalid_grant).
Delete the tenantthe generation endsRefresh tokens, authorization codes and device codes of the tenant are gone at once (they are stored in the tenant's database). Access tokens are self-contained and stay valid until they expire, but with the check in Enforce the services refuse them as soon as the tenant is recreated.
Delete and create again with the same idnew generationTokens of the deleted tenant are never accepted by the new tenant (with the check in Enforce).
Delete, create again and restore a backup of the deleted tenantthe new generation of the createTokens, refresh tokens and codes of the deleted tenant are refused, also those that the restored data brings back (with the check in Enforce).

The system tenant is never refused by the generation check. Tenants without a recorded creation time (created by older platform versions) carry no generation and are not checked.

Restoring old backups into a recreated tenant

Run the generation check in Enforce before you restore a backup of a deleted tenant into a new tenant with the same id. In LogOnly, a refresh token from that backup that was issued before the Identity Service stored generations is accepted on its next refresh and from then on carries the generation of the new tenant.

What the Check Refuses​

With the check in Enforce:

CallerToken of an earlier tenant, or token without tenant_genGeneration cannot be read
Service API (/{tenantId}/...)403 Forbidden with a JSON message503 Service Unavailable
CK method callMETHOD_FORBIDDEN, reason tenant-not-allowedMETHOD_HANDLER_FAILED
Identity token endpoint: refresh token, authorization code, device code, token exchange, on-behalf-ofinvalid_grant (a refused refresh token is not rotated)server_error

In LogOnly (the default) none of these requests is refused; each one is counted in the tenant authorization metrics, and a token of an earlier tenant or without tenant_gen is logged as "would be denied" (at most once a minute per client; a failed lookup is only counted).

Tokens issued before the Identity Service stamped tenant_gen carry no generation. They are refused in Enforce as well, which is why an environment switches to Enforce only after such tokens have expired (see Recommended Rollout).

What Clients Have to Do​

SituationWhat to do
Interactive user (Refinery Studio, apps, octo-cli LogIn) gets 403 or invalid_grant after the tenant was recreatedSign in again. The new sign-in issues tokens of the new tenant.
A client's refresh fails with invalid_grantDiscard the stored refresh token and run the sign-in again.
Client-credentials client gets 403Request a new token (acr_values=tenant:<tenantId>). If the tenant was recreated, the client must exist in the new tenant first (registered again or mirrored again).
Long-running service that caches tokensDo not keep a token past a tenant delete. On a 403 for a tenant that was recreated, drop the cached token and request a new one.

Configuration​

Self-hosted operators configure the checks per service with the configuration section TenantAuthorization (environment variable prefix OCTO_, double underscore between the levels).

Configuration keyEnvironment variableValuesDefaultServices
TenantAuthorization:TenantGenerationEnforcementOCTO_TENANTAUTHORIZATION__TENANTGENERATIONENFORCEMENTLogOnly, Enforce, DisabledLogOnlyEvery service that runs the check: identity (its API and its grants), asset repository, bot, communication controller, report services, MCP service, AI services
TenantAuthorization:UserTokenEnforcementOCTO_TENANTAUTHORIZATION__USERTOKENENFORCEMENTLogOnly, EnforceEnforce in identity, report, MCP and AI services; LogOnly in asset repository and communication controller. The octo-mesh chart renders LogOnly for identity, asset repository and communication controller unless a value is setRoute tenant check of user tokens
TenantAuthorization:ServiceTokenEnforcementOCTO_TENANTAUTHORIZATION__SERVICETOKENENFORCEMENTLogOnly, Enforce, DisabledEnforce in the service; the octo-mesh chart renders LogOnly for identity unless a value is setRoute tenant check of client-credentials tokens
TenantAuthorization:CrossTenantServiceClientIdsOCTO_TENANTAUTHORIZATION__CROSSTENANTSERVICECLIENTIDS__0, __1, ...client ids; a trailing * matches a prefixemptyService clients that may address other tenants

The modes:

  • LogOnly changes no outcome. A request that Enforce would refuse is logged as a warning that names the client and the tenants (never the token); the generation check logs at most once a minute per client and outcome and counts every check in the tenant authorization metrics.
  • Enforce refuses those requests as described above.
  • Disabled (generation check and service tokens only) turns the check off without log or metric. Use it only as an emergency exit and return to LogOnly afterwards, because it hides which clients would be affected.

The user-token modes of the asset repository and the communication controller (LogOnly by default) are switched on separately, by the same steps as in Recommended Rollout.

The asset repository additionally guards its file routes (REST, and GraphQL operations on System.Files) with its own switch Files:TenantGuard (OCTO_Files__TenantGuard, Enforce or LogOnly, default Enforce; an unknown value counts as Enforce). It refuses a user token of another tenant on those routes even while UserTokenEnforcement is LogOnly, is independent of the switches on this page and is set and observed on its own.

Set the generation check to the same value on every service of an installation. A partial switch is not unsafe, but the protection is only complete when every service enforces.

Helm Values​

In the octo-mesh chart:

tenantAuthorization:
tenantGenerationEnforcement: LogOnly # LogOnly | Enforce | Disabled, all services of the chart

services:
identity:
userTokenEnforcement: LogOnly # LogOnly | Enforce
serviceTokenEnforcement: LogOnly # LogOnly | Enforce | Disabled
# tenantGenerationEnforcement: Enforce # optional override for this service only
ValueDefaultRenders
tenantAuthorization.tenantGenerationEnforcementLogOnlyOCTO_TENANTAUTHORIZATION__TENANTGENERATIONENFORCEMENT on identity, asset repository, bot, communication controller and AI services
services.<service>.tenantGenerationEnforcement with <service> = identity, assetRepository, bot, communication, aiServicesunset = the global valuethe same variable on that service only
services.identity.userTokenEnforcementLogOnlyOCTO_TENANTAUTHORIZATION__USERTOKENENFORCEMENT on identity
services.identity.serviceTokenEnforcementLogOnlyOCTO_TENANTAUTHORIZATION__SERVICETOKENENFORCEMENT on identity
services.assetRepository.userTokenEnforcement, services.communication.userTokenEnforcementLogOnlyOCTO_TENANTAUTHORIZATION__USERTOKENENFORCEMENT on that service

The octo-mesh-reporting, octo-mesh-mcp and octo-mesh-ai charts have the same value tenantAuthorization.tenantGenerationEnforcement (default LogOnly); set it to the same value as in octo-mesh. Any other value than the ones listed fails the chart render. Platform services and Refinery Studio do not run the checks and get no value.

Switch one environment at a time, test environments first:

  1. Run LogOnly. Deploy the Identity Service release that stamps tenant_gen and checks its grants, together with services that run the generation check, all in LogOnly.

  2. Wait for old tokens to expire. Tokens and refresh tokens issued before that Identity Service release carry no generation and would be refused in Enforce. Wait at least the longest refresh-token lifetime configured on any client (default 30 days), and at least the longest access-token lifetime if that is larger. Instead of waiting for the refresh tokens you can accept that users with older sessions have to sign in again; the longest access-token lifetime must pass in any case.

  3. Observe. Check the logs for "would be denied" lines for at least one day before the switch:

    • This would be denied with TenantGenerationEnforcement=Enforce (generation check, all services and the token endpoint),
    • This would be denied with UserTokenEnforcement=Enforce and This would be denied with ServiceTokenEnforcement=Enforce from the Identity Service (route tenant check). The asset repository and the communication controller log the same user-token lines while their own userTokenEnforcement is LogOnly; that is a separate switch with its own values.

    Each line names the client. A line for a token of a deleted and recreated tenant is exactly what Enforce must refuse. Every other client is fixed first: it requests its tokens with acr_values=tenant:<tenantId>, signs in again, or — only when it genuinely works across tenants — is added to CrossTenantServiceClientIds. The tenant authorization metrics show the same outcomes per client without tenant or user data.

  4. Switch to Enforce. Set services.identity.userTokenEnforcement and services.identity.serviceTokenEnforcement, and tenantAuthorization.tenantGenerationEnforcement in octo-mesh and in the reporting, MCP and AI charts, then roll out the deployments. Right after the rollout run a smoke test: a fresh sign-in, a GraphQL query, a CK method call, a token refresh and a client-credentials call. Watch the Denied: ... log lines for a day.

  5. Rollback. Set the values back to LogOnly and roll out again. Requests that were refused work again at once; no data changes.

Troubleshooting​

403 after a tenant was deleted and recreated, or recreated and restored​

Symptom: API calls to the tenant answer 403 with the message "Access is denied: the token was issued for an earlier tenant '...' that was deleted and created again, or before the tenant generation was issued. Sign in again.", although the user or client has the right roles.

Cause: the token was issued for the deleted tenant with the same id. With the generation check in Enforce the new tenant never accepts it. A restore of a backup into the recreated tenant does not change that: the restored tenant keeps the generation of the create.

Fix: sign in again (users), or request a new token (client-credentials clients). A client that must also exist in the new tenant is registered or mirrored there first.

invalid_grant on token refresh after a restore​

Symptom: a token refresh, the redemption of an authorization code or device code, or a token exchange fails with invalid_grant after the tenant was deleted, created again and restored from a backup.

Cause: the refresh token or code belongs to the earlier tenant. The restore brings the stored grants of the backup back, but the Identity Service refuses grants whose tenant generation does not match the current tenant.

Fix: discard the refresh token and sign in again. A restore over the existing tenant (without delete) keeps the generation, so the refresh tokens in the backup stay valid; a refresh token issued after the backup was taken is not in the restored database and fails with invalid_grant as well, for that reason.

403 on the Identity Service API with a token of another tenant​

Symptom: calls to /{tenantId}/... endpoints of the Identity Service answer 403, and the message names the token tenant and the route tenant.

Fix: use a token of the route tenant: sign in to that tenant, use the token exchange, or (service clients) request the token with acr_values=tenant:<tenantId>. See The Identity Service Checks the Route Tenant Too.

503 from a service with the generation check in Enforce​

Symptom: the message reads "The tenant of the token cannot be verified right now. Try again later."

Cause: the service could not read the tenant's generation from the system database. In Enforce it answers 503 instead of accepting the token unchecked. The request succeeds again once the database is reachable; check the connection between the service and MongoDB.

See Also​