Tenant Database Ownership
Several OctoMesh instances can share one MongoDB server. Typical cases are the main instance and the
dev instance of a shared test environment (system databases octosystem and OctoSystemDev), and
the stack slots of local octo-devstack feature environments. Each instance is identified by its
system database (SystemDatabaseName); RabbitMQ and CrateDB are separated per instance in the
same way.
A tenant database on such a server must belong to exactly one instance. If two instances register the same database, both run their controllers, schedulers and pipelines on it (split brain), and neither notices. This page describes how OctoMesh keeps the ownership unambiguous and what an operator does when it is not.
The ownership marker
Every tenant database carries an ownership marker: one document in the collection
tenant_ownership of the tenant database itself, which names the owning instance by its system
database (normalized to lower case), the tenant id and the time it was written. Because it lives in the
tenant database, every instance on the server can read it.
| When | What happens to the marker |
|---|---|
| Create | Written for the creating instance. |
| Attach | Written for the attaching instance, after the attach guard (below) let the attach through. |
| Tenant resolve | Written only when there is none yet (insert-if-absent). Databases from before the marker existed become owned this way the first time the tenant is used. An existing marker is never overwritten. |
| Detach | Removed, but only when it names this instance. Detach is the one way to hand a database over to another instance. |
Attach guard
An attach is refused when the database
- carries the marker of another instance, or
- has no marker but is registered as a tenant database in another instance's system database on the same server (a database from before the marker that its owner has not used since), or
- is the system database of another instance.
The caller gets the same generic conflict as for every other unavailable name ("is not available"); which instance owns the database is written to the service log of the asset repository only. A database without a marker that no system database on the server registers can still be attached, and the attach writes the marker.
The check reads the tenant registries of all databases on the server with the admin user. A database counts as a system database when it carries the system-database marker, which an instance writes into its own system database when it creates its system tenant and on every start of its identity service. So the system database of an instance that has no tenant yet is recognised too. Until an instance runs a version that writes the marker, its system database counts as one when it holds a tenant registry, carries no ownership marker and is not registered as a tenant database anywhere else; this keeps tenants that have child tenants of their own out of the list.
Warning on resolve
When an instance resolves a tenant whose marker names another instance, it
- logs a warning once per process and tenant, naming the tenant, the database and the owning system database, and
- reports its service health check (
TenantContext, MongoDB tag) as Degraded with the same three values, for as long as the conflict exists.
The resolve itself is not refused and the marker is not changed, so existing tenants keep working. The warning means that two instances very likely work on the same database: resolve the double registration as described below.
Listing and stamping ownership
ReviewTenantOwnership lists the tenant databases of the instance you are connected to whose ownership
is not unambiguous. Run it in the system tenant context of each instance on the server.
octo-cli -c ReviewTenantOwnership
| Listed as | Meaning | --stamp |
|---|---|---|
Unmarked | No marker, registered only in this instance. | Writes the marker of this instance. |
MarkedByOtherInstance | The marker names another instance. | Leaves it untouched. |
RegisteredByOtherInstance | Another system database on the server registers the database as well (double registration). | Leaves it untouched. |
octo-cli -c ReviewTenantOwnership --stamp
Without --stamp nothing is written. With --stamp only the Unmarked databases are stamped; every
other database stays as it is and stays listed. A second run stamps nothing new and lists the same
ambiguous databases. Registry rows whose database does not exist are counted and never stamped.
The command calls POST system/v1/tenants/review-ownership?stamp=false|true of the asset repository.
It is answered only in the system tenant context (user tokens need the role AdminPanelManagement).
Resolving a double registration
-
Find out which instance really uses the tenant (its users, adapters, pipelines and data). That instance keeps it.
-
In the other instance, detach the tenant:
octo-cli -c Detach -tid "<tenant id in the other instance>"Detach keeps the database and changes no marker it does not own: it removes only a marker of its own instance, so the owner's marker stays in place, and a database without a marker stays without one. The same preconditions as for every detach apply (stream data, communication, reporting and AI services disabled), see Detaching a Tenant.
-
Run
ReviewTenantOwnershipin both instances again; the database is no longer listed as registered by another instance, and the Degraded health state clears once the services have processed the detach. A database that had no marker is still listed asUnmarkedin the instance that keeps it; runReviewTenantOwnership --stampthere to stamp it.
If the database must move to the other instance, detach it in the owning instance first; the marker is removed and the other instance can attach it.
Do not delete the tenant to end a double registration: delete drops the MongoDB database. The delete therefore refuses a database that belongs to another instance (see below); detach is the way.
Deleting a tenant
Deleting a tenant drops its MongoDB database. Before anything is deleted, the instance checks that the database is its own. The delete is refused when
- the ownership marker of the database names another instance, or
- the system database of another instance on the server registers the database (a double registration, also on the side of the instance whose marker the database carries).
The refusal is the same 409 as for every name that is not available (Database name '<name>' is not available.); nothing is deleted and both instances keep their tenant. The server log of the refusing
instance names the owner (warning Rejected database name '<name>' ... because the delete would drop a database that ...). Clearing a tenant (drop and re-create empty) is refused the same way.
To delete such a tenant, first resolve the double registration as described above, then delete it in the instance that owns it.
If the database only carries the marker of another instance that no longer registers it (an orphaned marker, for example of a removed instance), an administrator can drop it deliberately:
octo-cli -c Delete -tid "<tenant id>" --override-foreign-ownership -y
The service logs the override as a warning that starts with OWNERSHIP OVERRIDE and names the owner.
The override never drops a database that the system database of another instance still registers, and
never the system database of another instance: that instance may still work on it, so detach the
tenant there first. The REST call is
DELETE <tenant>/v1/tenants?childTenantId=<id>&overrideForeignOwnership=true.
The override covers exactly the marker the delete saw. When the drop runs later (the delete settle sweep finishes an interrupted delete) and the database carries another marker by then, for example because another instance created a new database under the same name, the drop is refused and the database stays.
A delete whose records are already gone but whose drop is refused stays pending: the tenant id stays blocked until an administrator detaches or cleans the database manually. There is no automatic recovery for a pending delete without a recorded override.
Shared test environment and local octo-devstack slots
- Main and dev instance on one MongoDB server: both instances see each other's registrations. Run
ReviewTenantOwnershipin both after an upgrade to list databases from before the marker, then--stampin the instance that owns them. A database listed asRegisteredByOtherInstancein both instances is a double registration; resolve it as above. - Local octo-devstack slots: every slot has its own system database on the shared local MongoDB. Tenants created in a slot belong to that slot. Attaching another slot's tenant database is refused; detach it there first.