Skip to main content

Tenant Database Ownership

Several OctoMesh instances can share one MongoDB server. Typical cases are the main instance and the dev instance of a shared test environment (system databases octosystem and OctoSystemDev), and the stack slots of local octo-devstack feature environments. Each instance is identified by its system database (SystemDatabaseName); RabbitMQ and CrateDB are separated per instance in the same way.

A tenant database on such a server must belong to exactly one instance. If two instances register the same database, both run their controllers, schedulers and pipelines on it (split brain), and neither notices. This page describes how OctoMesh keeps the ownership unambiguous and what an operator does when it is not.

The ownership marker​

Every tenant database carries an ownership marker: one document in the collection tenant_ownership of the tenant database itself, which names the owning instance by its system database (normalized to lower case), the tenant id and the time it was written. Because it lives in the tenant database, every instance on the server can read it.

WhenWhat happens to the marker
CreateWritten for the creating instance.
AttachWritten for the attaching instance, after the attach guard (below) let the attach through.
Tenant resolveWritten only when there is none yet (insert-if-absent). Databases from before the marker existed become owned this way the first time the tenant is used. An existing marker is never overwritten.
DetachRemoved, but only when it names this instance. Detach is the one way to hand a database over to another instance.

Attach guard​

An attach is refused when the database

  • carries the marker of another instance, or
  • has no marker but is registered as a tenant database in another instance's system database on the same server (a database from before the marker that its owner has not used since), or
  • is the system database of another instance.

The caller gets the same generic conflict as for every other unavailable name ("is not available"); which instance owns the database is written to the service log of the asset repository only. A database without a marker that no system database on the server registers can still be attached, and the attach writes the marker.

The check reads the tenant registries of all databases on the server with the admin user. A database counts as a system database when it carries the system-database marker, which an instance writes into its own system database when it creates its system tenant and on every start of its identity service. So the system database of an instance that has no tenant yet is recognised too. Until an instance runs a version that writes the marker, its system database counts as one when it holds a tenant registry, carries no ownership marker and is not registered as a tenant database anywhere else; this keeps tenants that have child tenants of their own out of the list.

Warning on resolve​

When an instance resolves a tenant whose marker names another instance, it

  • logs a warning once per process and tenant, naming the tenant, the database and the owning system database, and
  • reports its service health check (TenantContext, MongoDB tag) as Degraded with the same three values, for as long as the conflict exists.

The resolve itself is not refused and the marker is not changed, so existing tenants keep working. The warning means that two instances very likely work on the same database: resolve the double registration as described below.

Listing and stamping ownership​

ReviewTenantOwnership lists the tenant databases of the instance you are connected to whose ownership is not unambiguous. Run it in the system tenant context of each instance on the server.

octo-cli -c ReviewTenantOwnership
Listed asMeaning--stamp
UnmarkedNo marker, registered only in this instance.Writes the marker of this instance.
MarkedByOtherInstanceThe marker names another instance.Leaves it untouched.
RegisteredByOtherInstanceAnother system database on the server registers the database as well (double registration).Leaves it untouched.
octo-cli -c ReviewTenantOwnership --stamp

Without --stamp nothing is written. With --stamp only the Unmarked databases are stamped; every other database stays as it is and stays listed. A second run stamps nothing new and lists the same ambiguous databases. Registry rows whose database does not exist are counted and never stamped.

The command calls POST system/v1/tenants/review-ownership?stamp=false|true of the asset repository. It is answered only in the system tenant context (user tokens need the role AdminPanelManagement).

Resolving a double registration​

  1. Find out which instance really uses the tenant (its users, adapters, pipelines and data). That instance keeps it.

  2. In the other instance, detach the tenant:

    octo-cli -c Detach -tid "<tenant id in the other instance>"

    Detach keeps the database and changes no marker it does not own: it removes only a marker of its own instance, so the owner's marker stays in place, and a database without a marker stays without one. The same preconditions as for every detach apply (stream data, communication, reporting and AI services disabled), see Detaching a Tenant.

  3. Run ReviewTenantOwnership in both instances again; the database is no longer listed as registered by another instance, and the Degraded health state clears once the services have processed the detach. A database that had no marker is still listed as Unmarked in the instance that keeps it; run ReviewTenantOwnership --stamp there to stamp it.

If the database must move to the other instance, detach it in the owning instance first; the marker is removed and the other instance can attach it.

Do not delete the tenant to end a double registration: delete drops the MongoDB database. The delete therefore refuses a database that belongs to another instance (see below); detach is the way.

Deleting a tenant​

Deleting a tenant drops its MongoDB database. Before anything is deleted, the instance checks that the database is its own. The delete is refused when

  • the ownership marker of the database names another instance, or
  • the system database of another instance on the server registers the database (a double registration, also on the side of the instance whose marker the database carries).

The refusal is the same 409 as for every name that is not available (Database name '<name>' is not available.); nothing is deleted and both instances keep their tenant. The server log of the refusing instance names the owner (warning Rejected database name '<name>' ... because the delete would drop a database that ...). Clearing a tenant (drop and re-create empty) is refused the same way.

To delete such a tenant, first resolve the double registration as described above, then delete it in the instance that owns it.

If the database only carries the marker of another instance that no longer registers it (an orphaned marker, for example of a removed instance), an administrator can drop it deliberately:

octo-cli -c Delete -tid "<tenant id>" --override-foreign-ownership -y

The service logs the override as a warning that starts with OWNERSHIP OVERRIDE and names the owner. The override never drops a database that the system database of another instance still registers, and never the system database of another instance: that instance may still work on it, so detach the tenant there first. The REST call is DELETE <tenant>/v1/tenants?childTenantId=<id>&overrideForeignOwnership=true.

The override covers exactly the marker the delete saw. When the drop runs later (the delete settle sweep finishes an interrupted delete) and the database carries another marker by then, for example because another instance created a new database under the same name, the drop is refused and the database stays.

A delete whose records are already gone but whose drop is refused stays pending: the tenant id stays blocked until an administrator detaches or cleans the database manually. There is no automatic recovery for a pending delete without a recorded override.

Shared test environment and local octo-devstack slots​

  • Main and dev instance on one MongoDB server: both instances see each other's registrations. Run ReviewTenantOwnership in both after an upgrade to list databases from before the marker, then --stamp in the instance that owns them. A database listed as RegisteredByOtherInstance in both instances is a double registration; resolve it as above.
  • Local octo-devstack slots: every slot has its own system database on the shared local MongoDB. Tenants created in a slot belong to that slot. Attaching another slot's tenant database is refused; detach it there first.