Skip to main content

Implementing Method Handlers

A method is declared in a Construction Kit model. A handler is the code that runs when the method is called. This guide shows how to write one for each of the three ways a method can be handled, and which rules every handler follows. The consistency rules (transactions, failures, retries) are stated in the Method Consistency Model; read it first.

Status

The wire contract (request, response envelope, error codes) is fixed. The handler SDK that this guide describes (Meshmakers.Octo.Services.Methods, typed handlers, AddCkMethodHandlers), the dispatcher, the in-process transport and the pipeline handler are released, and callers reach them through the typed GraphQL method mutations. Asynchronous methods with operation handles are released as well: a service handler can accept a long-running call and report its progress (see Asynchronous handlers and OD-5).

Three ways to handle a method​

Remote handler (bus)In-process handlerPipeline handler
Declared byexecution.handler: service (default)execution.handler: serviceexecution.handler: pipeline
Runs inthe service that owns the model (for example the identity service for System.Identity)the gateway (asset repository) itself, for methods the gateway owns (planned: Tenant.*, Archive.*; none in production yet)a pipeline on a mesh adapter
How it is reachedevent hub, one queue per method, shared by all instances of the servicedirect call in the same process; nothing is published to the busthe communication controller starts the deployed pipeline whose well-known name equals the qualified method id
You writea handler classthe same handler class, registered as in-processa pipeline that ends with SetPipelineExecutionResult

Whatever the transport, the request and response envelope, the language-neutral error codes, the deadline, the timeout, the consistency rules, authorization and audit are the same. The caller cannot tell the transports apart: it always receives a CkMethodResponse.

The dispatcher​

Callers do not talk to a queue themselves. The gateway, a job or a pipeline calls the dispatcher (ICkMethodDispatcher), which resolves the handler, runs the call through the invocation pipeline (audit, authorization, idempotency) and returns a CkMethodResponse. It never throws for a failing call; only a cancellation by the caller surfaces.

  • The queue of the declaring element. A method cannot be overridden in a derived type, so there is exactly one handler for a method: the one of the element that declares it. A call that targets a derived type reaches the queue of the type that declares the inherited method, and a call on a type that implements an interface reaches the queue of the interface method's declaring element. You register a handler once, for the declaring element, and it serves every type that inherits or implements the method.
  • In-process first. If the service that runs the dispatcher has an in-process handler for the method (see below), the call is made directly in the same process and nothing is published to the bus. The same holds for a method the service itself hosts for the bus, when it calls that method through its own REST facade. Otherwise the dispatcher sends the call to the method's queue on the event hub. The same invocation pipeline applies to both transports, so one filter runs for both.
  • The send is mandatory and never retried. If no queue is bound or nobody consumes it (service down, not yet announced, last replica stopped), the broker returns the message and the caller receives METHOD_HANDLER_NOT_AVAILABLE at once (within one second locally), naming the method and carrying no stack trace. The dispatcher does not resend and has no publisher-side retry: a transport fault ends the call as METHOD_TIMEOUT or METHOD_HANDLER_NOT_AVAILABLE, never as a second delivery. Whether to retry is the caller's decision (retries); after METHOD_HANDLER_NOT_AVAILABLE it is always safe, because nothing ran.
  • One envelope. Success, a declared error and METHOD_HANDLER_FAILED arrive unchanged from the handler, whichever transport served the call.

Common rules​

These apply to every handler, whatever the transport.

  1. Re-validate the caller. The bus is internal and not authenticated per message. The caller in the request is informational. Authorize from the forwarded access token (signature, issuer, expiry, tenant, scope, and the audience if configured), never from the caller data. A forged caller must have no effect. The host does the validation for you when the service opts in: see Caller token validation.
  2. Check in this order: authenticate → authorize → look anything up → validate parameters → write. Answer METHOD_FORBIDDEN with the same text whether the target exists or not, so that a caller without the right cannot probe for entities.
  3. Own your data access. There is no session in the method context and you must not look for one. Do not use services that are bound to the calling request: the HTTP context, the request's tenant repository, a UserManager or store that finds its tenant from the HTTP request. Outside a request they fall back to the system tenant and would write into the wrong database. Resolve the tenant explicitly from tenantId of the request, create what you need for exactly that tenant and dispose it.
  4. Make your writes atomic and commit before you answer. One document: one atomic update. Several: your own session and transaction. See R3.
  5. Honor the deadline. The cancellation token is cancelled at the handler deadline. Check it before you start a write; do not start one afterwards.
  6. Never throw to the bus, never retry. The platform does not retry a call either: a handler that throws after a side effect has run exactly once, and the caller receives METHOD_HANDLER_FAILED. A delivery the broker flags as a possible duplicate, and an invocation id the instance has already seen, never reach your handler; the caller receives METHOD_OUTCOME_UNKNOWN. What is not detected is listed in limits, so keep a non-idempotent handler safe against the same logical operation arriving twice. Return an envelope for every outcome. An unexpected exception becomes METHOD_HANDLER_FAILED with a correlation id; the stack trace goes to the log under that id, never on the wire. The host does this for you; do not catch and hide exceptions that would hide the real cause.
  7. Declared errors are data. A business refusal is a code from the method's errors list with parameters (strings), never free text for the user. Parameter values of the method must not appear in error parameters.
  8. No secrets in logs. Parameters marked sensitive: true and the access token are not logged, not audited, not put in traces, outbox payloads or exception messages. Log the invocation id, the method id, the tenant and the outcome. What the platform masks for you is described in Sensitive parameters and filters.
  9. No state in instance fields. The next call can be served by another instance.
  10. Side effects after the commit. Publish or send only after the write committed; use an outbox if a lost message is not acceptable (side effects).
  11. Tolerate concurrency. Two calls on the same target can run at once. Use an atomic conditional update or a version check.

Caller token validation (remote handlers)​

A request that arrives over the bus carries the access token of the caller (CallerAccessToken) next to the informational caller data. The bus is not authenticated per message, so only the token proves who is calling. A remote handler host validates that token before the parameters are read and before the handler runs, with the same trust as the HTTP API of its service. The decision and its alternatives are recorded as OD-10.

The service opts in once:

builder.Services.AddCkMethodCallerTokenValidation(o => o.Authority = "<authority of the HTTP API of this service>");

The settings live in the configuration section Methods:Token. A valid token reaches the handler as context.CallerPrincipal (subject, client id, tenant, IsServiceToken). A service that has not opted in gets CallerPrincipal == null and nothing is validated: such a handler has no verified caller and must refuse methods that need one. The in-process transport validates no token; it runs in the process that already authenticated the caller and passes that principal on. It does apply the tenant rule (row 6 below) to every in-process call made with an access token, centrally and before the handler, whatever mode the HTTP tenant gate runs in: a token of another tenant is refused with tenant-not-allowed; only a service token whose client is on TenantAuthorization:CrossTenantServiceClientIds passes, and a user token of another tenant only for the backup, restore and archive data methods of a direct child tenant (or any tenant, for the system tenant), see Parent-tenant administration through CK methods. A principal that a service builds itself for its own job carries no access token and is not checked; that service vouches for the tenant of its calls. An in-process handler called through the gateway needs no tenant check of its own.

The steps​

#CheckOn failure (METHOD_FORBIDDEN, parameter reason)
1A token is present.token-missing
2Signature and issuer: the issuer is the authority of the service (with a trailing slash), the signing keys come from the identity service's key set. Only asymmetric algorithms are accepted; none and HS* are never accepted.token-invalid
3Expiry, with a clock skew of 30 seconds.token-expired (only for a token that is otherwise genuine; an expired and forged token is token-invalid, an expired token of another tenant is tenant-not-allowed)
4Audience, only if ValidAudiences is configured: the token carries one of the listed audiences.audience-mismatch
5Scope octo_api.token-invalid
6Tenant: the tenant_id claim equals the tenant of the request (case-insensitive). The allowed_tenants claim is only a hint for choosing a tenant and is never consulted. The only parent-tenant exception covers the backup, restore and archive data methods, see Parent-tenant administration through CK methods.tenant-not-allowed
7Subject: a user token carries sub. A token without sub is a service token (client credentials) and must carry client_id.token-invalid

The answer never names the claim that failed, so it gives a forger nothing to probe. The handler is not called and nothing is written. The log line of a refusal carries the tenant, the reason and an internal category, never the token, a claim value or a key id. If the host cannot load any signing key (identity unreachable and nothing cached), the call is answered METHOD_HANDLER_FAILED: the caller did nothing wrong, and a token is never accepted without a verified signature.

Service tokens​

Jobs, pipelines and other services call methods with a client-credentials token, which has no sub. Such a token is accepted when it carries client_id and its tenant_id equals the request tenant, or when its client is on the allow-list TenantAuthorization:CrossTenantServiceClientIds (the same list as for the HTTP API). The validation is always fail-closed: the LogOnly and Disabled modes of the HTTP side are never applied. The subject of the principal is the client id.

Audience​

The platform does not validate the audience on its HTTP APIs, so Methods:Token:ValidAudiences is empty by default and the audience is not checked. With it empty, every octo_api token of the tenant is accepted for a method call. Set it as soon as the identity service issues audience-scoped tokens.

Expired tokens​

token-expired is distinct from the other reasons so that a client can tell "get a fresh token and call again" from "not allowed". The code stays METHOD_FORBIDDEN. A call waits in the queue at most as long as its timeout (1 to 300 seconds, default 15), so a token that was valid when the call was made rarely expires while the call waits.

Pipeline handlers​

The pipeline handler of the communication controller validates the forwarded token with the same validator and builds the caller of the pipeline from the validated principal only, with trust level strong; the caller data of the request is never used. A call without a valid token is answered METHOD_FORBIDDEN (same reasons as above) and no pipeline is started. It needs no opt-in: the controller takes its authority from the settings of its own HTTP API (details). A remote handler that is written as a service handler gets the validated principal from the host as described above.

What the token does not protect​

The forwarded token is valid for the APIs of the tenant until it expires, and it is delivered to whoever consumes the method queue. Validation protects a handler against a forged caller; it does not protect the caller against a receiver on the broker. Who may consume the queue is decided and restricted at the broker, see OD-11, including the rule that no method that carries a password may run on a shared staging or production environment before the broker is restricted.

Worked example: User.ChangePassword​

The method changes the password of a user. A user changes their own password (and must give the current one); a user with the UserManagement role resets the password of someone else (no current password needed). The example is taken from the identity service; the model is shown as it would be declared and contains no real values.

1. The method in the model​

types:
- typeId: User
# ...
methods:
- methodId: ChangePassword-1
description: Changes (self service) or resets (administrator) the password of a user.
parameters:
- name: currentPassword
valueType: String
isOptional: true # required for self service, ignored for a reset
sensitive: true
- name: newPassword
valueType: String
sensitive: true
errors:
- code: PASSWORD_POLICY_VIOLATION
- code: CURRENT_PASSWORD_INVALID
- code: USER_LOCKED_OUT
authorization:
roles: [ UserManagement ]
allowSelf: true
execution:
timeoutSeconds: 15
idempotent: false # a second call with a new password is not the same as the first

The source generator emits, next to the model, the id constant SystemIdentityCkIds.UserChangePasswordMethodId (System.Identity/User.ChangePassword-1) and the record UserChangePasswordParameters with CurrentPassword and NewPassword. Both parameters are sensitive: the generated ToString() prints ***.

authorization is checked at the gateway and by the handler host: the declaration says who may call; before the handler runs, the host evaluates it again for the validated caller, with the same evaluator as the gateway. The handler does not need to repeat the declared check.

Where the declaration is enforced. The platform enforces authorization without any handler code, in four places that use the same rules: the gateway checks before it loads the target (GraphQL callers); the authorization filter of the invocation pipeline (band 200, AddCkMethodAuthorization() in Meshmakers.Octo.Services.Methods) checks every call that reaches the dispatcher, over the bus and in-process alike, and is the only check for callers that do not use the gateway (REST, jobs, pipelines); the handler host evaluates the declaration for the validated caller after the token validation and before the parameters are read, in the remote host and in the in-process host alike, so a valid token of the tenant that is published straight on the bus cannot reach a handler whose roles, scopes or allowSelf it does not meet; a method without a definition is refused, and a remote host that has not enabled the caller token validation has no validated caller and refuses every call; the handler re-validates what only it can know (rule 1). The filter takes the caller from the process that starts the call (a validated token, or the identity of the job), never from the caller data of a request that arrives over the bus. It runs inside the audit filter (band 100), so a refusal is audited, and outside the idempotency filter (band 300) and the transport, which a refused call never reaches. A refusal is the language-neutral error METHOD_FORBIDDEN with the parameters methodId and reason: not-authorized; it never says which rule failed and is identical whether the target exists or not.

2. The handler​

A handler is a class with one method. The host deserializes the parameters into the generated record, serializes the result and turns exceptions into envelopes.

internal sealed class ChangePasswordHandler(ChangePasswordService service)
: ICkMethodHandler<UserChangePasswordParameters> // no result
{
public async Task<CkMethodResult> InvokeAsync(
CkMethodContext context, // tenant, caller token, target id, deadline
UserChangePasswordParameters parameters,
CancellationToken cancellationToken) // cancelled at the handler deadline
{
// Thin adapter: everything that matters lives in a transport-independent service,
// which is unit-testable without the messaging package.
var outcome = await service.ChangeAsync(
context.TenantId, context.TargetRtId, context.CallerAccessToken, parameters, cancellationToken);

return outcome.Success
? CkMethodResult.Ok()
: CkMethodResult.Fail(outcome.ErrorCode, outcome.ErrorParameters);
}
}

The service performs the algorithm in the order of rule 2:

StepWhatOn failure
1Take the caller from context.CallerPrincipal, which the host filled after it validated the token (steps: signature, issuer, expiry, tenant_id equals the request tenant, scope octo_api, optional audience). A service that has not opted in has no validated caller; its host refuses the call before the handler runs.METHOD_FORBIDDEN
2Authorize on parsed ids, not strings: self when the principal's subject (sub) equals the target id, administrator when the token carries UserManagement. Compare OctoObjectIds (the parser accepts upper-case hex, a string comparison would not).METHOD_FORBIDDEN — the same answer whether the target exists or not
3Resolve the tenant context from the request's tenant id; check that the tenant has the identity model.METHOD_HANDLER_FAILED
4Load the target user with a user manager created for that tenant (see rule 3), disposed after the call.METHOD_TARGET_NOT_FOUND
5Validate parameters: newPassword present; currentPassword present for self service.METHOD_VALIDATION_FAILED with parameter field
6Refuse a locked-out user.USER_LOCKED_OUT
7Check the cancellation token, then write. Self: change with the current password; a wrong current password counts toward lockout like a failed sign-in. Administrator: reset. The write updates the password hash and the security stamp in one operation.CURRENT_PASSWORD_INVALID, PASSWORD_POLICY_VIOLATION
8Map the result to the envelope; log one line with the invocation id, tenant, target id, caller id and outcome — no parameters, no token.

Consistency decisions in this example:

  • The password change is a single logical write done by the identity framework in one operation, so it is atomic (R3). The handler does not need its own multi-document transaction.
  • It is not idempotent: repeating it with a different password is a different request. A repeated call with the same parameters from the same user usually fails the second time with CURRENT_PASSWORD_INVALID (a natural guard, not a guarantee).
  • If the handler crashes after the write and before the answer, the caller receives METHOD_TIMEOUT although the password changed (failure example 1). The client reads the state by signing in with the new password.
  • A notification mail after an administrator reset is a side effect: send it after the commit, through a mail path that is bound to that tenant (not through a service that finds its tenant from the HTTP request), and treat a mail failure as a warning, not as a failure of the method.
  • Session revocation after a password change (refresh tokens) is a separate write in another store. It belongs after the commit, with an outbox or a compensation if it must not be lost.

3. Registration​

// Program.cs of the service that owns the model
builder.Services.AddCkMethodHandlers("IdentityService", methods =>
methods.Handle<ChangePasswordHandler>(SystemIdentityCkIds.UserChangePasswordMethodId));

At startup the host checks the registration against the CK cache: an unknown method id or a parameter type that does not match the model stops the service with a message that names the method. The host then announces the handler on the event hub after its endpoint is ready, repeats the announcement as a heartbeat and withdraws before shutdown. You never touch queues, JSON or announcements.

With two replicas of the identity service both announce; the queue of the method is shared, so each call goes to one of them.

The gateway remembers its handlers across a restart. The gateway (and every other host that dispatches methods) keeps its list of announced handlers as a snapshot in the system database, so a restarted gateway answers a call without waiting for the next heartbeat round. The snapshot is a last seen record, not a promise that the handler is alive: a row counts as live only for as long as the heartbeat window it was announced with (45 seconds by default) has not passed. A handler that has stopped, whose window has expired or that withdrew is never used; the call ends as METHOD_HANDLER_NOT_AVAILABLE. The snapshot holds service data only (method id, service name, instance id, versions, times, served tenants), never a parameter value, token or user data. Without the snapshot store the list lives in memory only, exactly as before.

Heartbeat and TTL. A handler host repeats its announcement every 15 seconds and each announcement is valid for 45 seconds. These two values are final defaults; you normally do not change them. They are set in the Methods section of the host's configuration:

KeyEnvironment variableDefault
Methods:HeartbeatIntervalOCTO_Methods__HeartbeatInterval00:00:15
Methods:TtlOCTO_Methods__Ttl00:00:45
  • Rules. HeartbeatInterval must be at least 1 second, Ttl at least twice the HeartbeatInterval (so a live handler survives one lost heartbeat) and at most 24 hours. A host that breaks a rule does not start; the error names the key and the rule, for example Methods:Ttl is 00:00:20 but must be at least 2 x Methods:HeartbeatInterval (00:00:30).
  • Different values per host. Every announcement carries its own TTL, and a dispatcher counts it from the moment it receives the announcement. Hosts of different services may therefore use different values without any setting on the gateway.
  • Why these values. The list of handlers is advisory. Whether a call fails fast with METHOD_HANDLER_NOT_AVAILABLE is decided by the broker, which refuses a message nobody can receive, not by the TTL. The TTL only decides what diagnostics and the snapshot show as live. 45 seconds tolerates two lost heartbeats; shorter values only add message traffic.

4. Tests​

TestProves
Unit tests of the service with fake token validationevery step and error code of the table above, including that a forged caller is rejected and the same METHOD_FORBIDDEN text for existing and missing targets
Log capture testno password and no token in any log line, also on failure paths
Integration test against a real databasethe password is changed, the security stamp changes, lockout counts; a cancelled token before step 7 writes nothing
Handler registration testa wrong method id fails at startup
Failure testhandler throws → METHOD_HANDLER_FAILED with correlation id, no stack trace in the envelope

The same handler as an in-process handler​

The asset repository will own some methods itself (planned: Tenant.*, Archive.*; no production method is registered yet, the transport is released). Their handlers are the same classes; they are registered as in-process, so the dispatcher finds them before it looks for a remote handler. Nothing is published to the bus and the handler is not announced as a remote handler. The envelope, the deadline and the exception mapping are the same, and so are the pipeline steps for authorization and audit that run for every transport.

What changes for you:

  • There is no network between the gateway and the handler, so it is tempting to use what is at hand: the request's database session, the HTTP context, the request-scoped repository. Do not (rule 3). The method is a separate unit of work. An in-process handler is resolved in its own scope; a handler that reaches for the request's session would silently join the request transaction and break R1. The platform makes this a guarantee (OD-8), not only a rule: the in-process transport runs every handler detached from the ambient execution context of the caller, in a fresh dependency-injection scope per call. A value that the caller keeps in an AsyncLocal (the request session of the gateway, a principal scope, a logging scope) is empty inside the handler, also after its first await, and nothing the handler sets reaches the caller. A new scope alone would not be enough, because ambient values travel with the asynchronous flow, not with the scope.

Carried across on purpose: the trace (the span of the handler is a child of the caller's activity, so the trace of a request continues through the call), the culture of the caller (CurrentCulture and CurrentUICulture), the logging scope that identifies the call (MethodId, InvocationId, CorrelationId, TenantId), and the cancellation token with the deadline. Nothing else is carried; a library that reads another ambient value inside a handler sees its default. Pass what a handler needs in the request.

As a tripwire, a service registers a probe for each kind of request-bound state it owns (AddCkInProcessAmbientStateProbe<TProbe>(); the gateway registers one for the request session, HasSession). The probe is asked when the handler starts, in the handler's own flow. If it reports state, the call fails with METHOD_HANDLER_FAILED and the handler does not run; the log names the probe, never the data. With the detach in place this never happens; it catches a regression.

  • Deadlines and timeouts apply as for a remote call.
  • A handler that is registered for both transports (for example in a test) behaves the same way; that is the test of a transport-independent handler.

A REST facade over a method the service hosts​

A service that hosts a method handler can serve the same operation through its own REST endpoint, a facade (for example the identity service's user endpoints over the User.* methods). The facade calls the method through the dispatcher like any other caller, and the handler runs in the same process through the same invocation pipeline as a call from another service: declared authorization, parameter validation, masking of sensitive parameters, audit and the ck.method.* metrics (transport in-process). There is no bus hop, so no parameter (a password, for example) reaches the broker.

AddCkMethodHandlers registers this local transport by itself; AddCkMethodDispatcher gives the facade its ICkMethodDispatcher.

Two registrations, one rule.

RegistrationAnnounced, bus queueAnother process calls itThis process calls it
m.Handle<THandler>(methodId) in AddCkMethodHandlersyesover the busin-process (nothing is sent)
m.HandleLocalOnly<THandler>(methodId) in AddCkMethodHandlersnoMETHOD_HANDLER_NOT_AVAILABLE (no queue)in-process
m.Handle<THandler>(methodId) in AddCkInProcessMethodHandlersnoMETHOD_HANDLER_NOT_AVAILABLEin-process
  • HandleLocalOnly<THandler>(methodId) is for a method whose bus exposure is switched off while its facade must keep working, for example the identity password methods while the broker is not protected: EnablePasswordMethods ? m.Handle<H>(id) : m.HandleLocalOnly<H>(id).
  • One place per method. A method is registered either for the bus or local only; registering both for the same id stops the service at startup.
  • Same envelope. A bus-registered method that its own host calls answers with the announced handler instance id, so the envelope is byte-identical to the one a remote caller gets from that host.
  • Methods the service does not host still go over the bus. If a facade must never leave the process, check ICkInProcessMethodTransport.CanHandle(methodId) at startup.

The caller. The local path validates no token. The principal is the one the facade sets with CkMethodPrincipalScope, taken from the authenticated HTTP user. Without a scope there is no principal, and the declared authorization refuses the call with METHOD_FORBIDDEN; nothing falls back to a system identity. Like every method caller, the principal needs the scope octo_api.

The result. CkMethodFacadeResults.ToActionResult(response, options) (namespace Meshmakers.Octo.Services.Methods.Facades) turns the envelope into an MVC IActionResult:

OutcomeDefault status
success with a result / without a result200 with the result JSON / 204
accepted (asynchronous call, operation handle set)202 with the operation handle
METHOD_VALIDATION_FAILED400
METHOD_FORBIDDEN403
METHOD_TARGET_NOT_FOUND404
METHOD_REQUEST_IN_PROGRESS409
METHOD_PARAMETERS_TOO_LARGE413
METHOD_HANDLER_FAILED, METHOD_OUTCOME_UNKNOWN, an unknown METHOD_* code500
METHOD_HANDLER_NOT_AVAILABLE, METHOD_CONTRACT_VERSION_UNSUPPORTED, METHOD_NOT_SUPPORTED_BY_HANDLER_VERSION503
METHOD_TIMEOUT504
a code the method declares400

The default error body is problem details (application/problem+json) with the code as title and the extensions code, parameters and details. Error parameters never carry parameter values, so the body is safe to return. An endpoint that must keep its existing status codes and bodies configures CkMethodFacadeOptions: MapError(code, status), MapError(code, error => result) (also for platform codes) and MapSuccess(response => result). Build the options once per endpoint.

// Program.cs of the identity service
services.AddCkMethodHandlers("IdentityService", m =>
{
m.Handle<UserSetRolesHandler>(SystemIdentityCkIds.UserSetRolesMethodId);
if (options.EnablePasswordMethods) m.Handle<UserResetPasswordHandler>(SystemIdentityCkIds.UserResetPasswordMethodId);
else m.HandleLocalOnly<UserResetPasswordHandler>(SystemIdentityCkIds.UserResetPasswordMethodId);
});
services.AddCkMethodDispatcher(); // the facade's entry point; local methods never touch the bus

// UsersController
private static readonly CkMethodFacadeOptions ResetPasswordFacade = new CkMethodFacadeOptions()
.MapError("PASSWORD_POLICY_VIOLATED", error => new BadRequestObjectResult(error.Details))
.MapSuccess(_ => new NoContentResult());

[HttpPost("{id}/reset-password")]
public async Task<IActionResult> ResetPassword(string tenantId, string id, ResetPasswordDto dto)
{
var request = new CkMethodRequest(tenantId)
{
InvocationId = Guid.NewGuid(),
MethodId = SystemIdentityCkIds.UserResetPasswordMethodId,
CkTypeId = "System.Identity/User",
TargetRtId = id,
ParametersJson = JsonSerializer.Serialize(new UserResetPasswordParameters { NewPassword = dto.Password }, CkMethodJson.Options),
DeadlineUtc = timeouts.GetDeadlineUtc(tenantId, "System.Identity/User", SystemIdentityCkIds.UserResetPasswordMethodId)
};
using (CkMethodPrincipalScope.Begin(CkMethodPrincipal.FromClaimsPrincipal(User)))
{
var response = await dispatcher.InvokeAsync(request, HttpContext.RequestAborted);
return response.ToActionResult(ResetPasswordFacade);
}
}

A pipeline as handler​

A method with execution.handler: pipeline is implemented by a pipeline instead of C# code. This suits orchestration of existing nodes (read, transform, call, write) and models that do not own a service.

methods:
- methodId: UpdateFirmware-1
parameters:
- name: version
valueType: String
execution:
timeoutSeconds: 120 # mandatory for a pipeline method
handler: pipeline

How the call is handled:

  1. The communication controller announces itself as handler of every method whose compiled definition says handler: pipeline.
  2. For a call it looks up, in the call's tenant, the deployed pipeline whose well-known name equals the qualified method id (for example Acme.Assets/Device.UpdateFirmware-1; no further binding exists). No such pipeline: METHOD_HANDLER_NOT_AVAILABLE with reason pipeline-not-deployed. A changed pipeline takes effect after it is deployed again.
  3. It starts the pipeline with the method parameters as input and the caller's identity, exactly as for a manual execution, so the pipeline runs with the caller's rights.
  4. The pipeline starts with the execute-command trigger and ends with SetPipelineExecutionResult, whose output must match the method's result record.
  5. The outcome maps to the envelope: success with matching result → success; pipeline failure → METHOD_HANDLER_FAILED; an error code set by the pipeline → that code; a result that does not match the result record → METHOD_HANDLER_FAILED with a validation parameter; the deadline passes → METHOD_TIMEOUT.

Consistency of a pipeline handler:

  • A pipeline run is its own unit of work like any handler; the pipeline's nodes commit as they run. There is no rollback of earlier nodes when a later node fails. Design the pipeline so that the write nodes come last, or model compensation as nodes.
  • On METHOD_TIMEOUT the pipeline may still finish. This is documented, not prevented. A non-idempotent pipeline method is the author's responsibility.
  • A method with a sensitive parameter cannot be a pipeline method: a pipeline execution stores its input and output, so the compiler refuses handler: pipeline for such a method with message 132 (decision OD-9). Implement it as a service handler. The pipeline handler refuses such a method at runtime as well (reason sensitive-parameters-not-supported). The same holds for a result marked sensitive: true, which the compiler refuses for a pipeline method with message 134 (Sensitive results); the pipeline handler refuses it at runtime too, with reason sensitive-result-not-supported.

The complete, tested sample pipeline for a pipeline-handled method is in Pipeline method handlers.

Asynchronous handlers​

A method declared execution.mode: async (Execution mode) does its work after the call has returned. The handler accepts the call, the platform writes a durable operation record and answers the caller with an operation handle; the caller reads state, progress and result later through the handle (GraphQL runtime.operations, see Asynchronous methods and operations). The guarantees are decided in OD-5: at most once after accept, no cancellation, a lost handler ends as METHOD_OUTCOME_UNKNOWN.

Only a service handler can be asynchronous, and by decision OD-10 only one of a bot job type (the bot services run the work; the repository dump is the first). A pipeline handler answers synchronously, and the compiler refuses mode: async together with handler: pipeline (message 142). For an asynchronous method, timeoutSeconds bounds the accept step, not the operation.

The handler​

Validate the parameters synchronously and answer a declared error at once when they are wrong. Return Accepted only when the work can start:

public sealed class BackupDumpHandler(IDumpService dumps)
: ICkMethodHandler<BackupDumpParameters, BackupDumpResult>
{
public Task<CkMethodOutcome<BackupDumpResult>> HandleAsync(
CkMethodContext<BackupDumpParameters> context, CancellationToken cancellationToken)
{
if (string.IsNullOrEmpty(context.Parameters.Target))
{
return Task.FromResult(CkMethodOutcome<BackupDumpResult>.ValidationFailed("target", "required"));
}

var tenantId = context.TenantId; // copy what the work needs; never keep the caller's token
return Task.FromResult(CkMethodOutcome<BackupDumpResult>.Accepted(async (progress, ct) =>
{
await progress.ReportAsync(CkMethodOperationProgress.Create(10, "COPYING_COLLECTIONS"), ct);
var file = await dumps.DumpAsync(tenantId, ct);
return new BackupDumpResult { Value = file }; // or a declared error
}));
}
}

CkMethodOutcome.Accepted(work) is the same for a method without a result. The work delegate receives a progress reporter (ICkMethodOperationProgress, which also carries the OperationId) and a cancellation token. The repository dump of the bot services (System.Bot/TenantRepository.Dump) is the first method in production built this way.

Rules for the work:

  • It runs once, after the accepted answer has left, detached from the caller (no ambient state of the request, as for an in-process handler), in the handler's dependency-injection scope, which stays open until the work ends. Copy what the work needs from the context before you return; the caller's token is not kept (OD-10).
  • Its token is cancelled when the host stops. The operation then ends Failed with METHOD_OUTCOME_UNKNOWN (reason handler-lost). There is no resume and no retry.
  • It ends with a result or a declared error. An undeclared code or an exception becomes METHOD_HANDLER_FAILED (with a correlation id, logged with secrets masked), an unacknowledged write METHOD_OUTCOME_UNKNOWN — the same mapping as for a synchronous call. The work cannot accept again.
  • progress.ReportAsync(...) takes an optional percentage (0–100), an optional language-neutral stage code (UPPER_SNAKE_CASE) and up to 16 string parameters; never message text. Each report is also a heartbeat. A store problem while reporting is logged and never fails the work.
  • Returning Accepted from a method that is not declared execution.mode: async is a handler bug: the call is answered METHOD_HANDLER_FAILED and nothing is recorded.

Registration​

services.AddCkMethodHandlers("BotServices", m => m.Handle<BackupDumpHandler>("MyModel/Backup.Dump-1"));
services.AddCkMethodMongoOperationStore(); // package Meshmakers.Octo.Services.Methods.MongoDb, system database
services.AddCkMethodOperations(); // runner, heartbeat, sweep

Without AddCkMethodOperations() a handler that returns Accepted is answered METHOD_HANDLER_FAILED. A host that only reads operations (the asset repository, which serves runtime.operations) registers the store and AddCkMethodOperationSweep(). Tests and single-replica hosts may use AddCkMethodInMemoryOperationStore(), which is neither durable nor shared.

What happens on a call​

  1. The handler returns Accepted(work).
  2. The host writes the operation record before it answers. If the write fails, the caller gets METHOD_HANDLER_NOT_AVAILABLE (reason operation-store-unavailable) and the work never starts. A host that is shutting down records nothing and answers METHOD_HANDLER_NOT_AVAILABLE (reason host-stopping), so the caller can retry.
  3. The caller gets success: true with operation { operationId, acceptedAtUtc, retryAfterSeconds }; expiresAtUtc stays empty until the operation completes.
  4. The work starts: state RUNNING, progress, and a heartbeat every HeartbeatInterval.
  5. The work ends: SUCCEEDED with the result (none for a result the model marks sensitive) or FAILED with the code and its parameters; expiresAtUtc is the completion time plus the retention. A final record never changes again.
  6. A handler that dies (killed, crashed) sends no heartbeat. After StaleAfter the sweep of any replica ends the record FAILED / METHOD_OUTCOME_UNKNOWN / handler-lost. A late outcome of that work is dropped, and nothing restarts it.

The record holds the tenant, the operation id, the method id, the invocation and correlation id, the owner (the sub of a user, else the client_id of a service caller, taken from the validated principal), the state, progress, the result or error and the timestamps. It never contains the parameters, the caller's token or a sensitive result. It lives in the collection MethodOperations of the system database; every read names the tenant, and expired records are removed by a TTL index.

Deleted and re-created tenants. When a tenant is deleted, its records are removed. A tenant created later with the same id never sees a record of the deleted tenant, also when that removal failed or a call admitted just before the delete wrote its record afterwards: the store registrations (AddCkMethodMongoOperationStore(), AddCkMethodInMemoryOperationStore()) wrap the store so that every read ignores records accepted before the current tenant was created (the creation time of the tenant lifecycle record), and the setup of the bot services removes them. A host that registers its own ICkMethodOperationStore must wrap it in CkMethodTenantGenerationOperationStore itself, and a custom store without ICkMethodOperationTenantPurge keeps such records until they expire (invisible behind the wrapper).

Settings​

Methods:Operations (environment OCTO_Methods__Operations__*):

SettingDefaultBounds
Retention7 days1 hour to 30 days
HeartbeatInterval15 sat least 1 s
StaleAfter2 minat least 3 × HeartbeatInterval
SweepInterval1 minat least 1 s
RetryAfterSeconds51 to 3600

An invalid value stops the service at start; the message names the setting and its bounds, never the configured value.

Idempotency and metrics​

A keyed call to an asynchronous method behaves like any keyed call (R8): a repeat with the same key and parameters gets the stored accepted envelope with the same operationId, the handler is not called again, and only one operation exists — within the key lifetime: after the key has expired, a repeat starts a second operation, even while the first is still running, so a caller that retries late first lists its own operations (runtime.operations with methodId and since). The list cannot tell which entry belongs to a call whose accept answer was lost when several calls of the method match; then the state of the target decides whether to start the work again. A matching operation in state ACCEPTED or RUNNING may still change the target: wait until it has ended (SUCCEEDED or FAILED) before you read the target and decide on a retry. FAILED with METHOD_OUTCOME_UNKNOWN / handler-lost is no proof that the work has stopped: the sweep sets it when the heartbeats stop, it does not cancel the work, and a late outcome is only dropped. Treat the target as possibly still changing; retry only when its state shows that the work did not take effect and is not in progress. Two more limits: a method declared execution.idempotent: true ignores the key, so every call starts a new operation; and the replayed handle is only readable while the operation is kept, so keep Methods:Operations:Retention at least as long as the key lifetime Methods:Idempotency:Ttl (24 hours by default, at most 7 days) on every host — with a shorter retention a replay can return a handle whose read answers OPERATION_NOT_FOUND.

The counter ck.method.operations (meter Meshmakers.Octo.Methods, labels method_id and state) counts the state changes accepted, running, succeeded and failed; see Method metrics.

Testing an asynchronous handler​

  • Validation errors are answered synchronously, before anything is accepted.
  • The work reports progress and ends with the result or each declared error; an exception ends as METHOD_HANDLER_FAILED.
  • The work copies what it needs and does not use the caller's token or anything bound to the request.
  • The SDK's own tests (octo-common-services, tests/Methods.Tests/Operations/, tests/Methods.IntegrationTests/Operations/) cover the record order, the lost handler and the sweep; your handler does not repeat them.

Idempotency keys and your handler​

The idempotency filter (band 300) runs on the dispatcher side, in the gateway. Your handler does not implement anything for it and never sees the key or a repeat: a call that is answered from the store does not reach the handler. What a handler author must know:

  • A key makes the platform run the method at most once; it does not make your writes safe against two different calls (R7). Keep concurrent calls on one target safe.
  • A failure of your handler (METHOD_HANDLER_FAILED) burns the key for the caller, who must use a new key after reading the state. Keep your writes atomic (R3) so that a retry with a new key is safe.
  • A method with a sensitive result cannot be called with a key. If the first method that needs both appears, the decision OD-4 G2 has to be revisited.
  • execution.idempotent: true makes the platform skip the store: a keyed call to such a method is accepted, the key is ignored and the method runs every time, as if no key had been sent. Declare it only when repeating the call is really safe, because no repeat is ever answered from the store. Details: R8.

Rules for callers and the operator settings: R8.

Audit​

The audit filter is the outermost filter of the invocation pipeline (band 100). It writes one audit event per invocation id with the final outcome, so a call refused by authorization (band 200), a replay answered by the idempotency filter (band 300) and a timeout are audited like a success. A gateway that refuses a call before it reaches the dispatcher writes its event through the same shared recorder (ICkMethodAuditRecorder, register with AddCkMethodAudit()), so the event log holds one event per invocation whichever way the call ended. What the event contains, how replays, exceptions and cancellations are recorded, and the settings for a failing audit write (Methods:Audit:WriteTimeout, FailCallOnAuditFailure, FailCallOnAuditFailureTenants, answer METHOD_OUTCOME_UNKNOWN) are described in Audit: one event per invocation.

For handler and filter authors: the event never contains parameters (masked or not), results, tokens, idempotency keys or fingerprints, so a handler or filter has nothing to do for the audit, and must not write such values into an audit event itself.

The counters and histograms of the invocation pipeline, including the refused duplicate deliveries, are described in Method metrics.

Sensitive parameters and filters​

Parameters marked sensitive: true in the model are protected on the way to the handler. This section is for authors of invocation pipeline filters (audit, authorization, idempotency) and for handler authors who want to know what is masked.

  • Filters see a redacted view. The invocation context hands every filter a view of the request: the values of sensitive parameters (the whole value, also nested objects and arrays; names match case-insensitively) are replaced by "***", and the caller's access token is null. A payload that cannot be parsed, or that is not a JSON object, becomes {}. Only the transport, and so the handler, gets the raw values. An audit filter may therefore serialize the request it sees, and a filter cannot leak what it cannot see.
  • Fail-closed. A method that the definition source does not know (or that has no definition source at all) is redacted completely: every parameter value is masked. A validation error names the parameter (field) and never repeats its value.
  • A filter that edits the parameters keeps its edit, merged per member. When a filter passes the request on, the pipeline puts the raw values back for the transport. When it edits the parameters JSON (normalizes, enriches), the edit is used, merged member by member: a sensitive member that is still "***" is restored from the raw request, so editing a non-sensitive member never sends the placeholder to the handler; a sensitive member the filter gave another value is an explicit replacement and is used as is; a removed member stays removed. A "***" for a sensitive member that the request does not carry cannot be restored, and the call is refused with METHOD_VALIDATION_FAILED (field names the member) before the handler is called. A filter cannot tell the placeholder from a real value "***"; the raw value wins.
  • The response is redacted the same way. For a method whose result is marked sensitive (and for a method the platform does not know), the filters get the response with ResultJson replaced by the JSON string "***". The real response goes back to the caller as long as the filters pass the view on; a filter that returns a different response (for example a failed one) keeps its change, and a filter that answers without calling the next step is returned as is. Error responses, Success, the operation handle and the ids are never touched. CkMethodResponse.ToString() never prints the result, and the handler host never writes it into a log line, an exception message, a trace tag or a metric label. A filter behind which results are stored (idempotency) holds the redacted view, never the real value.
  • A filter cannot fingerprint secrets. Because it sees only the masked view, it cannot hash or compare the real values of sensitive parameters. Two different passwords look the same to a filter, so a key derived from the view must not be used to detect a duplicate call.
  • Exception text is masked in every written form. Exceptions that the pipeline and the dispatcher log, and exception messages a handler throws, are masked for the plain value and for the forms a serializer or a URL writes it in: JSON-escaped (quote, backslash, \uXXXX forms for plus, ampersand, angle brackets, apostrophe, control and non-ASCII characters, in upper and lower case hex, escaped slash; the System.Text.Json and Newtonsoft.Json encoders), the same once more for JSON inside a JSON string, and percent-encoded. A form that no serializer produces (a hash, a base64 copy, a truncation) is not recognized: never put a derived form of a secret into a message.
  • Mark a secret result. A result that carries a secret is marked sensitive: true in the model (Sensitive results); the platform then masks it for the filters, logs and traces (see the next bullet), while the caller still gets the real value. In every case, do not write a secret into a log, audit event or outbox payload yourself: the platform cannot know what a handler writes.

Canary scan on the running services​

The rules above are tested in-process in the platform libraries. The method canary scan checks them on the running services: with the local stack started, it calls methods through the typed method mutation of the asset repository with fake secret values (canaries) as a sensitive parameter, and then searches every place a value could end up for the canaries and for the caller's access token, in every form a serializer or URL writes them (plain, JSON-escaped, JSON in JSON, percent-encoded):

SearchedWhat the scan reads
GraphQL answersthe body of every call
Service log fileswhat each logFiles/*.log of Start-Octo gained during the scan
Audit and event logthe events of the scan tenant written during the scan
Exported traces, metrics and logsthe output of a local OpenTelemetry collector with the file exporter
The handling servicelog lines and spans of the handler host

Two methods are scanned: a test method of the asset repository (MethodCanaryScan/CanaryAccount.Rotate-1, handled by the scan's own handler host over the bus: success, failing handler, timeout, validation errors, a declared error and refused calls) and System.Identity/User.CreateUser-1 (sensitive password) of the identity service (success, declared errors, validation, refused calls). Every call also checks that it got the expected outcome, so a clean result cannot come from calls that never reached the method path.

Rerun it from octo-asset-repo-services (set-up of the collector, the stack and the scratch tenant: tests/AssetRepositoryServices.MethodCanaryScan/README.md):

OCTO_METHOD_CANARY_SCAN=1 OCTO_CANARY_TENANT=<scratch tenant> OCTO_CANARY_OTLP_DIR=<collector output> \
dotnet test tests/AssetRepositoryServices.MethodCanaryScan -c DebugL

The scan writes one report per method (calls, sinks, hits; secrets and token parts are replaced by labels). It fails on any hit and when no telemetry arrived. Every hit becomes a bug. Result of the first run (2026-11-05): the method path itself had no hit in GraphQL answers, log files, the audit log or the handler host; the identity service writes every issued access token into its log at level Trace (a known bug, outside the method path). Second run (2026-11-05, services exporting to a local collector): no hit in the exported traces and metrics of either method; the services export no logs over OTLP, their log output is covered by the log files. For the exporter, start the services with OTLP_ENDPOINT_URL (without the OCTO_ prefix) and, for the identity method, with OCTO_METHODS__IDENTITY__ENABLEPASSWORDMETHODS=true; the README of the scan has the details.

Performance and replica sharing​

All instances of a service that owns a method share one queue for it, and the broker delivers each call to exactly one of them (competing consumers). The dispatcher sends to a stable exchange and neither knows nor cares how many replicas exist. Tests against a real broker show that every caller gets one envelope, that every invocation runs exactly once, that load spreads over the replicas, and that a replica which is stopped gracefully in the middle of a burst loses no call. If the last replica stops, new calls fail fast with METHOD_HANDLER_NOT_AVAILABLE, and a new replica is served again as soon as it starts.

The overhead of the dispatcher itself is small: with a handler that does nothing, on one machine with broker, gateway and handlers on loopback, the 95th percentile of the round trip is about 1 ms, the same with one or three replicas. A cluster adds network hops to the broker and the handler's own time on top; plan the method timeout for the handler, not for the platform.

One rule for operators and authors of changes: never change the arguments of an existing method queue in place. If a newer build declares the queue differently than an older one, the broker refuses it, the new instance does not start, and the rollout waits on the old instances, which keep serving. A change of the queue arguments needs a new queue name so that both builds can run side by side.

Rolling update: stopping a replica​

A replica that is stopped gracefully (rolling update, scale down) first withdraws itself, then lets its running calls finish and answer, and removes its method endpoints. A call that runs is never cut off. The shared queue and the other replicas keep serving, so callers normally notice nothing.

One narrow case remains. In the moment the stop begins, the broker may already have pushed one or two calls to the stopping replica. The replica does not start them, and the broker hands them back to another replica flagged as redelivered. That replica refuses a flagged delivery as a possible duplicate (it cannot know that the first holder never started it), and the caller receives METHOD_OUTCOME_UNKNOWN although the call did not run. Nothing runs twice.

Traffic during the stopFlagged calls that never ran
Steady (about 80 calls/s, handler time 200-600 ms, 30 stops, 9000 calls)1, i.e. about 0.03 per stop
Synchronized bursts that coincide with the stop (worst case, 10 stops, 3000 calls)29, i.e. about 3 per stop

Measured on one machine with two replicas; the rate is the arrival rate of the stopping replica times the round trip of the broker's cancel, so a busier service sees proportionally more. What this means for you:

  • Treat METHOD_OUTCOME_UNKNOWN as always: read the state before you call again (R5). After a rolling update this is the expected, rare cause for a method that did not change anything.
  • A call with an idempotency key uses up its key on this answer (R8); send the next attempt with a new key after you have read the state.
  • The platform accepts this residual rate: the alternative would run a call twice after a lost connection. Removing it needs a change of the message transport (stop fetching before the consumer is marked as stopping) or a handler-side claim for calls with a key; neither is part of the current platform.
  • Hard loss of a replica (killed process, lost connection) gives more flagged calls than a graceful stop, and in that case the first execution may have run; see limits.

Timeouts and sizes​

  • execution.timeoutSeconds (1–300, default 15) is the time the caller waits. The deadline in the request is now plus this timeout, and the cancellation token of the handler is cancelled at that deadline. The dispatcher waits a short grace period (default 2 seconds) longer, so that the METHOD_TIMEOUT of your host arrives before the dispatcher answers itself. See deadlines and timeouts.
  • Choose the timeout for the slowest normal case, not the average. Work that can take longer than the maximum needs an asynchronous method that returns an operation handle (see Asynchronous handlers); for such a method the timeout bounds only the accept step.
  • Keep results small. They travel in one message.

Errors a handler returns​

You want to sayReturn
The caller may not do this (or you cannot tell if they may)METHOD_FORBIDDEN
A parameter is missing or wrongMETHOD_VALIDATION_FAILED, parameter field
The target is goneMETHOD_TARGET_NOT_FOUND
A business rule refuseda code from the method's errors, with string parameters
Something broke that you did not expectthrow; the host returns METHOD_HANDLER_FAILED with a correlation id
You cannot know whether your write happenedMETHOD_OUTCOME_UNKNOWN
You received the request after its deadlineMETHOD_TIMEOUT without executing (the host does this)

The platform codes and what a caller does with each are in the error code table. Never declare a code that starts with METHOD_.

Checklist for a new method​

  • Is it one method, or did I split an operation that must succeed or fail as a whole?
  • Parameters that are secrets are sensitive: true; none of them reaches a log, audit event, trace, outbox payload or error parameter.
  • authorization is declared (default is administrators only; the host enforces it before the handler runs), the service has enabled the caller token validation, and the handler does not use the caller data.
  • Order of checks: authenticate, authorize, look up, validate, write; the same answer for a missing and a forbidden target.
  • Declared error codes cover the business refusals; the model has no METHOD_ code.
  • The handler uses nothing bound to the request; the tenant comes from the request's tenant id.
  • Writes are atomic and committed before the answer; unknown commit state returns METHOD_OUTCOME_UNKNOWN.
  • The cancellation token is checked before a write.
  • Concurrent calls on the same target are safe.
  • idempotent is set honestly. A non-idempotent method that must not run twice is called with an idempotency key (R8); idempotent does not switch the key off.
  • A method with a sensitive result is never called with a key (it is refused, result-sensitive); callers read the state instead.
  • Side effects after the commit; an outbox where a lost message is not acceptable.
  • timeoutSeconds fits the slowest normal case; long work is an asynchronous method (execution.mode: async, service handler) that validates synchronously and reports progress.
  • Tests: each error code, forged caller, log scan for secrets, crash-before-answer reasoning, registration check.

Diagnostics: which handlers are registered​

Which service answers a method, how many instances are alive, and when each was last seen? The asset repository answers this with the GraphQL root query methodHandlers (written MethodHandlers in the schema description; GraphQL field names are camelCase).

Who may read it. System administrators only. The registry is platform-wide: it names services, instance ids, service versions and the tenant ids a handler serves, which is platform topology across all tenants. The query uses the same caller check as the cross-tenant system endpoints (orphan sweep, tenant registry reconcile): the token must be issued for the system tenant; a user token must also hold the role AdminPanelManagement; a client-credentials token of the system tenant passes without the role. Any other caller, including a tenant administrator of a customer tenant, gets methodHandlers: null and the error code MethodHandlersForbidden; no handler data is read or returned. There is no reduced view for tenant administrators.

{
methodHandlers {
answeredBy {
dispatcherInstanceId serviceVersion snapshotEnabled
snapshotLoadedAtUtc snapshotRowsRestored snapshotLoadFailed lastFlushUtc lastFlushFailed
}
methods(search: "Identity", states: [NO_LIVE_HANDLER, NOT_ANNOUNCED]) {
methodId transport state liveInstanceCount lastSeenUtc
instances {
serviceName instanceId version contractVersion tenants
lastSeenUtc expiresUtc state source persistedLastSeenUtc
}
}
}
}

Methods. The list is the union of three sources: the handler rows the answering gateway replica holds, the methods the gateway serves in-process, and the methods of the system tenant that declare a service handler (execution.handler: service, the default) and that no instance has announced.

state of a methodMeaning
LIVEAt least one instance is live, or the method is served in-process (transport: IN_PROCESS).
NO_LIVE_HANDLERInstances are known, but all are expired, withdrawn or have no queue bound. A call ends as METHOD_HANDLER_NOT_AVAILABLE.
NOT_ANNOUNCEDThe model declares a service handler, but no instance ever announced the method: the handler is missing (service not deployed, not started, or the method is not registered).

transport is BUS for methods handled by another service and IN_PROCESS for methods the gateway handles itself. liveInstanceCount counts live bus instances; an in-process method has no instances. lastSeenUtc of a method is the latest lastSeenUtc of its instances. The arguments search (case-insensitive part of the method id or of a service name) and states narrow the list.

Instances. Every known instance is listed, expired and withdrawn ones too (rows are kept for 24 hours).

FieldMeaning
stateLIVE, EXPIRED (no heartbeat within the announced TTL), WITHDRAWN (graceful stop) or NO_QUEUE_BOUND (the broker returned a mandatory send).
sourceHEARTBEAT when the row came from an announcement, a discovery answer or a withdrawal. SNAPSHOT when it was restored from the persisted snapshot after a gateway restart and no heartbeat has confirmed it yet. A SNAPSHOT row is never LIVE on its own authority: it is a last-seen record.
lastSeenUtc / expiresUtcLast announcement or withdrawal received, and the end of the announced TTL.
persistedLastSeenUtcThe last-seen time of the row in the snapshot store that all gateway replicas share. null if the store has no row for the instance, or the store could not be read (the query still answers and the failure is logged).
tenantsThe tenants the instance serves (* = all).

One replica answers. Each gateway replica holds its registry in memory, so the answer describes the replica that served the request. answeredBy names it: dispatcherInstanceId, serviceVersion, whether the snapshot is enabled, the last snapshot load (snapshotLoadedAtUtc, snapshotRowsRestored, snapshotLoadFailed) and the last successful write (lastFlushUtc, lastFlushFailed). Compare lastSeenUtc with persistedLastSeenUtc to see what other replicas wrote. Ask again to reach another replica.

The response holds service data only: ids, service names, versions, times, tenant ids and flags. It never contains method parameters, tokens or user data, and a failing snapshot store is reported as a flag, never as exception text.

In Refinery Studio​

System administrators see the same data without writing a query: in the system tenant, Operate › Method handlers (route /<system tenant>/diagnostics/method-handlers). The page needs the role AdminPanelManagement; in any other tenant, or without the role, the menu entry is hidden and a direct link shows a short notice instead of data.

The header reads "Answered by replica X (version Y)" with the snapshot facts of answeredBy (loaded at, rows restored, last flush, and a warning when a load or flush failed; "Snapshot off" when the snapshot is disabled). The list shows every method with its transport, state (Not announced is highlighted with an icon and text), live instance count and last seen time; search and state filters map to the search and states arguments. Expanding a method lists its instances with all fields above; rows restored from the snapshot are marked "Last seen, not confirmed live". The page is read-only, refreshes on demand, and has an optional auto refresh (every 15 seconds, off by default). Like the query, it shows what the replica that answers knows: refresh again to reach another replica.

See Also​