Implementing Method Handlers
A method is declared in a Construction Kit model. A handler is the code that runs when the method is called. This guide shows how to write one for each of the three ways a method can be handled, and which rules every handler follows. The consistency rules (transactions, failures, retries) are stated in the Method Consistency Model; read it first.
The wire contract (request, response envelope, error codes) is fixed. The handler SDK that this guide describes (Meshmakers.Octo.Services.Methods, typed handlers, AddCkMethodHandlers), the dispatcher, the in-process transport and the pipeline handler are released, and callers reach them through the typed GraphQL method mutations. Asynchronous methods with operation handles are released as well: a service handler can accept a long-running call and report its progress (see Asynchronous handlers and OD-5).
Three ways to handle a method
| Remote handler (bus) | In-process handler | Pipeline handler | |
|---|---|---|---|
| Declared by | execution.handler: service (default) | execution.handler: service | execution.handler: pipeline |
| Runs in | the service that owns the model (for example the identity service for System.Identity) | the gateway (asset repository) itself, for methods the gateway owns (planned: Tenant.*, Archive.*; none in production yet) | a pipeline on a mesh adapter |
| How it is reached | event hub, one queue per method, shared by all instances of the service | direct call in the same process; nothing is published to the bus | the communication controller starts the deployed pipeline whose well-known name equals the qualified method id |
| You write | a handler class | the same handler class, registered as in-process | a pipeline that ends with SetPipelineExecutionResult |
Whatever the transport, the request and response envelope, the language-neutral error codes, the deadline, the timeout, the consistency rules, authorization and audit are the same. The caller cannot tell the transports apart: it always receives a CkMethodResponse.
The dispatcher
Callers do not talk to a queue themselves. The gateway, a job or a pipeline calls the dispatcher (ICkMethodDispatcher), which resolves the handler, runs the call through the invocation pipeline (audit, authorization, idempotency) and returns a CkMethodResponse. It never throws for a failing call; only a cancellation by the caller surfaces.
- The queue of the declaring element. A method cannot be overridden in a derived type, so there is exactly one handler for a method: the one of the element that declares it. A call that targets a derived type reaches the queue of the type that declares the inherited method, and a call on a type that implements an interface reaches the queue of the interface method's declaring element. You register a handler once, for the declaring element, and it serves every type that inherits or implements the method.
- In-process first. If the service that runs the dispatcher has an in-process handler for the method (see below), the call is made directly in the same process and nothing is published to the bus. The same holds for a method the service itself hosts for the bus, when it calls that method through its own REST facade. Otherwise the dispatcher sends the call to the method's queue on the event hub. The same invocation pipeline applies to both transports, so one filter runs for both.
- The send is mandatory and never retried. If no queue is bound or nobody consumes it (service down, not yet announced, last replica stopped), the broker returns the message and the caller receives
METHOD_HANDLER_NOT_AVAILABLEat once (within one second locally), naming the method and carrying no stack trace. The dispatcher does not resend and has no publisher-side retry: a transport fault ends the call asMETHOD_TIMEOUTorMETHOD_HANDLER_NOT_AVAILABLE, never as a second delivery. Whether to retry is the caller's decision (retries); afterMETHOD_HANDLER_NOT_AVAILABLEit is always safe, because nothing ran. - One envelope. Success, a declared error and
METHOD_HANDLER_FAILEDarrive unchanged from the handler, whichever transport served the call.
Common rules
These apply to every handler, whatever the transport.
- Re-validate the caller. The bus is internal and not authenticated per message. The
callerin the request is informational. Authorize from the forwarded access token (signature, issuer, expiry, tenant, scope, and the audience if configured), never from thecallerdata. A forgedcallermust have no effect. The host does the validation for you when the service opts in: see Caller token validation. - Check in this order: authenticate → authorize → look anything up → validate parameters → write. Answer
METHOD_FORBIDDENwith the same text whether the target exists or not, so that a caller without the right cannot probe for entities. - Own your data access. There is no session in the method context and you must not look for one. Do not use services that are bound to the calling request: the HTTP context, the request's tenant repository, a
UserManageror store that finds its tenant from the HTTP request. Outside a request they fall back to the system tenant and would write into the wrong database. Resolve the tenant explicitly fromtenantIdof the request, create what you need for exactly that tenant and dispose it. - Make your writes atomic and commit before you answer. One document: one atomic update. Several: your own session and transaction. See R3.
- Honor the deadline. The cancellation token is cancelled at the handler deadline. Check it before you start a write; do not start one afterwards.
- Never throw to the bus, never retry. The platform does not retry a call either: a handler that throws after a side effect has run exactly once, and the caller receives
METHOD_HANDLER_FAILED. A delivery the broker flags as a possible duplicate, and an invocation id the instance has already seen, never reach your handler; the caller receivesMETHOD_OUTCOME_UNKNOWN. What is not detected is listed in limits, so keep a non-idempotent handler safe against the same logical operation arriving twice. Return an envelope for every outcome. An unexpected exception becomesMETHOD_HANDLER_FAILEDwith a correlation id; the stack trace goes to the log under that id, never on the wire. The host does this for you; do not catch and hide exceptions that would hide the real cause. - Declared errors are data. A business refusal is a code from the method's
errorslist with parameters (strings), never free text for the user. Parameter values of the method must not appear in error parameters. - No secrets in logs. Parameters marked
sensitive: trueand the access token are not logged, not audited, not put in traces, outbox payloads or exception messages. Log the invocation id, the method id, the tenant and the outcome. What the platform masks for you is described in Sensitive parameters and filters. - No state in instance fields. The next call can be served by another instance.
- Side effects after the commit. Publish or send only after the write committed; use an outbox if a lost message is not acceptable (side effects).
- Tolerate concurrency. Two calls on the same target can run at once. Use an atomic conditional update or a version check.
Caller token validation (remote handlers)
A request that arrives over the bus carries the access token of the caller (CallerAccessToken) next to the informational caller data. The bus is not authenticated per message, so only the token proves who is calling. A remote handler host validates that token before the parameters are read and before the handler runs, with the same trust as the HTTP API of its service. The decision and its alternatives are recorded as OD-10.
The service opts in once:
builder.Services.AddCkMethodCallerTokenValidation(o => o.Authority = "<authority of the HTTP API of this service>");
The settings live in the configuration section Methods:Token. A valid token reaches the handler as context.CallerPrincipal (subject, client id, tenant, IsServiceToken). A service that has not opted in gets CallerPrincipal == null and nothing is validated: such a handler has no verified caller and must refuse methods that need one. The in-process transport validates no token; it runs in the process that already authenticated the caller and passes that principal on. It does apply the tenant rule (row 6 below) to every in-process call made with an access token, centrally and before the handler, whatever mode the HTTP tenant gate runs in: a token of another tenant is refused with tenant-not-allowed; only a service token whose client is on TenantAuthorization:CrossTenantServiceClientIds passes, and a user token of another tenant only for the backup, restore and archive data methods of a direct child tenant (or any tenant, for the system tenant), see Parent-tenant administration through CK methods. A principal that a service builds itself for its own job carries no access token and is not checked; that service vouches for the tenant of its calls. An in-process handler called through the gateway needs no tenant check of its own.
The steps
| # | Check | On failure (METHOD_FORBIDDEN, parameter reason) |
|---|---|---|
| 1 | A token is present. | token-missing |
| 2 | Signature and issuer: the issuer is the authority of the service (with a trailing slash), the signing keys come from the identity service's key set. Only asymmetric algorithms are accepted; none and HS* are never accepted. | token-invalid |
| 3 | Expiry, with a clock skew of 30 seconds. | token-expired (only for a token that is otherwise genuine; an expired and forged token is token-invalid, an expired token of another tenant is tenant-not-allowed) |
| 4 | Audience, only if ValidAudiences is configured: the token carries one of the listed audiences. | audience-mismatch |
| 5 | Scope octo_api. | token-invalid |
| 6 | Tenant: the tenant_id claim equals the tenant of the request (case-insensitive). The allowed_tenants claim is only a hint for choosing a tenant and is never consulted. The only parent-tenant exception covers the backup, restore and archive data methods, see Parent-tenant administration through CK methods. | tenant-not-allowed |
| 7 | Subject: a user token carries sub. A token without sub is a service token (client credentials) and must carry client_id. | token-invalid |
The answer never names the claim that failed, so it gives a forger nothing to probe. The handler is not called and nothing is written. The log line of a refusal carries the tenant, the reason and an internal category, never the token, a claim value or a key id. If the host cannot load any signing key (identity unreachable and nothing cached), the call is answered METHOD_HANDLER_FAILED: the caller did nothing wrong, and a token is never accepted without a verified signature.
Service tokens
Jobs, pipelines and other services call methods with a client-credentials token, which has no sub. Such a token is accepted when it carries client_id and its tenant_id equals the request tenant, or when its client is on the allow-list TenantAuthorization:CrossTenantServiceClientIds (the same list as for the HTTP API). The validation is always fail-closed: the LogOnly and Disabled modes of the HTTP side are never applied. The subject of the principal is the client id.
Audience
The platform does not validate the audience on its HTTP APIs, so Methods:Token:ValidAudiences is empty by default and the audience is not checked. With it empty, every octo_api token of the tenant is accepted for a method call. Set it as soon as the identity service issues audience-scoped tokens.
Expired tokens
token-expired is distinct from the other reasons so that a client can tell "get a fresh token and call again" from "not allowed". The code stays METHOD_FORBIDDEN. A call waits in the queue at most as long as its timeout (1 to 300 seconds, default 15), so a token that was valid when the call was made rarely expires while the call waits.
Pipeline handlers
The pipeline handler of the communication controller validates the forwarded token with the same validator and builds the caller of the pipeline from the validated principal only, with trust level strong; the caller data of the request is never used. A call without a valid token is answered METHOD_FORBIDDEN (same reasons as above) and no pipeline is started. It needs no opt-in: the controller takes its authority from the settings of its own HTTP API (details). A remote handler that is written as a service handler gets the validated principal from the host as described above.
What the token does not protect
The forwarded token is valid for the APIs of the tenant until it expires, and it is delivered to whoever consumes the method queue. Validation protects a handler against a forged caller; it does not protect the caller against a receiver on the broker. Who may consume the queue is decided and restricted at the broker, see OD-11, including the rule that no method that carries a password may run on a shared staging or production environment before the broker is restricted.
Worked example: User.ChangePassword
The method changes the password of a user. A user changes their own password (and must give the current one); a user with the UserManagement role resets the password of someone else (no current password needed). The example is taken from the identity service; the model is shown as it would be declared and contains no real values.
1. The method in the model
types:
- typeId: User
# ...
methods:
- methodId: ChangePassword-1
description: Changes (self service) or resets (administrator) the password of a user.
parameters:
- name: currentPassword
valueType: String
isOptional: true # required for self service, ignored for a reset
sensitive: true
- name: newPassword
valueType: String
sensitive: true
errors:
- code: PASSWORD_POLICY_VIOLATION
- code: CURRENT_PASSWORD_INVALID
- code: USER_LOCKED_OUT
authorization:
roles: [ UserManagement ]
allowSelf: true
execution:
timeoutSeconds: 15
idempotent: false # a second call with a new password is not the same as the first
The source generator emits, next to the model, the id constant SystemIdentityCkIds.UserChangePasswordMethodId (System.Identity/User.ChangePassword-1) and the record UserChangePasswordParameters with CurrentPassword and NewPassword. Both parameters are sensitive: the generated ToString() prints ***.
authorization is checked at the gateway and by the handler host: the declaration says who may call; before the handler runs, the host evaluates it again for the validated caller, with the same evaluator as the gateway. The handler does not need to repeat the declared check.
Where the declaration is enforced. The platform enforces authorization without any handler code, in four places that use the same rules: the gateway checks before it loads the target (GraphQL callers); the authorization filter of the invocation pipeline (band 200, AddCkMethodAuthorization() in Meshmakers.Octo.Services.Methods) checks every call that reaches the dispatcher, over the bus and in-process alike, and is the only check for callers that do not use the gateway (REST, jobs, pipelines); the handler host evaluates the declaration for the validated caller after the token validation and before the parameters are read, in the remote host and in the in-process host alike, so a valid token of the tenant that is published straight on the bus cannot reach a handler whose roles, scopes or allowSelf it does not meet; a method without a definition is refused, and a remote host that has not enabled the caller token validation has no validated caller and refuses every call; the handler re-validates what only it can know (rule 1). The filter takes the caller from the process that starts the call (a validated token, or the identity of the job), never from the caller data of a request that arrives over the bus. It runs inside the audit filter (band 100), so a refusal is audited, and outside the idempotency filter (band 300) and the transport, which a refused call never reaches. A refusal is the language-neutral error METHOD_FORBIDDEN with the parameters methodId and reason: not-authorized; it never says which rule failed and is identical whether the target exists or not.
2. The handler
A handler is a class with one method. The host deserializes the parameters into the generated record, serializes the result and turns exceptions into envelopes.
internal sealed class ChangePasswordHandler(ChangePasswordService service)
: ICkMethodHandler<UserChangePasswordParameters> // no result
{
public async Task<CkMethodResult> InvokeAsync(
CkMethodContext context, // tenant, caller token, target id, deadline
UserChangePasswordParameters parameters,
CancellationToken cancellationToken) // cancelled at the handler deadline
{
// Thin adapter: everything that matters lives in a transport-independent service,
// which is unit-testable without the messaging package.
var outcome = await service.ChangeAsync(
context.TenantId, context.TargetRtId, context.CallerAccessToken, parameters, cancellationToken);
return outcome.Success
? CkMethodResult.Ok()
: CkMethodResult.Fail(outcome.ErrorCode, outcome.ErrorParameters);
}
}
The service performs the algorithm in the order of rule 2:
| Step | What | On failure |
|---|---|---|
| 1 | Take the caller from context.CallerPrincipal, which the host filled after it validated the token (steps: signature, issuer, expiry, tenant_id equals the request tenant, scope octo_api, optional audience). A service that has not opted in has no validated caller; its host refuses the call before the handler runs. | METHOD_FORBIDDEN |
| 2 | Authorize on parsed ids, not strings: self when the principal's subject (sub) equals the target id, administrator when the token carries UserManagement. Compare OctoObjectIds (the parser accepts upper-case hex, a string comparison would not). | METHOD_FORBIDDEN — the same answer whether the target exists or not |
| 3 | Resolve the tenant context from the request's tenant id; check that the tenant has the identity model. | METHOD_HANDLER_FAILED |
| 4 | Load the target user with a user manager created for that tenant (see rule 3), disposed after the call. | METHOD_TARGET_NOT_FOUND |
| 5 | Validate parameters: newPassword present; currentPassword present for self service. | METHOD_VALIDATION_FAILED with parameter field |
| 6 | Refuse a locked-out user. | USER_LOCKED_OUT |
| 7 | Check the cancellation token, then write. Self: change with the current password; a wrong current password counts toward lockout like a failed sign-in. Administrator: reset. The write updates the password hash and the security stamp in one operation. | CURRENT_PASSWORD_INVALID, PASSWORD_POLICY_VIOLATION |
| 8 | Map the result to the envelope; log one line with the invocation id, tenant, target id, caller id and outcome — no parameters, no token. |
Consistency decisions in this example:
- The password change is a single logical write done by the identity framework in one operation, so it is atomic (R3). The handler does not need its own multi-document transaction.
- It is not idempotent: repeating it with a different password is a different request. A repeated call with the same parameters from the same user usually fails the second time with
CURRENT_PASSWORD_INVALID(a natural guard, not a guarantee). - If the handler crashes after the write and before the answer, the caller receives
METHOD_TIMEOUTalthough the password changed (failure example 1). The client reads the state by signing in with the new password. - A notification mail after an administrator reset is a side effect: send it after the commit, through a mail path that is bound to that tenant (not through a service that finds its tenant from the HTTP request), and treat a mail failure as a warning, not as a failure of the method.
- Session revocation after a password change (refresh tokens) is a separate write in another store. It belongs after the commit, with an outbox or a compensation if it must not be lost.
3. Registration
// Program.cs of the service that owns the model
builder.Services.AddCkMethodHandlers("IdentityService", methods =>
methods.Handle<ChangePasswordHandler>(SystemIdentityCkIds.UserChangePasswordMethodId));
At startup the host checks the registration against the CK cache: an unknown method id or a parameter type that does not match the model stops the service with a message that names the method. The host then announces the handler on the event hub after its endpoint is ready, repeats the announcement as a heartbeat and withdraws before shutdown. You never touch queues, JSON or announcements.
With two replicas of the identity service both announce; the queue of the method is shared, so each call goes to one of them.
The gateway remembers its handlers across a restart. The gateway (and every other host that dispatches methods) keeps its list of announced handlers as a snapshot in the system database, so a restarted gateway answers a call without waiting for the next heartbeat round. The snapshot is a last seen record, not a promise that the handler is alive: a row counts as live only for as long as the heartbeat window it was announced with (45 seconds by default) has not passed. A handler that has stopped, whose window has expired or that withdrew is never used; the call ends as METHOD_HANDLER_NOT_AVAILABLE. The snapshot holds service data only (method id, service name, instance id, versions, times, served tenants), never a parameter value, token or user data. Without the snapshot store the list lives in memory only, exactly as before.
Heartbeat and TTL. A handler host repeats its announcement every 15 seconds and each announcement is valid for 45 seconds. These two values are final defaults; you normally do not change them. They are set in the Methods section of the host's configuration:
| Key | Environment variable | Default |
|---|---|---|
Methods:HeartbeatInterval | OCTO_Methods__HeartbeatInterval | 00:00:15 |
Methods:Ttl | OCTO_Methods__Ttl | 00:00:45 |
- Rules.
HeartbeatIntervalmust be at least 1 second,Ttlat least twice theHeartbeatInterval(so a live handler survives one lost heartbeat) and at most 24 hours. A host that breaks a rule does not start; the error names the key and the rule, for exampleMethods:Ttl is 00:00:20 but must be at least 2 x Methods:HeartbeatInterval (00:00:30). - Different values per host. Every announcement carries its own TTL, and a dispatcher counts it from the moment it receives the announcement. Hosts of different services may therefore use different values without any setting on the gateway.
- Why these values. The list of handlers is advisory. Whether a call fails fast with
METHOD_HANDLER_NOT_AVAILABLEis decided by the broker, which refuses a message nobody can receive, not by the TTL. The TTL only decides what diagnostics and the snapshot show as live. 45 seconds tolerates two lost heartbeats; shorter values only add message traffic.
4. Tests
| Test | Proves |
|---|---|
| Unit tests of the service with fake token validation | every step and error code of the table above, including that a forged caller is rejected and the same METHOD_FORBIDDEN text for existing and missing targets |
| Log capture test | no password and no token in any log line, also on failure paths |
| Integration test against a real database | the password is changed, the security stamp changes, lockout counts; a cancelled token before step 7 writes nothing |
| Handler registration test | a wrong method id fails at startup |
| Failure test | handler throws → METHOD_HANDLER_FAILED with correlation id, no stack trace in the envelope |
The same handler as an in-process handler
The asset repository will own some methods itself (planned: Tenant.*, Archive.*; no production method is registered yet, the transport is released). Their handlers are the same classes; they are registered as in-process, so the dispatcher finds them before it looks for a remote handler. Nothing is published to the bus and the handler is not announced as a remote handler. The envelope, the deadline and the exception mapping are the same, and so are the pipeline steps for authorization and audit that run for every transport.
What changes for you:
- There is no network between the gateway and the handler, so it is tempting to use what is at hand: the request's database session, the HTTP context, the request-scoped repository. Do not (rule 3). The method is a separate unit of work. An in-process handler is resolved in its own scope; a handler that reaches for the request's session would silently join the request transaction and break R1. The platform makes this a guarantee (OD-8), not only a rule: the in-process transport runs every handler detached from the ambient execution context of the caller, in a fresh dependency-injection scope per call. A value that the caller keeps in an
AsyncLocal(the request session of the gateway, a principal scope, a logging scope) is empty inside the handler, also after its firstawait, and nothing the handler sets reaches the caller. A new scope alone would not be enough, because ambient values travel with the asynchronous flow, not with the scope.
Carried across on purpose: the trace (the span of the handler is a child of the caller's activity, so the trace of a request continues through the call), the culture of the caller (CurrentCulture and CurrentUICulture), the logging scope that identifies the call (MethodId, InvocationId, CorrelationId, TenantId), and the cancellation token with the deadline. Nothing else is carried; a library that reads another ambient value inside a handler sees its default. Pass what a handler needs in the request.
As a tripwire, a service registers a probe for each kind of request-bound state it owns (AddCkInProcessAmbientStateProbe<TProbe>(); the gateway registers one for the request session, HasSession). The probe is asked when the handler starts, in the handler's own flow. If it reports state, the call fails with METHOD_HANDLER_FAILED and the handler does not run; the log names the probe, never the data. With the detach in place this never happens; it catches a regression.
- Deadlines and timeouts apply as for a remote call.
- A handler that is registered for both transports (for example in a test) behaves the same way; that is the test of a transport-independent handler.
A REST facade over a method the service hosts
A service that hosts a method handler can serve the same operation through its own REST endpoint, a facade (for example the
identity service's user endpoints over the User.* methods). The facade calls the method through the dispatcher
like any other caller, and the handler runs in the same process through the same invocation pipeline as a call from another
service: declared authorization, parameter validation, masking of sensitive parameters, audit and the ck.method.* metrics
(transport in-process). There is no bus hop, so no parameter (a password, for example) reaches the broker.
AddCkMethodHandlers registers this local transport by itself; AddCkMethodDispatcher gives the facade its ICkMethodDispatcher.
Two registrations, one rule.
| Registration | Announced, bus queue | Another process calls it | This process calls it |
|---|---|---|---|
m.Handle<THandler>(methodId) in AddCkMethodHandlers | yes | over the bus | in-process (nothing is sent) |
m.HandleLocalOnly<THandler>(methodId) in AddCkMethodHandlers | no | METHOD_HANDLER_NOT_AVAILABLE (no queue) | in-process |
m.Handle<THandler>(methodId) in AddCkInProcessMethodHandlers | no | METHOD_HANDLER_NOT_AVAILABLE | in-process |
HandleLocalOnly<THandler>(methodId)is for a method whose bus exposure is switched off while its facade must keep working, for example the identity password methods while the broker is not protected:EnablePasswordMethods ? m.Handle<H>(id) : m.HandleLocalOnly<H>(id).- One place per method. A method is registered either for the bus or local only; registering both for the same id stops the service at startup.
- Same envelope. A bus-registered method that its own host calls answers with the announced handler instance id, so the envelope is byte-identical to the one a remote caller gets from that host.
- Methods the service does not host still go over the bus. If a facade must never leave the process, check
ICkInProcessMethodTransport.CanHandle(methodId)at startup.
The caller. The local path validates no token. The principal is the one the facade sets with CkMethodPrincipalScope, taken
from the authenticated HTTP user. Without a scope there is no principal, and the declared authorization refuses the call with
METHOD_FORBIDDEN; nothing falls back to a system identity. Like every method caller, the principal needs the scope octo_api.
The result. CkMethodFacadeResults.ToActionResult(response, options) (namespace Meshmakers.Octo.Services.Methods.Facades)
turns the envelope into an MVC IActionResult:
| Outcome | Default status |
|---|---|
| success with a result / without a result | 200 with the result JSON / 204 |
| accepted (asynchronous call, operation handle set) | 202 with the operation handle |
METHOD_VALIDATION_FAILED | 400 |
METHOD_FORBIDDEN | 403 |
METHOD_TARGET_NOT_FOUND | 404 |
METHOD_REQUEST_IN_PROGRESS | 409 |
METHOD_PARAMETERS_TOO_LARGE | 413 |
METHOD_HANDLER_FAILED, METHOD_OUTCOME_UNKNOWN, an unknown METHOD_* code | 500 |
METHOD_HANDLER_NOT_AVAILABLE, METHOD_CONTRACT_VERSION_UNSUPPORTED, METHOD_NOT_SUPPORTED_BY_HANDLER_VERSION | 503 |
METHOD_TIMEOUT | 504 |
| a code the method declares | 400 |
The default error body is problem details (application/problem+json) with the code as title and the extensions code,
parameters and details. Error parameters never carry parameter values, so the body is safe to return. An endpoint that must
keep its existing status codes and bodies configures CkMethodFacadeOptions: MapError(code, status),
MapError(code, error => result) (also for platform codes) and MapSuccess(response => result). Build the options once per
endpoint.
// Program.cs of the identity service
services.AddCkMethodHandlers("IdentityService", m =>
{
m.Handle<UserSetRolesHandler>(SystemIdentityCkIds.UserSetRolesMethodId);
if (options.EnablePasswordMethods) m.Handle<UserResetPasswordHandler>(SystemIdentityCkIds.UserResetPasswordMethodId);
else m.HandleLocalOnly<UserResetPasswordHandler>(SystemIdentityCkIds.UserResetPasswordMethodId);
});
services.AddCkMethodDispatcher(); // the facade's entry point; local methods never touch the bus
// UsersController
private static readonly CkMethodFacadeOptions ResetPasswordFacade = new CkMethodFacadeOptions()
.MapError("PASSWORD_POLICY_VIOLATED", error => new BadRequestObjectResult(error.Details))
.MapSuccess(_ => new NoContentResult());
[HttpPost("{id}/reset-password")]
public async Task<IActionResult> ResetPassword(string tenantId, string id, ResetPasswordDto dto)
{
var request = new CkMethodRequest(tenantId)
{
InvocationId = Guid.NewGuid(),
MethodId = SystemIdentityCkIds.UserResetPasswordMethodId,
CkTypeId = "System.Identity/User",
TargetRtId = id,
ParametersJson = JsonSerializer.Serialize(new UserResetPasswordParameters { NewPassword = dto.Password }, CkMethodJson.Options),
DeadlineUtc = timeouts.GetDeadlineUtc(tenantId, "System.Identity/User", SystemIdentityCkIds.UserResetPasswordMethodId)
};
using (CkMethodPrincipalScope.Begin(CkMethodPrincipal.FromClaimsPrincipal(User)))
{
var response = await dispatcher.InvokeAsync(request, HttpContext.RequestAborted);
return response.ToActionResult(ResetPasswordFacade);
}
}
A pipeline as handler
A method with execution.handler: pipeline is implemented by a pipeline instead of C# code. This suits orchestration of existing nodes (read, transform, call, write) and models that do not own a service.
methods:
- methodId: UpdateFirmware-1
parameters:
- name: version
valueType: String
execution:
timeoutSeconds: 120 # mandatory for a pipeline method
handler: pipeline
How the call is handled:
- The communication controller announces itself as handler of every method whose compiled definition says
handler: pipeline. - For a call it looks up, in the call's tenant, the deployed pipeline whose well-known name equals the qualified method id (for example
Acme.Assets/Device.UpdateFirmware-1; no further binding exists). No such pipeline:METHOD_HANDLER_NOT_AVAILABLEwithreasonpipeline-not-deployed. A changed pipeline takes effect after it is deployed again. - It starts the pipeline with the method parameters as input and the caller's identity, exactly as for a manual execution, so the pipeline runs with the caller's rights.
- The pipeline starts with the execute-command trigger and ends with
SetPipelineExecutionResult, whose output must match the method's result record. - The outcome maps to the envelope: success with matching result → success; pipeline failure →
METHOD_HANDLER_FAILED; an error code set by the pipeline → that code; a result that does not match the result record →METHOD_HANDLER_FAILEDwith a validation parameter; the deadline passes →METHOD_TIMEOUT.
Consistency of a pipeline handler:
- A pipeline run is its own unit of work like any handler; the pipeline's nodes commit as they run. There is no rollback of earlier nodes when a later node fails. Design the pipeline so that the write nodes come last, or model compensation as nodes.
- On
METHOD_TIMEOUTthe pipeline may still finish. This is documented, not prevented. A non-idempotent pipeline method is the author's responsibility. - A method with a
sensitiveparameter cannot be a pipeline method: a pipeline execution stores its input and output, so the compiler refuseshandler: pipelinefor such a method with message 132 (decision OD-9). Implement it as a service handler. The pipeline handler refuses such a method at runtime as well (reasonsensitive-parameters-not-supported). The same holds for a result markedsensitive: true, which the compiler refuses for a pipeline method with message 134 (Sensitive results); the pipeline handler refuses it at runtime too, withreasonsensitive-result-not-supported.
The complete, tested sample pipeline for a pipeline-handled method is in Pipeline method handlers.
Asynchronous handlers
A method declared execution.mode: async (Execution mode) does its work after the call has returned. The handler accepts the call, the platform writes a durable operation record and answers the caller with an operation handle; the caller reads state, progress and result later through the handle (GraphQL runtime.operations, see Asynchronous methods and operations). The guarantees are decided in OD-5: at most once after accept, no cancellation, a lost handler ends as METHOD_OUTCOME_UNKNOWN.
Only a service handler can be asynchronous, and by decision OD-10 only one of a bot job type (the bot services run the work; the repository dump is the first). A pipeline handler answers synchronously, and the compiler refuses mode: async together with handler: pipeline (message 142). For an asynchronous method, timeoutSeconds bounds the accept step, not the operation.
The handler
Validate the parameters synchronously and answer a declared error at once when they are wrong. Return Accepted only when the work can start:
public sealed class BackupDumpHandler(IDumpService dumps)
: ICkMethodHandler<BackupDumpParameters, BackupDumpResult>
{
public Task<CkMethodOutcome<BackupDumpResult>> HandleAsync(
CkMethodContext<BackupDumpParameters> context, CancellationToken cancellationToken)
{
if (string.IsNullOrEmpty(context.Parameters.Target))
{
return Task.FromResult(CkMethodOutcome<BackupDumpResult>.ValidationFailed("target", "required"));
}
var tenantId = context.TenantId; // copy what the work needs; never keep the caller's token
return Task.FromResult(CkMethodOutcome<BackupDumpResult>.Accepted(async (progress, ct) =>
{
await progress.ReportAsync(CkMethodOperationProgress.Create(10, "COPYING_COLLECTIONS"), ct);
var file = await dumps.DumpAsync(tenantId, ct);
return new BackupDumpResult { Value = file }; // or a declared error
}));
}
}
CkMethodOutcome.Accepted(work) is the same for a method without a result. The work delegate receives a progress reporter (ICkMethodOperationProgress, which also carries the OperationId) and a cancellation token. The repository dump of the bot services (System.Bot/TenantRepository.Dump) is the first method in production built this way.
Rules for the work:
- It runs once, after the accepted answer has left, detached from the caller (no ambient state of the request, as for an in-process handler), in the handler's dependency-injection scope, which stays open until the work ends. Copy what the work needs from the context before you return; the caller's token is not kept (OD-10).
- Its token is cancelled when the host stops. The operation then ends
FailedwithMETHOD_OUTCOME_UNKNOWN(reasonhandler-lost). There is no resume and no retry. - It ends with a result or a declared error. An undeclared code or an exception becomes
METHOD_HANDLER_FAILED(with a correlation id, logged with secrets masked), an unacknowledged writeMETHOD_OUTCOME_UNKNOWN— the same mapping as for a synchronous call. The work cannot accept again. progress.ReportAsync(...)takes an optional percentage (0–100), an optional language-neutral stage code (UPPER_SNAKE_CASE) and up to 16 string parameters; never message text. Each report is also a heartbeat. A store problem while reporting is logged and never fails the work.- Returning
Acceptedfrom a method that is not declaredexecution.mode: asyncis a handler bug: the call is answeredMETHOD_HANDLER_FAILEDand nothing is recorded.
Registration
services.AddCkMethodHandlers("BotServices", m => m.Handle<BackupDumpHandler>("MyModel/Backup.Dump-1"));
services.AddCkMethodMongoOperationStore(); // package Meshmakers.Octo.Services.Methods.MongoDb, system database
services.AddCkMethodOperations(); // runner, heartbeat, sweep
Without AddCkMethodOperations() a handler that returns Accepted is answered METHOD_HANDLER_FAILED. A host that only reads operations (the asset repository, which serves runtime.operations) registers the store and AddCkMethodOperationSweep(). Tests and single-replica hosts may use AddCkMethodInMemoryOperationStore(), which is neither durable nor shared.
What happens on a call
- The handler returns
Accepted(work). - The host writes the operation record before it answers. If the write fails, the caller gets
METHOD_HANDLER_NOT_AVAILABLE(reasonoperation-store-unavailable) and the work never starts. A host that is shutting down records nothing and answersMETHOD_HANDLER_NOT_AVAILABLE(reasonhost-stopping), so the caller can retry. - The caller gets
success: truewithoperation { operationId, acceptedAtUtc, retryAfterSeconds };expiresAtUtcstays empty until the operation completes. - The work starts: state
RUNNING, progress, and a heartbeat everyHeartbeatInterval. - The work ends:
SUCCEEDEDwith the result (none for a result the model markssensitive) orFAILEDwith the code and its parameters;expiresAtUtcis the completion time plus the retention. A final record never changes again. - A handler that dies (killed, crashed) sends no heartbeat. After
StaleAfterthe sweep of any replica ends the recordFAILED/METHOD_OUTCOME_UNKNOWN/handler-lost. A late outcome of that work is dropped, and nothing restarts it.
The record holds the tenant, the operation id, the method id, the invocation and correlation id, the owner (the sub of a user, else the client_id of a service caller, taken from the validated principal), the state, progress, the result or error and the timestamps. It never contains the parameters, the caller's token or a sensitive result. It lives in the collection MethodOperations of the system database; every read names the tenant, and expired records are removed by a TTL index.
Deleted and re-created tenants. When a tenant is deleted, its records are removed. A tenant created later with the same id never sees a record of the deleted tenant, also when that removal failed or a call admitted just before the delete wrote its record afterwards: the store registrations (AddCkMethodMongoOperationStore(), AddCkMethodInMemoryOperationStore()) wrap the store so that every read ignores records accepted before the current tenant was created (the creation time of the tenant lifecycle record), and the setup of the bot services removes them. A host that registers its own ICkMethodOperationStore must wrap it in CkMethodTenantGenerationOperationStore itself, and a custom store without ICkMethodOperationTenantPurge keeps such records until they expire (invisible behind the wrapper).
Settings
Methods:Operations (environment OCTO_Methods__Operations__*):
| Setting | Default | Bounds |
|---|---|---|
Retention | 7 days | 1 hour to 30 days |
HeartbeatInterval | 15 s | at least 1 s |
StaleAfter | 2 min | at least 3 × HeartbeatInterval |
SweepInterval | 1 min | at least 1 s |
RetryAfterSeconds | 5 | 1 to 3600 |
An invalid value stops the service at start; the message names the setting and its bounds, never the configured value.
Idempotency and metrics
A keyed call to an asynchronous method behaves like any keyed call (R8): a repeat with the same key and parameters gets the stored accepted envelope with the same operationId, the handler is not called again, and only one operation exists — within the key lifetime: after the key has expired, a repeat starts a second operation, even while the first is still running, so a caller that retries late first lists its own operations (runtime.operations with methodId and since). The list cannot tell which entry belongs to a call whose accept answer was lost when several calls of the method match; then the state of the target decides whether to start the work again. A matching operation in state ACCEPTED or RUNNING may still change the target: wait until it has ended (SUCCEEDED or FAILED) before you read the target and decide on a retry. FAILED with METHOD_OUTCOME_UNKNOWN / handler-lost is no proof that the work has stopped: the sweep sets it when the heartbeats stop, it does not cancel the work, and a late outcome is only dropped. Treat the target as possibly still changing; retry only when its state shows that the work did not take effect and is not in progress. Two more limits: a method declared execution.idempotent: true ignores the key, so every call starts a new operation; and the replayed handle is only readable while the operation is kept, so keep Methods:Operations:Retention at least as long as the key lifetime Methods:Idempotency:Ttl (24 hours by default, at most 7 days) on every host — with a shorter retention a replay can return a handle whose read answers OPERATION_NOT_FOUND.
The counter ck.method.operations (meter Meshmakers.Octo.Methods, labels method_id and state) counts the state changes accepted, running, succeeded and failed; see Method metrics.
Testing an asynchronous handler
- Validation errors are answered synchronously, before anything is accepted.
- The work reports progress and ends with the result or each declared error; an exception ends as
METHOD_HANDLER_FAILED. - The work copies what it needs and does not use the caller's token or anything bound to the request.
- The SDK's own tests (octo-common-services,
tests/Methods.Tests/Operations/,tests/Methods.IntegrationTests/Operations/) cover the record order, the lost handler and the sweep; your handler does not repeat them.
Idempotency keys and your handler
The idempotency filter (band 300) runs on the dispatcher side, in the gateway. Your handler does not implement anything for it and never sees the key or a repeat: a call that is answered from the store does not reach the handler. What a handler author must know:
- A key makes the platform run the method at most once; it does not make your writes safe against two different calls (R7). Keep concurrent calls on one target safe.
- A failure of your handler (
METHOD_HANDLER_FAILED) burns the key for the caller, who must use a new key after reading the state. Keep your writes atomic (R3) so that a retry with a new key is safe. - A method with a
sensitiveresult cannot be called with a key. If the first method that needs both appears, the decision OD-4 G2 has to be revisited. execution.idempotent: truemakes the platform skip the store: a keyed call to such a method is accepted, the key is ignored and the method runs every time, as if no key had been sent. Declare it only when repeating the call is really safe, because no repeat is ever answered from the store. Details: R8.
Rules for callers and the operator settings: R8.
Audit
The audit filter is the outermost filter of the invocation pipeline (band 100). It writes one audit event per invocation id with the final outcome, so a call refused by authorization (band 200), a replay answered by the idempotency filter (band 300) and a timeout are audited like a success. A gateway that refuses a call before it reaches the dispatcher writes its event through the same shared recorder (ICkMethodAuditRecorder, register with AddCkMethodAudit()), so the event log holds one event per invocation whichever way the call ended. What the event contains, how replays, exceptions and cancellations are recorded, and the settings for a failing audit write (Methods:Audit:WriteTimeout, FailCallOnAuditFailure, FailCallOnAuditFailureTenants, answer METHOD_OUTCOME_UNKNOWN) are described in Audit: one event per invocation.
For handler and filter authors: the event never contains parameters (masked or not), results, tokens, idempotency keys or fingerprints, so a handler or filter has nothing to do for the audit, and must not write such values into an audit event itself.
The counters and histograms of the invocation pipeline, including the refused duplicate deliveries, are described in Method metrics.
Sensitive parameters and filters
Parameters marked sensitive: true in the model are protected on the way to the handler. This section is for authors of invocation pipeline filters (audit, authorization, idempotency) and for handler authors who want to know what is masked.
- Filters see a redacted view. The invocation context hands every filter a view of the request: the values of sensitive parameters (the whole value, also nested objects and arrays; names match case-insensitively) are replaced by
"***", and the caller's access token isnull. A payload that cannot be parsed, or that is not a JSON object, becomes{}. Only the transport, and so the handler, gets the raw values. An audit filter may therefore serialize the request it sees, and a filter cannot leak what it cannot see. - Fail-closed. A method that the definition source does not know (or that has no definition source at all) is redacted completely: every parameter value is masked. A validation error names the parameter (
field) and never repeats its value. - A filter that edits the parameters keeps its edit, merged per member. When a filter passes the request on, the pipeline puts the raw values back for the transport. When it edits the parameters JSON (normalizes, enriches), the edit is used, merged member by member: a sensitive member that is still
"***"is restored from the raw request, so editing a non-sensitive member never sends the placeholder to the handler; a sensitive member the filter gave another value is an explicit replacement and is used as is; a removed member stays removed. A"***"for a sensitive member that the request does not carry cannot be restored, and the call is refused withMETHOD_VALIDATION_FAILED(fieldnames the member) before the handler is called. A filter cannot tell the placeholder from a real value"***"; the raw value wins. - The response is redacted the same way. For a method whose result is marked
sensitive(and for a method the platform does not know), the filters get the response withResultJsonreplaced by the JSON string"***". The real response goes back to the caller as long as the filters pass the view on; a filter that returns a different response (for example a failed one) keeps its change, and a filter that answers without calling the next step is returned as is. Error responses,Success, the operation handle and the ids are never touched.CkMethodResponse.ToString()never prints the result, and the handler host never writes it into a log line, an exception message, a trace tag or a metric label. A filter behind which results are stored (idempotency) holds the redacted view, never the real value. - A filter cannot fingerprint secrets. Because it sees only the masked view, it cannot hash or compare the real values of sensitive parameters. Two different passwords look the same to a filter, so a key derived from the view must not be used to detect a duplicate call.
- Exception text is masked in every written form. Exceptions that the pipeline and the dispatcher log, and exception messages a handler throws, are masked for the plain value and for the forms a serializer or a URL writes it in: JSON-escaped (quote, backslash,
\uXXXXforms for plus, ampersand, angle brackets, apostrophe, control and non-ASCII characters, in upper and lower case hex, escaped slash; the System.Text.Json and Newtonsoft.Json encoders), the same once more for JSON inside a JSON string, and percent-encoded. A form that no serializer produces (a hash, a base64 copy, a truncation) is not recognized: never put a derived form of a secret into a message. - Mark a secret result. A result that carries a secret is marked
sensitive: truein the model (Sensitive results); the platform then masks it for the filters, logs and traces (see the next bullet), while the caller still gets the real value. In every case, do not write a secret into a log, audit event or outbox payload yourself: the platform cannot know what a handler writes.
Canary scan on the running services
The rules above are tested in-process in the platform libraries. The method canary scan checks them on the running services: with the local stack started, it calls methods through the typed method mutation of the asset repository with fake secret values (canaries) as a sensitive parameter, and then searches every place a value could end up for the canaries and for the caller's access token, in every form a serializer or URL writes them (plain, JSON-escaped, JSON in JSON, percent-encoded):
| Searched | What the scan reads |
|---|---|
| GraphQL answers | the body of every call |
| Service log files | what each logFiles/*.log of Start-Octo gained during the scan |
| Audit and event log | the events of the scan tenant written during the scan |
| Exported traces, metrics and logs | the output of a local OpenTelemetry collector with the file exporter |
| The handling service | log lines and spans of the handler host |
Two methods are scanned: a test method of the asset repository (MethodCanaryScan/CanaryAccount.Rotate-1, handled by the scan's own handler host over the bus: success, failing handler, timeout, validation errors, a declared error and refused calls) and System.Identity/User.CreateUser-1 (sensitive password) of the identity service (success, declared errors, validation, refused calls). Every call also checks that it got the expected outcome, so a clean result cannot come from calls that never reached the method path.
Rerun it from octo-asset-repo-services (set-up of the collector, the stack and the scratch tenant: tests/AssetRepositoryServices.MethodCanaryScan/README.md):
OCTO_METHOD_CANARY_SCAN=1 OCTO_CANARY_TENANT=<scratch tenant> OCTO_CANARY_OTLP_DIR=<collector output> \
dotnet test tests/AssetRepositoryServices.MethodCanaryScan -c DebugL
The scan writes one report per method (calls, sinks, hits; secrets and token parts are replaced by labels). It fails on any hit and when no telemetry arrived. Every hit becomes a bug. Result of the first run (2026-11-05): the method path itself had no hit in GraphQL answers, log files, the audit log or the handler host; the identity service writes every issued access token into its log at level Trace (a known bug, outside the method path). Second run (2026-11-05, services exporting to a local collector): no hit in the exported traces and metrics of either method; the services export no logs over OTLP, their log output is covered by the log files. For the exporter, start the services with OTLP_ENDPOINT_URL (without the OCTO_ prefix) and, for the identity method, with OCTO_METHODS__IDENTITY__ENABLEPASSWORDMETHODS=true; the README of the scan has the details.
Performance and replica sharing
All instances of a service that owns a method share one queue for it, and the broker delivers each call to exactly one of them (competing consumers). The dispatcher sends to a stable exchange and neither knows nor cares how many replicas exist. Tests against a real broker show that every caller gets one envelope, that every invocation runs exactly once, that load spreads over the replicas, and that a replica which is stopped gracefully in the middle of a burst loses no call. If the last replica stops, new calls fail fast with METHOD_HANDLER_NOT_AVAILABLE, and a new replica is served again as soon as it starts.
The overhead of the dispatcher itself is small: with a handler that does nothing, on one machine with broker, gateway and handlers on loopback, the 95th percentile of the round trip is about 1 ms, the same with one or three replicas. A cluster adds network hops to the broker and the handler's own time on top; plan the method timeout for the handler, not for the platform.
One rule for operators and authors of changes: never change the arguments of an existing method queue in place. If a newer build declares the queue differently than an older one, the broker refuses it, the new instance does not start, and the rollout waits on the old instances, which keep serving. A change of the queue arguments needs a new queue name so that both builds can run side by side.
Rolling update: stopping a replica
A replica that is stopped gracefully (rolling update, scale down) first withdraws itself, then lets its running calls finish and answer, and removes its method endpoints. A call that runs is never cut off. The shared queue and the other replicas keep serving, so callers normally notice nothing.
One narrow case remains. In the moment the stop begins, the broker may already have pushed one or two calls to the stopping replica. The replica does not start them, and the broker hands them back to another replica flagged as redelivered. That replica refuses a flagged delivery as a possible duplicate (it cannot know that the first holder never started it), and the caller receives METHOD_OUTCOME_UNKNOWN although the call did not run. Nothing runs twice.
| Traffic during the stop | Flagged calls that never ran |
|---|---|
| Steady (about 80 calls/s, handler time 200-600 ms, 30 stops, 9000 calls) | 1, i.e. about 0.03 per stop |
| Synchronized bursts that coincide with the stop (worst case, 10 stops, 3000 calls) | 29, i.e. about 3 per stop |
Measured on one machine with two replicas; the rate is the arrival rate of the stopping replica times the round trip of the broker's cancel, so a busier service sees proportionally more. What this means for you:
- Treat
METHOD_OUTCOME_UNKNOWNas always: read the state before you call again (R5). After a rolling update this is the expected, rare cause for a method that did not change anything. - A call with an idempotency key uses up its key on this answer (R8); send the next attempt with a new key after you have read the state.
- The platform accepts this residual rate: the alternative would run a call twice after a lost connection. Removing it needs a change of the message transport (stop fetching before the consumer is marked as stopping) or a handler-side claim for calls with a key; neither is part of the current platform.
- Hard loss of a replica (killed process, lost connection) gives more flagged calls than a graceful stop, and in that case the first execution may have run; see limits.
Timeouts and sizes
execution.timeoutSeconds(1–300, default 15) is the time the caller waits. The deadline in the request is now plus this timeout, and the cancellation token of the handler is cancelled at that deadline. The dispatcher waits a short grace period (default 2 seconds) longer, so that theMETHOD_TIMEOUTof your host arrives before the dispatcher answers itself. See deadlines and timeouts.- Choose the timeout for the slowest normal case, not the average. Work that can take longer than the maximum needs an asynchronous method that returns an operation handle (see Asynchronous handlers); for such a method the timeout bounds only the accept step.
- Keep results small. They travel in one message.
Errors a handler returns
| You want to say | Return |
|---|---|
| The caller may not do this (or you cannot tell if they may) | METHOD_FORBIDDEN |
| A parameter is missing or wrong | METHOD_VALIDATION_FAILED, parameter field |
| The target is gone | METHOD_TARGET_NOT_FOUND |
| A business rule refused | a code from the method's errors, with string parameters |
| Something broke that you did not expect | throw; the host returns METHOD_HANDLER_FAILED with a correlation id |
| You cannot know whether your write happened | METHOD_OUTCOME_UNKNOWN |
| You received the request after its deadline | METHOD_TIMEOUT without executing (the host does this) |
The platform codes and what a caller does with each are in the error code table. Never declare a code that starts with METHOD_.
Checklist for a new method
- Is it one method, or did I split an operation that must succeed or fail as a whole?
- Parameters that are secrets are
sensitive: true; none of them reaches a log, audit event, trace, outbox payload or error parameter. -
authorizationis declared (default is administrators only; the host enforces it before the handler runs), the service has enabled the caller token validation, and the handler does not use thecallerdata. - Order of checks: authenticate, authorize, look up, validate, write; the same answer for a missing and a forbidden target.
- Declared error codes cover the business refusals; the model has no
METHOD_code. - The handler uses nothing bound to the request; the tenant comes from the request's tenant id.
- Writes are atomic and committed before the answer; unknown commit state returns
METHOD_OUTCOME_UNKNOWN. - The cancellation token is checked before a write.
- Concurrent calls on the same target are safe.
-
idempotentis set honestly. A non-idempotent method that must not run twice is called with an idempotency key (R8);idempotentdoes not switch the key off. - A method with a
sensitiveresult is never called with a key (it is refused,result-sensitive); callers read the state instead. - Side effects after the commit; an outbox where a lost message is not acceptable.
-
timeoutSecondsfits the slowest normal case; long work is an asynchronous method (execution.mode: async, service handler) that validates synchronously and reports progress. - Tests: each error code, forged caller, log scan for secrets, crash-before-answer reasoning, registration check.
Diagnostics: which handlers are registered
Which service answers a method, how many instances are alive, and when each was last seen? The asset repository answers this with the GraphQL root query methodHandlers (written MethodHandlers in the schema description; GraphQL field names are camelCase).
Who may read it. System administrators only. The registry is platform-wide: it names services, instance ids, service versions and the tenant ids a handler serves, which is platform topology across all tenants. The query uses the same caller check as the cross-tenant system endpoints (orphan sweep, tenant registry reconcile): the token must be issued for the system tenant; a user token must also hold the role AdminPanelManagement; a client-credentials token of the system tenant passes without the role. Any other caller, including a tenant administrator of a customer tenant, gets methodHandlers: null and the error code MethodHandlersForbidden; no handler data is read or returned. There is no reduced view for tenant administrators.
{
methodHandlers {
answeredBy {
dispatcherInstanceId serviceVersion snapshotEnabled
snapshotLoadedAtUtc snapshotRowsRestored snapshotLoadFailed lastFlushUtc lastFlushFailed
}
methods(search: "Identity", states: [NO_LIVE_HANDLER, NOT_ANNOUNCED]) {
methodId transport state liveInstanceCount lastSeenUtc
instances {
serviceName instanceId version contractVersion tenants
lastSeenUtc expiresUtc state source persistedLastSeenUtc
}
}
}
}
Methods. The list is the union of three sources: the handler rows the answering gateway replica holds, the methods the gateway serves in-process, and the methods of the system tenant that declare a service handler (execution.handler: service, the default) and that no instance has announced.
state of a method | Meaning |
|---|---|
LIVE | At least one instance is live, or the method is served in-process (transport: IN_PROCESS). |
NO_LIVE_HANDLER | Instances are known, but all are expired, withdrawn or have no queue bound. A call ends as METHOD_HANDLER_NOT_AVAILABLE. |
NOT_ANNOUNCED | The model declares a service handler, but no instance ever announced the method: the handler is missing (service not deployed, not started, or the method is not registered). |
transport is BUS for methods handled by another service and IN_PROCESS for methods the gateway handles itself. liveInstanceCount counts live bus instances; an in-process method has no instances. lastSeenUtc of a method is the latest lastSeenUtc of its instances. The arguments search (case-insensitive part of the method id or of a service name) and states narrow the list.
Instances. Every known instance is listed, expired and withdrawn ones too (rows are kept for 24 hours).
| Field | Meaning |
|---|---|
state | LIVE, EXPIRED (no heartbeat within the announced TTL), WITHDRAWN (graceful stop) or NO_QUEUE_BOUND (the broker returned a mandatory send). |
source | HEARTBEAT when the row came from an announcement, a discovery answer or a withdrawal. SNAPSHOT when it was restored from the persisted snapshot after a gateway restart and no heartbeat has confirmed it yet. A SNAPSHOT row is never LIVE on its own authority: it is a last-seen record. |
lastSeenUtc / expiresUtc | Last announcement or withdrawal received, and the end of the announced TTL. |
persistedLastSeenUtc | The last-seen time of the row in the snapshot store that all gateway replicas share. null if the store has no row for the instance, or the store could not be read (the query still answers and the failure is logged). |
tenants | The tenants the instance serves (* = all). |
One replica answers. Each gateway replica holds its registry in memory, so the answer describes the replica that served the request. answeredBy names it: dispatcherInstanceId, serviceVersion, whether the snapshot is enabled, the last snapshot load (snapshotLoadedAtUtc, snapshotRowsRestored, snapshotLoadFailed) and the last successful write (lastFlushUtc, lastFlushFailed). Compare lastSeenUtc with persistedLastSeenUtc to see what other replicas wrote. Ask again to reach another replica.
The response holds service data only: ids, service names, versions, times, tenant ids and flags. It never contains method parameters, tokens or user data, and a failing snapshot store is reported as a flag, never as exception text.
In Refinery Studio
System administrators see the same data without writing a query: in the system tenant, Operate › Method handlers (route /<system tenant>/diagnostics/method-handlers). The page needs the role AdminPanelManagement; in any other tenant, or without the role, the menu entry is hidden and a direct link shows a short notice instead of data.
The header reads "Answered by replica X (version Y)" with the snapshot facts of answeredBy (loaded at, rows restored, last flush, and a warning when a load or flush failed; "Snapshot off" when the snapshot is disabled). The list shows every method with its transport, state (Not announced is highlighted with an icon and text), live instance count and last seen time; search and state filters map to the search and states arguments. Expanding a method lists its instances with all fields above; rows restored from the snapshot are marked "Last seen, not confirmed live". The page is read-only, refreshes on demand, and has an optional auto refresh (every 15 seconds, off by default). Like the query, it shows what the replica that answers knows: refresh again to reach another replica.