Skip to main content

Method Consistency Model

This page states what a method may assume about transactions, failures, ordering and repeated calls, and what the caller can rely on. It is the contract between the platform, the author of a method handler and the caller of a method. The practical how-to is in Implementing Method Handlers.

Status

The wire format (request, response envelope, error codes) is fixed. The handler SDK and the dispatcher are released; idempotency keys are part of the platform (see R8); asynchronous methods with operation handles are released (see OD-5). Statements marked Planned describe the intended behavior of a part that is not released yet. Points that are not decided are marked Open decision and are collected at the end of this page, together with the decisions taken so far.

The one decision behind everything​

A method runs outside the Mongo transaction of the request that called it.

A GraphQL request that changes data (for example createDevice) runs in one MongoDB transaction: either all of its generic mutations commit or none. A method call does not join that transaction. It is delivered to a handler, usually in another service, over the event hub, and the handler commits (or fails) on its own. A handler in the gateway process (an in-process handler) is no exception: it also gets its own unit of work.

This is deliberate. A transaction cannot span services, and a handler must not hold a request open for the duration of a long operation. The price is that the platform does not make a method call and the writes around it atomic. This page describes what holds instead.

The rules​

#RuleConsequence
R1No shared transaction. A handler never joins the request's transaction. The method context contains no session and no transaction.A handler cannot see uncommitted writes of the calling request. It cannot be rolled back by the request.
R2Every call is its own unit of work.A failing method does not roll back earlier writes of the request. A failing earlier write does not stop an independent method call. There is no atomicity across calls.
R3The handler owns the atomicity of its own writes and answers success only after its writes are acknowledged by the database.One call is all-or-nothing as far as the handler makes it so (single-document update, or its own session for several documents).
R4No automatic retry. Duplicates are detected where possible, METHOD_OUTCOME_UNKNOWN otherwise. The platform never sends a call again and never runs a delivery it knows to be a possible duplicate; retries are the caller's decision.The same logical operation can run twice only if the caller sends it twice (new invocation) — unless an idempotency key is used (R8). What the platform does not detect is listed under limits.
R5Exactly one envelope per call, with two uncertain outcomes. The caller receives one CkMethodResponse. The outcome is certain except for METHOD_TIMEOUT and METHOD_OUTCOME_UNKNOWN.After those two codes the caller must read the state before it acts again.
R6After a success envelope, reads see the handler's committed state.A caller that waits for the envelope and then reads sees the effect.
R7No ordering between calls. Calls to the same method may run concurrently, on different handler instances, in any order.The handler must be safe under concurrency (optimistic concurrency or atomic updates).
R8Idempotency key. A repeated key within its lifetime (24 hours by default) returns the first result without executing again. A key is burned by every outcome that may have run the method. A method declared execution.idempotent ignores the key.Safe retries for methods that must not run twice. See R8.
R9Authorization is checked twice — declaratively at the gateway and again, from the forwarded token, in the handler.The handler cannot rely on the gateway having checked, and must not trust the informational caller data. How the token is validated: OD-10; which data permissions apply: OD-12.

R1 — What "outside the transaction" means for the handler​

  • The method context carries the tenant, the caller, the target entity id, the cancellation token and the deadline. It has no database session. A unit test of the SDK guards this.
  • A handler loads the target itself. Between the gateway's check and the handler's read, the entity may have been changed or deleted. The handler authorizes and validates against what it reads, and answers METHOD_TARGET_NOT_FOUND when the entity is gone.
  • A handler runs in its own dependency-injection scope. It must not use services that are bound to the calling request (the HTTP context, the request's database session or tenant repository). For a remote handler this is automatic; for an in-process handler it is a rule the handler author must keep. See Implementing Method Handlers.
  • Consequence for callers in one GraphQL request: a method that targets an entity created or changed earlier in the same request does not see it, because the request has not committed. Do not combine "create" and "call a method on the new entity" in one request; the gateway rejects such a request before anything runs (decision OD-1 and OD-2).

R2 — No atomicity across calls​

Everything the platform guarantees is per call. For a request that contains several mutations:

SituationResult
Generic mutation fails after a method succeededThe request transaction rolls back its generic writes; the method's effect stays.
Method fails after generic mutations were appliedThe method failure is reported as data in the result; the generic writes are not rolled back by it.
Two methods in one request, the second failsThe first method's effect stays.

Method failures are always reported as data (an error code in the result), never as a GraphQL error. A GraphQL error would abort the request transaction and discard sibling generic mutations, which is not wanted for a failure that is only about the method. A request carries either generic mutations or exactly one method call: the gateway rejects a mix and a second method call before it executes anything (decision OD-1 and OD-2). The rows above therefore describe what a caller can still observe when requests are sent one after the other, and they are the reason for the rule.

If a business operation needs several steps to succeed or fail together, it is one method with one handler that does the steps in its own transaction, not several method calls. If the steps live in different services, use compensation.

R3 — The handler's own atomicity​

  • One document: a single update is atomic.
  • Several documents: the handler opens its own session and transaction, and commits before it answers.
  • Never answer success and write afterwards. The success envelope means "committed".
  • If the handler cannot know whether its write was applied (for example the database connection dropped during commit), it answers METHOD_OUTCOME_UNKNOWN, not METHOD_HANDLER_FAILED, so that the caller checks the state instead of assuming nothing happened.
  • The handler host does this for the database case by itself: when a write is not acknowledged in time, the exception is answered METHOD_OUTCOME_UNKNOWN with a correlationId, also when the repository wraps it. That covers a write-concern error (the repositories write with w:majority and a wtimeout of 30 seconds; when the majority does not confirm in time the write may still be on the primary) and a commit whose result the database driver reports as unknown. The audit event and the ck.method.calls metric carry the same outcome. A write the database refused (for example a duplicate key) stays METHOD_HANDLER_FAILED. Do not catch these exceptions in the handler. A pipeline handler follows the same rule: the adapter classifies the failed write with the same classifier and the controller answers METHOD_OUTCOME_UNKNOWN.
  • The deadline is a request to stop: the handler checks the cancellation token before it starts a write and does not start one after the deadline. A write that already started may still complete.

R4 and R5 — Delivery, retry and outcome​

Delivery. A method call is one message to a queue that is shared by all handler instances of that method (competing consumers): each message reaches exactly one instance. The message is mandatory: if no queue is bound, the broker returns it and the caller receives METHOD_HANDLER_NOT_AVAILABLE immediately. The message has a time to live equal to the call timeout and is not persisted by the broker, so a call that nobody picked up does not run later.

No retry. The event hub retries failed message handling for many other endpoints. Method endpoints disable this, because a retried non-idempotent method would run again. A handler never throws to the bus: every exception becomes an envelope (METHOD_HANDLER_FAILED with a correlation id).

What this guarantees. The platform sends a call once and never sends it again on its own (no retry on the caller side, none on the handler side). A delivery that the broker flags as a possible duplicate, and an invocation id that the handler instance has already seen, are not executed; the caller receives METHOD_OUTCOME_UNKNOWN. This is not a guarantee of "at most once": a duplicate that carries a new invocation id, or one that lands on another handler instance without a broker flag, is not recognized (see limits). At-least-once behavior is built by the caller (retry) together with an idempotency key (R8), which together give effectively once within the key's lifetime.

GuaranteeWithout idempotency keyWith idempotency key
One invocation executesonce, or not at all when the duplicate is detected (see limits)same
One logical operation executesat most once per call the caller makes; a caller retry runs it againat most once within the key lifetime
Caller learns the resultexactly one envelope, or METHOD_TIMEOUT / METHOD_OUTCOME_UNKNOWNsame; a repeat returns the stored first result, or METHOD_OUTCOME_UNKNOWN when the first outcome was unknown

A broker can in rare cases deliver a message twice (for example when a handler instance dies after reading and before acknowledging). The envelope code METHOD_OUTCOME_UNKNOWN ("possible duplicate delivery, not executed again") exists for this. Which duplicates the platform recognizes and which it does not is decided in OD-3 and listed under limits.

What the platform does with a delivery. The handler host checks, in this order, and executes only a delivery that passes both:

  1. The broker flags the delivery as redelivered, or the bus is retrying it: not executed, METHOD_OUTCOME_UNKNOWN. The log line names the invocation id, the method, the tenant and the delivery info (redelivered flag, retry attempt, redelivery count, message id), never the parameters. A typical case is a handler instance that loses its broker connection before it acknowledges the message: the broker hands the message to another instance, and that instance refuses it. The first execution may have completed.
  2. The instance has already seen this invocation id: not executed, METHOD_OUTCOME_UNKNOWN. The instance remembers an id until the call deadline plus 5 minutes, at most 1 hour after the delivery.

A handler that throws after a side effect still ran exactly once: the caller receives METHOD_HANDLER_FAILED, and the bus does not run the method again.

The dispatcher sends a call once. It has no publisher-side retry or resend: a transport fault ends the call as METHOD_TIMEOUT (no answer) or METHOD_HANDLER_NOT_AVAILABLE (the broker returned the message), never as a second delivery. If the only handler instance loses its connection while it works, the message is lost with the queue and the caller receives METHOD_TIMEOUT.

Limits: what is not detected​

The platform makes no "at most once" guarantee. The table lists every kind of duplicate with the answer to "does the platform recognize it?" and the decision that owns it (OD-3 for deliveries of one invocation id, OD-4 for a re-sent operation). Only a duplicate that is recognized is not executed; the others reach the handler and run.

DuplicateRecognizedHow or why notOwner
The broker redelivers a message after a lost acknowledgement (handler instance died, connection dropped, acknowledgement later than the broker's consumer timeout), to the same or another instance, also after a restart.YesThe delivery carries the broker's redelivery flag; the handler host refuses it.OD-3
The same message reaches the same handler instance twice without a flag.YesThe instance remembers the invocation id (until the call deadline plus 5 minutes, at most 1 hour).OD-3
The same invocation id reaches another handler instance without a flag.NoThe memory of seen ids is per instance and there is no shared store. No path in the platform produces this case: the dispatcher never sends a call again and a broker requeue carries the flag. It is possible only through something outside the platform, such as a component that duplicates messages or a foreign publisher on the bus.OD-3
A handler instance restarted after it ran a call and receives the same invocation id again without a flag.NoThe memory of seen ids does not survive a restart. Same remark as above.OD-3
The caller sends the same logical operation again (retry after a timeout, double click).NoThe gateway creates a new invocation id for every call, and no API lets a client choose it. A re-sent call is therefore a new invocation, not a duplicate delivery. Only an idempotency key covers it.OD-4

Late acknowledgement (broker consumer timeout). A handler that holds a delivery longer than the broker's consumer timeout (RabbitMQ consumer_timeout, 30 minutes by default) loses its channel: the broker closes it and requeues the unacknowledged message flagged as redelivered. This is the first row of the table: the instance that receives the requeued message refuses it and the caller receives METHOD_OUTCOME_UNKNOWN, while the first execution is still running and commits its effect (the caller is told "unknown" for a call that ran once). When the first execution ends, its late answer finds the call already answered and changes nothing; nothing runs twice. The instance that lost its channel took no further calls in the test (CkMethodDispatcherBusTests); the other instances serve them, so restart such a replica. Keep the broker's consumer timeout above the longest method timeout (300 seconds): a handler is answered at its deadline, so the case needs a broker timeout below that or a handler that ignores cancellation.

A method that must never run twice. Do not rely on the recognition above: it is incomplete by design and gives no "at most once". Such a method is called with an idempotency key (R8) and writes idempotently (a unique id or a conditional update instead of a blind insert or increment), so that a duplicate that slips through changes nothing. The key protects against the realistic duplicate, the re-sent operation. The decision does not add a handler-side claim for keyed calls; when that becomes necessary is described under OD-3. How many refused duplicates occur, and why, is visible in the counter ck.method.duplicate_refused (Method metrics).

After METHOD_TIMEOUT, METHOD_OUTCOME_UNKNOWN and METHOD_HANDLER_FAILED the caller reads the state before it calls again (R5).

How the caller learns the outcome.

EnvelopeMeaningCaller action
success: true, resultJsonThe method ran and committed.Use the result.
success: true, operationAn asynchronous method was accepted. Nothing is said about completion.Poll the operation handle (retryAfterSeconds tells when). The rules of this page apply to the work behind the handle.
success: false, declared error codeThe method ran and refused for a business reason. Nothing was changed, unless the method documents otherwise.Fix the input or inform the user.
success: false, platform code that says "method did not run" (METHOD_HANDLER_NOT_AVAILABLE, METHOD_FORBIDDEN, METHOD_VALIDATION_FAILED, METHOD_TARGET_NOT_FOUND, METHOD_CONTRACT_VERSION_UNSUPPORTED, METHOD_REQUEST_IN_PROGRESS)Nothing was changed.Retry later (not available, version, request in progress) or fix the cause.
METHOD_HANDLER_FAILEDThe handler failed unexpectedly. Whether partial effects remain depends on the handler: a handler that follows R3 leaves none.Report the correlation id. Check the state before retrying a non-idempotent method.
METHOD_TIMEOUTNo answer in time. The handler may still run or may have finished.Read the state. Do not retry a non-idempotent method blindly.
METHOD_OUTCOME_UNKNOWNPossible duplicate delivery or unknown commit state.Read the state.

The full code table with parameters is in the wire contract; all codes are language-neutral: the wire carries a stable code and parameters, never message text.

Deadlines and timeouts​

  • Where the timeout comes from. execution.timeoutSeconds of the method definition (1–300 seconds, default 15 when the key is omitted; the compiler rejects other values). The caller's deadline is now plus that timeout. A gateway reads the timeout from the Construction Kit definition and sets the absolute deadline in the request; the request also carries the timeout in seconds, which is reported back in the timeoutSeconds parameter of METHOD_TIMEOUT.
  • At the deadline. The cancellation token of the handler is cancelled at the deadline, and the handler host answers METHOD_TIMEOUT. The dispatcher waits a short grace period longer than the deadline (default 2 seconds), so that the answer of the handler arrives first; if it does not (the handler instance is gone, or the message was lost), the dispatcher answers METHOD_TIMEOUT itself. A caller therefore has its answer within deadline + grace.
  • A handler that ignores its token. An in-process handler that does not react to the token is not waited for: the caller gets METHOD_TIMEOUT at the deadline, the handler may still finish and its late result is dropped. A remote handler that does not react to the token is answered by the dispatcher after the grace period.
  • A request that arrives after its deadline is not executed; the handler host answers METHOD_TIMEOUT.
  • Absurd values never break a call. A deadline further away than the maximum timeout of the dispatcher (default 10 minutes) is cut to it, before any filter or transport sees the request, so the handler sees the cut deadline as well. A timeout of any size is clamped, a deadline in the far past is simply expired, and a handler host never throws for a deadline sent by another sender.
  • In-process and remote handlers behave the same: same envelope, same clamping, same timeout answer.

R6 and R7 — Visibility and concurrency​

  • Read-your-method. A caller that received success and then reads through the platform sees the committed effect. A caller that did not receive an answer (timeout) may or may not see it yet.
  • No ordering. Two calls that a client sends one after the other without waiting for the first answer may run in either order, on different instances, even at the same time. A client that needs an order waits for the answer of the first call.
  • Concurrent calls on one target (two admins reset the same password, two imports start) are the handler's problem: use an atomic update with a condition, a version check, or a lock document with a lifetime. Return a declared error (for example OPERATION_ALREADY_RUNNING) for the loser.
  • Several replicas of a service share the traffic of a method. The handler must not keep state about earlier invocations in instance fields; a later call can land on another instance.

R8 — Idempotency keys​

A request carries an optional idempotencyKey (in GraphQL the argument idempotencyKey of the method mutation, see Idempotency key). A call with a key runs at most once per tenant, method, caller and key within the lifetime of the key; a repeat returns the first result without running the method again. A call without a key is not affected: nothing is stored and the idempotency step costs nothing. The rules below are the decision OD-4.

When to send a key. With every call of a method that must not run twice (a second run would send the mail again, book again, create a second record). A method with execution.idempotent: true is safe to repeat without a key, and the platform treats it that way: a key sent to it is accepted and ignored. The call does not touch the idempotency store and runs as a call without a key, every time. That means: no format or length check of the key, no refusal for a sensitive result or for a missing fingerprint key, no parameters-changed for the same key with other parameters, no METHOD_REQUEST_IN_PROGRESS for two concurrent calls, and no replay (two calls with the same key both run). Audit and metrics see the call like any other: exactly one audit event and one count in ck.method.calls, never marked as a replay. A method that does not declare the flag (or an unknown method) keeps the full behavior described below.

How a caller uses it:

  • Generate one key per logical operation (for example when the user presses the button), not per attempt. Use the same key on every retry of that operation. A key has 1 to 128 characters.
  • A key is only reusable after a call that did not run. After METHOD_HANDLER_NOT_AVAILABLE (and METHOD_CONTRACT_VERSION_UNSUPPORTED) the method did not run and the same key runs it. After METHOD_REQUEST_IN_PROGRESS wait and send the same key again. After a transport error without an answer send the same key again: the call either runs once or the first result is returned.
  • After METHOD_TIMEOUT and METHOD_OUTCOME_UNKNOWN the key is burned: the method may have run, so the platform never runs it again under that key. Read the state of the target, then use a new key for a new attempt. The same holds after METHOD_HANDLER_FAILED.
  • Never retry a business error unchanged, with or without a key.

What a repeated key gets, by the outcome of the first call:

First callThe same key again
still running (until the request deadline plus the response grace of the dispatcher, default 2 s)METHOD_REQUEST_IN_PROGRESS; the method is not run. Send the same key later.
succeeded, was accepted (operation handle) or ended with an error the method declaresthe stored first result, with the invocation id of the repeat; the audit event of the repeat names the first invocation as replayOf.
METHOD_HANDLER_FAILEDthe stored failure; the method is not run again. The key is burned: read the state and use a new key.
METHOD_TIMEOUT, METHOD_OUTCOME_UNKNOWN, cancelled by the caller, or the claim was never settled (for example the gateway stopped)METHOD_OUTCOME_UNKNOWN; the key is burned.
not executed (METHOD_HANDLER_NOT_AVAILABLE, METHOD_CONTRACT_VERSION_UNSUPPORTED)the key is released: the repeat runs the method.

A call that the gateway refuses before it reaches the dispatcher (METHOD_FORBIDDEN, METHOD_TARGET_NOT_FOUND, METHOD_PARAMETERS_TOO_LARGE, an invalid parameter) never takes a key and can be sent again with the same key once the cause is fixed.

Scope. A key is the combination of tenant, method, caller and key. A user is identified by the sub of the validated token; a service token (client credentials, no sub: a pipeline, a bot, the CLI with client credentials) by client:<client_id>. A user and a client therefore never share a key, even if the user's sub and the client id are the same string; a token whose sub itself starts with client: has no scope and is refused with caller-unknown. The caller never comes from the informational caller data of a request. Two callers can use the same key independently. The audit names a service caller by its plain client id. The target entity is not part of the scope but part of the parameter fingerprint: the same key on another entity counts as other parameters. Authorization and the data-permission check run before the key is looked at, so a key never widens a permission and an unauthorized caller learns nothing about stored keys. The raw key is not stored, only a SHA-256 hash of the four parts.

Parameter fingerprint. The platform compares a fingerprint of the method's type, target and canonical parameters, computed from the real values before sensitive parameters are masked, so two different new passwords under one key are told apart. It is a keyed hash (HMAC-SHA-256); the key id is stored with it. Without the secret a reader of the database or of a backup cannot derive a low-entropy value such as a password from it. A method without sensitive parameters uses a plain SHA-256 when no fingerprint key is configured. The fingerprint is never logged, audited or returned.

Refusals. All of these are METHOD_VALIDATION_FAILED with field = idempotencyKey; the method does not run and nothing is stored.

reasonCauseWhat to do
parameters-changedThe key was used before with other parameters, another target or another type, or its stored fingerprint cannot be verified (its key id is no longer configured).Use a new key for the new request.
result-sensitiveThe result of the method is marked sensitive (or the platform does not know the method). Such a method refuses a key: replaying the first result would mean storing a secret in the tenant database.Call without a key; read the state to find out whether the first call ran.
caller-unknownThe validated token has neither a sub nor a client_id, so the key has no scope.Authenticate the call.
key-too-longThe key has more than 128 characters.Shorten the key.
fingerprint-key-missingA key was sent to a method with sensitive parameters, but the service has no fingerprint key (an operator task, see Operating idempotency keys).Ask the operator; call without a key meanwhile.
fingerprint-unavailableThe service has no fingerprint facility for keyed calls (the idempotency options are not registered in the host).Ask the operator.

When the idempotency store cannot decide (database unreachable, no credentials, unknown tenant), the call is not executed: METHOD_HANDLER_NOT_AVAILABLE with reason = idempotency-store-unavailable. The method did not run; retry later with the same key. A call without a key never touches the store.

Operating idempotency keys​

For operators of the service that hosts the method gateway (the asset repository). Configuration section Methods:Idempotency (appsettings or environment variables with the prefix OCTO_ and __ as separator, for example OCTO_Methods__Idempotency__Ttl):

KeyDefaultMeaning
Methods:Idempotency:Ttl24 hoursLifetime of a key, counted from its first call. A .NET time span (24:00:00). Maximum 7 days (7.00:00:00): a larger value is reduced to it and a warning is logged. Zero or negative fails the start of the service. A changed value applies to new keys only.
Methods:Idempotency:Fingerprint:Current:IdnonePublic name of the current fingerprint key: 1 to 64 characters of A-Z a-z 0-9 . _ -. Stored with every fingerprint.
Methods:Idempotency:Fingerprint:Current:SecretnoneThe key material: base64 of at least 32 random bytes (for example from openssl rand -base64 32).
Methods:Idempotency:Fingerprint:Previous:Id / :SecretnoneThe key of the rotation before; only used to verify fingerprints that are already stored.
  • The secret is a secret of the environment. Platform Ops provides it per environment (Vault and a secret Helm value); it never goes into a checked-in file, a ticket or a log. Every replica of the service must hold the same keys, otherwise a repeat that lands on another replica looks like changed parameters. Keys are read when the service starts; a rotation needs a rollout.
  • Without a configured Current key keyed calls to methods with sensitive parameters are refused (fingerprint-key-missing); keyed calls to other methods work with a plain SHA-256. Configure the key before keyed calls are enabled on a shared environment.
  • Rotation. (1) Move the values of Current to Previous. (2) Set a new Current with a new Id and a new secret. (3) Roll out to all replicas. (4) Remove Previous after at least the configured Ttl has passed; a key that is still in its lifetime and was fingerprinted under a removed key is answered as parameters-changed. The ids must differ, and Previous without Current is refused. Malformed or half-configured keys fail the start of the service; the message names the setting, never the value.
  • The store is the tenant database. The collection MethodIdempotency of each tenant holds one document per key (key hash, state, method id, first invocation id, fingerprint with key id, the times and the first response). It is part of the tenant backup and restore, and it is removed with the tenant. Every replica shares it. Expired documents are removed by a MongoDB TTL index; readers check the expiry themselves, so the removal may lag by about a minute without effect.
  • A tenant copied under another tenant id carries rows that no call can reach any more, because the tenant id is part of the key hash. They expire by themselves; nothing needs to be cleaned up.
  • Fail closed. While the tenant database is unavailable, keyed calls are answered METHOD_HANDLER_NOT_AVAILABLE (idempotency-store-unavailable); calls without a key are not affected.
  • Mixed rollout. A replica of the gateway without the idempotencyKey argument rejects a query that carries it at schema validation; the key is never ignored silently.

Audit: one event per invocation​

An auditor can answer "who called which method on which entity with what outcome" from the event log. The platform writes exactly one audit event per invocation id, for every transport (bus and in-process) and every outcome:

  • success, a declared business error, METHOD_HANDLER_FAILED, METHOD_TIMEOUT, METHOD_HANDLER_NOT_AVAILABLE and METHOD_OUTCOME_UNKNOWN;
  • a call refused by authorization (METHOD_FORBIDDEN), although the handler never ran, and a call the gateway refuses before it reaches the dispatcher (forbidden, unknown or invisible target, oversized or invalid parameters);
  • a replay: a call answered from the idempotency store is its own invocation with its own invocation id and its own event; the event names the first invocation id as replayOf, never the idempotency key. METHOD_REQUEST_IN_PROGRESS and the refusal for a key that was reused with other parameters are recorded with their own code and are not replays;
  • an exception in a pipeline filter: one event with METHOD_HANDLER_FAILED, and the call is answered as before;
  • a fault of the gateway itself, before the call reaches the pipeline (an exception in the check of the target, in the parameter validation or in a dispatcher that throws): one event with the transport gateway and METHOD_HANDLER_FAILED, and the caller gets the same error as without the audit;
  • a fault of the dispatcher before the call reaches the pipeline (for example the method cannot be resolved): the caller gets the usual METHOD_HANDLER_FAILED answer and one event with the same outcome;
  • a cancellation by the caller: one event with the outcome CALLER_CANCELLED, also when the caller gives up before the call has reached the pipeline. The method may or may not have run; check the state of the target before calling again.

The event has the category Method.Invocation and carries the invocation id, tenant, method id, target, the caller (subject and name), the transport (in-process, bus, or gateway for a refusal before the dispatcher), the handler instance, the outcome (SUCCESS, ACCEPTED for a long-running operation, or the error code), the duration and replayOf.

Refusals by the gateway. The gateway answers some calls before they reach the invocation pipeline, so the audit filter never sees them. The gateway records each of these through the same recorder, with the transport gateway, the invocation id it returned to the caller, no handler instance and the error code as the outcome:

RefusalOutcome
The caller may not call the method (roles, scopes, allowSelf, unknown method, not authenticated)METHOD_FORBIDDEN
The parameters exceed Methods:MaxParametersBytesMETHOD_PARAMETERS_TOO_LARGE
The target does not exist, or the caller may not see itMETHOD_TARGET_NOT_FOUND
The caller sees the target but the data permission forbids the callMETHOD_FORBIDDEN
The parameters do not match the declarationMETHOD_VALIDATION_FAILED

Faults of the gateway. A fault is no refusal: an exception in the gateway's own steps (for example the store is down while the gateway loads the target) or in a dispatcher that throws. The gateway records it once, with the transport gateway, the invocation id it minted, no handler instance and the outcome METHOD_HANDLER_FAILED, and then rethrows: the caller receives exactly the error it received before. The platform does not use a separate outcome for this: METHOD_HANDLER_FAILED is the code that the audit filter already writes for an exception inside the pipeline, so one query finds every unexpected platform failure; metrics and the audit level (Error) know exactly this code, whereas an invented code would be counted as invalid_code. Read the combination: transport gateway and no handler instance mean that the failure happened before a handler was reached; the details are in the service log under the invocation id. Whether the method ran cannot be told from the event alone for a dispatcher fault; check the state of the target before calling again. A call that reached the pipeline is never recorded by the gateway: the audit filter records its failure, so there is no second event.

Cancellation and dispatcher faults before the pipeline. Whoever observes the end of a call records it, and only that one, so there is one event and one count in ck.method.calls per invocation id. If the caller gives up while the gateway still checks the target, no filter and no dispatcher has seen the call: the gateway records one event with the transport gateway and CALLER_CANCELLED, and the cancellation reaches the caller as before. Once the call was handed to the dispatcher, the audit filter (or, for a cancellation before the pipeline, the dispatcher) records it and the gateway does not. If the dispatcher itself fails before the pipeline runs, it answers METHOD_HANDLER_FAILED as a normal answer and records one event with the transport of the call (bus, or in-process once the in-process transport was chosen) and no handler instance; the gateway sees an answer, not an exception, and records nothing more. A failing audit write never changes the answer of the call.

A call that passes the gateway is recorded by the audit filter only, so there is still exactly one event per invocation id. Refusals around the idempotency key (a key that is too long, reused with other parameters, still running) happen inside the pipeline and are recorded by the filter with the transport of the call. Like every audit event, a gateway event carries no parameters, no token and no idempotency key. If the audit event of a refused call cannot be written, the caller still receives the refusal; the failure is logged and counted (ck.method.audit.failures).

What is never recorded: no parameters (not even masked ones), no result, no error parameters, no access token or other credential, no idempotency key, no parameter fingerprint, no exception text. This holds whether or not the parameters are marked sensitive.

When the audit cannot be written. By default a failing audit write never fails the method call: the platform logs the failure and counts it (ck.method.audit.failures). Each write is bounded by a timeout (Methods:Audit:WriteTimeout, default 5 seconds), so a stalled event store cannot keep a call waiting; a write that runs into the timeout counts as failed. For regulated tenants an operator can switch this around (Methods:Audit:FailCallOnAuditFailure for all tenants, Methods:Audit:FailCallOnAuditFailureTenants for named tenants): the call is then failed when its audit event could not be written. Because the event is written after the call, the method may already have run, so the answer is METHOD_OUTCOME_UNKNOWN (check the state of the target before calling again). A call that was not executed, such as a refusal, keeps its own refusal code.

Side effects​

A handler often writes to the database and does something else: publish a message, send an e-mail, call another service. Because there is no shared transaction and no platform retry, a handler must reason about the order explicitly.

Write first, announce after commit. Never publish an event or send a mail before the database commit succeeded: a rollback would leave a message about something that did not happen.

The gap after commit. If the process dies between the commit and the publish, the announcement is lost. If the handler publishes first and then dies before commit, the announcement is false. Choose by the cost of each failure and, where it matters, close the gap with an outbox:

PatternUse whenHow
Commit, then best-effort publishLosing the message is acceptable (cache hint, notification that can be recomputed).Commit; publish; on publish failure log a warning.
OutboxThe message must be sent if (and only if) the write committed.Built by the service itself; the platform has no outbox component (OD-6). In the same transaction as the business write, insert an outbox document { id: invocationId, kind, payload, state: pending }. A separate relay (a background service of the same service) reads pending documents, publishes them with the outbox id as the message id, and marks them sent. The relay may send twice, so the receiver deduplicates by id (it is idempotent).
CompensationSteps span several services and cannot share a transaction.Each step records its state before and after. When step n fails, the handler runs the inverse of steps n−1…1 (each inverse must itself be safe to repeat) and reports the failure with a declared error code and parameters that name the failed step. If a compensation fails, record it for an operator and answer METHOD_HANDLER_FAILED with the correlation id.
Idempotent receiverWhatever the sender does, the receiver must cope with a duplicate.Give every message or external call a stable id derived from the invocation id and deduplicate on it.

The platform provides no outbox component (OD-6, decided: guidance only). Before reaching for one, check whether ordering or compensation closes the gap; for every case the platform has today, one of them does.

Ordering: make the crash leave a safe state. Put an idempotent protective write before the risky write and repeat it after. Example: changing or resetting a password stores the new password hash, renews the user's security stamp and revokes the user's refresh tokens and authorizations, three separate writes. If the handler dies after the password write and before the revocation, the new password is active while old refresh tokens can still be redeemed until they expire. The rule: revoke the tokens before the password is written and once more after it. A crash at any point then leaves at most the tokens issued in the moment between the two revocations, and a retry is harmless because revoking is idempotent. (When the writes are in the same database, one transaction over all of them closes the gap completely, see R3.)

Compensation: a reconciliation job repairs what a best-effort step lost. Example: adding or removing a secret of a client that is mirrored into child tenants commits the change on the client first and then updates the mirrors as a best-effort step. If that step fails after a secret was removed, the removed secret stays valid in the child tenants. The handler does not retry; a periodic reconciliation job (default every 5 minutes) plus the re-provisioning endpoint derive every mirror from its source client again, so a lost update heals by itself. The job must be idempotent and must be able to run while handlers write: every writer of the derived data writes with the version it read (a write without the expected version does not advance it), otherwise a slow job can write an older state back.

Commit, then best-effort, when the loss heals by itself. Example: the message that tells other services to refresh their cached CORS settings after a client change; a lost message is repaired when the cache expires.

Behaviour the platform already defines needs no extra pattern: the audit event and the completion of an idempotency key after the handler.

Durability of the write itself is not a side effect you have to handle: the tenant repository writes with majority write concern (OD-7), so a committed write has reached a majority of the database members before the handler answers.

Keep secrets out of side effects: sensitive method parameters (sensitive: true in the model) must not reach logs, audit events, traces, outbox payloads or error parameters. What the platform masks for you, and what it does not, is described in Sensitive parameters and filters.

Failure examples​

All examples use the method User.ChangePassword (the caller changes a password; see the worked example). The handler writes the new password hash, and then answers.

#What happensCaller seesStateRight reaction
1Handler crashes after the write, before the answer.METHOD_TIMEOUTThe password is changed.Read state: try to sign in with the new password. With an idempotency key the first call is burned (unknown outcome): a retry with the same key answers METHOD_OUTCOME_UNKNOWN; after reading the state use a new key.
2Handler crashes before the write.METHOD_TIMEOUTUnchanged.Same as 1 — the caller cannot tell 1 and 2 apart from the envelope alone. Read state.
3Handler throws an unexpected exception.METHOD_HANDLER_FAILED with correlationIdUnchanged if the handler follows R3; the exception is in the handler log under the correlation id.Report the id; retry later.
4Deadline passes while the handler is slow. The handler's token is cancelled at the deadline and the handler answers; the dispatcher waits a short grace period (default 2 s) longer and then answers itself.METHOD_TIMEOUTThe handler does not start a write after its deadline; a write already running may complete.Read state.
5Request arrives after its deadline (queue was backed up).METHOD_TIMEOUTNot executed.Retry with a new deadline.
6No handler is running (service down, not yet announced).METHOD_HANDLER_NOT_AVAILABLE in under one secondNot executed.Retry later. Safe even for non-idempotent methods.
7The user double-clicks. Two invocations.Two answers; the second may fail with CURRENT_PASSWORD_INVALID because the first already changed the password.Changed once, by luck. For a method without such a natural guard it would run twice.Disable the button while a call is open; use an idempotency key (R8): the second click returns the first result.
8Broker delivers the message twice.One answer; the duplicate answers METHOD_OUTCOME_UNKNOWN.Not executed again, as long as the duplicate is recognized (see limits).Read state.
9Rolling update: a newer gateway sends a newer wire contract version to an old handler.METHOD_CONTRACT_VERSION_UNSUPPORTEDNot executed.Retry after the update.
10Caller aborts the GraphQL request (browser closed) while the call is in flight.Nothing (no client)The handler runs to completion unless its deadline passes.None — the method is not tied to the request's lifetime.
11Mixed request: createDevice together with a method call in one GraphQL request (or two method calls).The gateway rejects the whole request before it executes anything.Nothing changed: no generic mutation and no method ran.Send the generic mutations in one request and the method call in its own request, after the first one succeeded. See decision OD-1 and OD-2.
12Handler wrote two documents without a transaction and crashed between them.METHOD_TIMEOUT or METHOD_HANDLER_FAILEDHalf done.Handler bug: violates R3. Use a transaction or a compensation.

Decision table​

Use it to decide how to build and how to call.

QuestionAnswer
Does my handler see the caller's uncommitted data?No (R1). Commit before you call.
May a handler use the request's database session?No (R1). The context has none; do not reach for the request-scoped repository.
Can I get two method calls to commit or fail together?No (R2). Make it one method, or use compensation.
What if the handler writes to several documents?Own session and transaction, commit, then answer (R3).
What if the handler cannot tell whether it committed?METHOD_OUTCOME_UNKNOWN.
Will the platform retry a failed call?Never (R4). A delivery it knows to be a possible duplicate is not executed (METHOD_OUTCOME_UNKNOWN).
May a caller retry?Yes after METHOD_HANDLER_NOT_AVAILABLE, METHOD_CONTRACT_VERSION_UNSUPPORTED (nothing ran). After METHOD_TIMEOUT, METHOD_OUTCOME_UNKNOWN, METHOD_HANDLER_FAILED only after reading the state, or if the method is declared idempotent. With an idempotency key the same key may be sent again after METHOD_HANDLER_NOT_AVAILABLE, METHOD_CONTRACT_VERSION_UNSUPPORTED and METHOD_REQUEST_IN_PROGRESS; after METHOD_TIMEOUT, METHOD_OUTCOME_UNKNOWN and METHOD_HANDLER_FAILED the key is burned (use a new one after reading the state).
Does a duplicate call run twice?Yes, unless an idempotency key is used (R8).
What order do concurrent calls run in?Undefined (R7). Wait for the answer if order matters.
When do I see the effect of a call?After the success envelope (R6).
When do I publish an event or send a mail?After the commit. If it must not be lost: order the writes so that a crash leaves a safe state, or let a reconciliation job repair it (Side effects); there is no platform outbox (OD-6).
Where do I put a retry loop?In the caller, not in the handler (the bus does not retry).
Long-running work?An asynchronous method returns a handle after accepting; the same rules apply to the work behind it.

Open decisions​

Points marked Decided are settled; they stay here with their reasoning. The others are not decided: the text above states the documented default only where the platform already behaves in a defined way, and each open point lists the options; the decision is taken by the owning feature.

OD-1 — Visibility of uncommitted request writes to a method in the same request​

Decided (OD-1 option C, OD-2 option A): one gateway rule for OD-1 and OD-2. A GraphQL request carries either generic mutations or exactly one method call. A request that mixes generic mutations with a method call, or that calls a second method, is rejected before anything is executed: no generic mutation and no method runs. This was the earlier recommendation ("at most one method per request"); the two problems it solves are:

  • A method that targets an entity created or changed earlier in the same request would not find it, because the request transaction has not committed and the method runs outside it (R1).
  • With several calls in one request, the order, the handling of a failure in the middle and the reporting of partial results would all have to be defined; with one call per request, the envelope of that call is the whole answer.

The alternatives were to run all methods after the request commits (methods would see the generic writes, but would be skipped on failure and run in another order than written) and to run methods at their written position outside the transaction (simple, but side effects stay when a later mutation fails). Both leave the partial-failure question open and were not chosen.

Consequences for callers: create the entity in one request, call the method in the next. Several method calls are several requests; they follow R7 (no ordering unless the client waits for the answer).

Decided (option A): there is no exemption for asynchronous methods. A method that only returns an operation handle counts like any other method: one method call per request, several handle-returning calls in one request are rejected.

The rule is enforced in the validation phase of the GraphQL request, before any resolver runs; the refusal is one error with the code ONE_METHOD_CALL_PER_REQUEST (details). The code belongs to the GraphQL layer, not to the response envelope of the method wire contract. The test that decides whether a method counts is one function, so a later change of the rule would touch only that function.

OD-2 — Several method mutations in one request​

Decided together with OD-1: there are no several method mutations in one request. A second method call is rejected before execution (error ONE_METHOD_CALL_PER_REQUEST), so ordering, stop-or-continue after a failure and a summary of partial failure do not arise. Asynchronous methods are not exempt, see OD-1.

OD-3 — Recognizing a duplicate delivery​

Decided (2026-10-30, decision note of the spike, Odin). Option A, as shipped. The handler host does not execute a delivery that the broker flags as a possible duplicate, nor an invocation id that the instance has already seen (a per-instance memory); the caller receives METHOD_OUTCOME_UNKNOWN. There is no shared store for invocation ids. Which duplicates this recognizes and which not: limits.

Reasons:

  • The gateway creates the invocation id for every call, and no API lets a client choose it. A re-sent call is therefore a new invocation and a matter of the idempotency key (OD-4). A duplicate of the same invocation id comes only from the broker (redelivery, flagged) or from a foreign publisher on the bus, which a claim store would not stop either: it simply uses a new id.
  • A shared claim for every call would add a database round trip to a dispatch path that costs about 1 ms, and every handler host would depend on that store.
  • The store of the idempotency key is consulted at the gateway before the message is sent, so it cannot see a second delivery of one invocation id.

Trigger to revisit (option B'). B' is a claim taken by the handler host only for calls with an idempotency key, reusing the document of the key. It is defined but not scheduled. It is cut when one of these happens:

  1. A duplicate delivery without a broker flag is measured: the counter ck.method.duplicate_refused (Method metrics) shows refusals with the reason seen_id while broker_flag stays flat, or a forced-failure test shows it.
  2. A method is declared that cannot tolerate double execution and has no natural guard of its own.

Until then, a method that must never run twice uses an idempotency key and writes idempotently (see limits).

Owner: method dispatch and idempotency features.

OD-4 — Idempotency key details​

Decided (2026-10-30, decision note of the spike, Odin and Gerald). The rules of R8, in six rows plus two security decisions:

#QuestionDecision
1Scope of a keyTenant, qualified method id, caller (from the principal provider of the validated token, never from the informational caller data: the sub of a user, client:<client_id> of a service token) and the key. Stored as a SHA-256 hash; the raw key is not at rest. The target is not part of the scope but of the fingerprint.
2Same key, other parametersRefused with METHOD_VALIDATION_FAILED (field = idempotencyKey, reason = parameters-changed). The fingerprint covers type, target and canonical parameters and is computed before redaction.
3Same key while the first call runsNew additive platform code METHOD_REQUEST_IN_PROGRESS (the call is not executed; send the same key again later). No waiting.
4Which outcomes are storedSuccess, accepted and method-declared errors are stored and replayed. An unknown outcome (timeout, outcome unknown, a claim that expired in flight) burns the key (METHOD_OUTCOME_UNKNOWN). METHOD_HANDLER_FAILED burns the key too, as a stored failure: the caller retries with a new key, so the method never runs twice by mistake. Only not-executed platform codes release the key.
5LifetimeMethods:Idempotency:Ttl, default 24 hours, maximum 7 days, per-document expiry with a TTL index as housekeeping.
6StoreThe tenant database, collection MethodIdempotency; part of the tenant backup (it is not excluded); fails closed.
G1Fingerprint of sensitive parameters (option A)A keyed HMAC computed before redaction, with a configuration secret shared by all replicas and a previous-key slot for rotation (Methods:Idempotency:Fingerprint:Current and Previous). Without a configured key a keyed call to a method with sensitive parameters is refused (fingerprint-key-missing). Platform Ops provides the secret per environment before keyed calls are enabled on shared environments.
G2Methods with a sensitive result (option A)Such a method refuses a key (reason = result-sensitive); nothing is stored. To be revisited when the first such method needs idempotency (then probably encrypted storage).

Consequences, behavior and operation: R8 and Operating idempotency keys.

OD-5 — Asynchronous methods​

Decided (2026-10-18): async is declared on the method definition (execution.mode: async, default sync). There is no per-call flag; one method is synchronous or asynchronous for all callers. The mode is part of the invocation contract (it changes the response shape): changing it is a major change, and a type that redeclares an interface method with another mode is rejected. For an asynchronous method, timeoutSeconds bounds the accept step (the time until the handle is returned), not the operation.

Decided (2026-11-07, decision note of the spike, comment 6651539; decision in comment 6651544, Odin): at most once after accept, a durable operation record in the system database, 7 days retention, no cancellation. The behavior below is implemented: the operation record, the handler SDK and the sweep in the platform's method runtime, the handle and runtime.operations in the GraphQL API (Asynchronous methods and operations); how a handler accepts a call is described in Asynchronous handlers.

#QuestionDecision
1What does "accepted" guarantee?At most once after accept. The operation record is written durably before the accepted envelope (the handle) is returned. The work starts once; the platform never starts it a second time.
2What happens when the handler is lost after accept?The operation ends Failed with METHOD_OUTCOME_UNKNOWN (reason handler-lost). It is never re-run, also not after a restart. Whether to run it again is the caller's decision, with a new idempotency key.
3Does a handle survive a restart, and how long is it kept?The handle survives restarts and rolling updates of the services. It is readable while the operation runs and for 7 days after completion (configurable from 1 hour to 30 days). After that, and for unknown or foreign ids, the answer is OPERATION_NOT_FOUND.
4Where is the operation state stored?In the system database, with the tenant id mandatory in every key and every query. Not in the tenant database: a restore replaces the tenant database and would erase (or resurrect) the record of the restore operation itself. Records of a deleted tenant are deleted with it. A tenant created later with the same id never reads an operation of the deleted tenant: every read ignores records accepted before the current tenant was created, also when the deletion could not remove them.
5Who may read a handle?Within the tenant of the route only: the caller who started the operation and tenant administrators. Callers can list their own operations, so an operation can be found again when the accept answer was lost. Anyone else gets the same OPERATION_NOT_FOUND as for an unknown id (no existence oracle).
6Can an operation be cancelled?No, not in this feature (non-scope; a later feature if needed). The existing cancel of the job endpoints stays for the migrated job types until then.
7What does a repeated idempotency key return?The same operation id as the first call, as long as the key lives (R8); no second operation is started. The 7-day retention is at least the maximum key lifetime, so a replayed key never returns an expired handle.
8Which polling mechanisms does the handle replace?The job polling of the bot services, one job type at a time: dump first, then restore and archive export/import. system/v1/jobs keeps answering for the migrated job types and is marked deprecated ([OctoRestDeprecated]) with a sunset date that is to be announced (decided once the migration is implemented, Gerald 2026-11-11; see Deprecated REST Endpoints); it is not removed in this feature.

Delivery state of the points above: 1–3, 5 and 7 are implemented as decided; point 7 holds while Methods:Operations:Retention is at least Methods:Idempotency:Ttl (the defaults, 7 days and 24 hours, satisfy it) and not for a method declared execution.idempotent: true, which ignores the key. Point 4: the record lives in the system database and every read names the tenant; when a tenant is deleted, the bot services remove its records, and a tenant created later with the same id ignores (and its setup removes) any record accepted before it was created, also when that removal failed. Point 6: there is no cancel. Point 8: the repository dump runs as the asynchronous method TenantRepository.Dump; restore and archive export/import follow, and the deprecation of system/v1/jobs comes with them.

Reasons: this is the only combination consistent with the no-retry rule (R4), the stored accepted outcome of idempotency keys (R8) and OD-10 option C (the validated principal is kept in the process and no token is stored, so a restarted instance cannot resume work on the caller's behalf). The alternatives were rejected: at least once after accept (a durable queue that resumes or re-runs after a restart) needs a stored principal or token and handlers that tolerate a re-run, which dump and restore do not; best effort (an in-memory handle) loses the handle on a restart or on another replica and fails the requirement that handles survive a restart; the bot job store as the operation store would couple the public handle to an implementation detail of the bot services and to its retry defaults.

Consequences for callers: an accepted call means "durably recorded and started at most once", not "will complete eventually". After Failed with handler-lost, read the state of the target before starting the operation again. The handle shape in the envelope is fixed (operation id, accepted time, retryAfterSeconds, expiry).

Owner: asynchronous methods feature.

OD-6 — Outbox as a platform component​

Options: guidance only (this page), a reusable outbox helper in the handler SDK, or a platform-wide outbox service.

Decided (2026-11-01, decision note of the spike, comment 6629556; decision in comment 6629611, Odin and Gerald): option A, guidance only. The platform provides no outbox component. Handlers order and compensate their side effects as described in Side effects.

Reasons:

  • None of the seven cases in sight needs an outbox. The password change needs ordering (revoke tokens before and after the password write), the mirrored client secrets need a reconciliation job, and the rest (CORS cache message, audit event, idempotency completion, role and permission methods with a single write) is ordering or behaviour that is already defined.
  • A helper in the handler SDK (about three days) brings a relay with leases, an outbox collection per tenant and an at-least-once contract every receiver must honour; a platform-wide service would take weeks and adds a component every handler depends on. Neither pays off without a case that needs it.

Trigger to revisit. The first method that must send a mail or call an external system if and only if its write committed (today no method sends a mail). Then the helper (option B) is built next to that method first and generalised afterwards. A platform-wide outbox service (option C) is not planned.

Owner: handler SDK.

OD-7 — Durability before the success envelope​

Whether the platform requires a particular database write concern (for example majority) before a handler may answer success, or leaves the tenant repository defaults.

Decided (2026-11-01, decision note of the spike, comment 6629557; decision in comment 6629612, Odin and Gerald): option A, keep the defaults and document them, plus a guard test.

  • Every tenant and system repository client sets write concern majority with a wtimeout of 30 seconds and read concern majority in code. They do not come from configuration, and a connection option cannot weaken them. Journaling follows the replica set (writeConcernMajorityJournalDefault, MongoDB default true), so a majority acknowledgement means the write is in the journal of a majority of the members. A handler's own transaction (R3) inherits this write concern.
  • All shared environments run three-member replica sets without arbiters, so majority writes keep working with one member down. A local development stack runs a single member, where majority means that member.
  • A unit test of the MongoDB persistence layer pins all three values on the settings of the real database clients; it fails if one of them changes. Changing them needs a new decision here.
  • Cost: measured locally, majority adds about 0.3 ms per write over w: 1 (the journal wait); in a three-member cluster one replication round trip is added.

Reasons: the guarantee is already set in code for every client, which is stronger than a configuration default. Checking the replica set at service start (option B) needs a privileged database command for a case that supported installations do not have; forcing majority per write in every handler (option C) adds discipline without adding a guarantee.

If majority is not reached within the 30 seconds, the database reports a write concern error, but the write is not undone on the primary. The handler therefore cannot know whether the write is durable; that is an unknown outcome (R3), not a failure.

Owner: handler SDK.

OD-8 — Isolation of in-process handlers from the request​

The SDK test proves that the method context has no session member. It cannot prove that an in-process handler does not resolve a request-scoped repository that holds the session.

Decided: an in-process handler runs detached from the ambient execution context of the calling request, in a fresh dependency-injection scope that registers no request session, and the SDK adds a runtime check that fails the call when a session is still active in the ambient context (HasSession). The rule in Implementing Method Handlers remains the author's duty; the scope and the check make a violation fail loudly instead of silently joining the request transaction. The detached dispatch, the explicit carry-over of trace, culture and logging scope, and the probe hook (AddCkInProcessAmbientStateProbe<TProbe>()) are part of the handler SDK (Meshmakers.Octo.Services.Methods); the gateway registers the probe for its request session. Details: Implementing Method Handlers.

Owner: handler SDK, gateway features.

OD-9 — Sensitive parameters in pipeline handlers​

Decided: forbid. A pipeline handler receives the method parameters as pipeline input, and a pipeline execution stores its input and output. A method with at least one sensitive parameter therefore cannot declare execution.handler: pipeline: the compiler refuses the combination with message 132 (see Message Codes), so no tenant can run such a model. Methods that take a secret, such as setting a password, are service handlers. The pipeline handler refuses such a method at runtime as well, as a second gate.

The alternative of allowing the combination and redacting the values in every component that touches pipeline data was rejected: it needs a leak-free proof for every node and would need to be kept that way. It stays possible as a later extension if a concrete need appears.

The same rule holds for a sensitive result: the model marks it with sensitive: true on the result (Sensitive results), and the compiler refuses handler: pipeline for such a method with message 134, because the execution store keeps the output. The redaction of sensitive results in audit events, logs and traces follows with the method runtime; until it is released, do not rely on it.

Owner: method security, pipeline handler.

OD-10 — Trust of the caller token in remote handlers​

Decided on 2026-10-19 (option A with four adjustments). The bus is not authenticated per message, so the caller data of a request proves nothing. The question was how a remote handler learns who is calling.

Options:

  • A. Forwarded token. The handler validates the access token that the request forwards, with the same settings as the HTTP API of its service (issuer, signing keys from the identity service, expiry, tenant, scope).
  • B. A token per call. B1: identity mints a short-lived token (about 60 seconds) bound to the invocation, the method and the target. B2: the dispatcher signs it with its own key.
  • C. A stored principal. The platform keeps the validated principal instead of a token, for asynchronous methods.

Decision: A for synchronous methods, with four adjustments:

  1. An optional audience setting Methods:Token:ValidAudiences. The platform does not validate the audience on its HTTP APIs, so the default is empty = not validated; with it set, one of the listed audiences is required.
  2. Service tokens (client credentials, no sub) are accepted through client_id, fail-closed.
  3. No parent-tenant rule: the tenant_id claim must equal the tenant of the request. (Later narrowed: a parent tenant's user token may call the backup, restore and archive data methods of a child tenant, see Parent-tenant administration through CK methods.)
  4. An expired token has its own reason, token-expired; the code stays METHOD_FORBIDDEN, so a client can retry with a fresh token.

Sub-decisions:

  • The pipeline handler validates the forwarded token as well instead of trusting the unauthenticated Caller data. The pipeline caller is built from the validated token only, with trust level strong; a refused token starts no pipeline.
  • Asynchronous methods in Phase 3 use option C: the principal is kept in the process, no token is stored, and asynchronous methods are limited to bot job types (see OD-5).
  • B1 only if the security review of the credential-leak epic rejects the replay window of the forwarded token. Who may receive the token on the bus is OD-11.

Unaffected: the in-process transport (the process that started the call already authenticated the caller), the authentication at the gateway on the GraphQL path, and the redaction of tokens and sensitive parameters.

What option A accepts: without an audience setting, every octo_api token of the tenant is valid for a call, and a handler (or whoever can read the method queue) can replay the forwarded token against the APIs of the tenant until it expires. The steps and settings are in Caller token validation.

Owner: method security.

OD-11 — Who may serve a method on the bus​

Decided on 2026-10-23: B0 first, then B1 together with the restriction of the public broker port; option C applies until B0 is live.

The bus is not authenticated per message. The handler announcement only feeds the dispatcher's registry, which is advisory; the calls themselves go to the method queue, so whoever can consume that queue on the broker receives the parameters and the forwarded caller token of every call routed to it, announced or not. What decides the exposure is therefore who holds broker credentials with read permission on the method queues, not who may announce.

Options:

  • A. Authenticated announcement. Handler hosts announce with a service token (client credentials) and the registry accepts only announcements from the clients allowed for a method. It protects the registry (availability), but not the queue.
  • B. Restrict the broker. B0: a separate broker user for adapters and edge sites without read rights on the method queues. B1: TLS on the broker connection and no public plain port. A separate virtual host for the method queues is a later step.
  • C. Accept the risk for the current phase and document it, with a detective control.

Decision:

  1. B0 first, then B1 together with the restriction of the public broker port. Option A is later hardening of the registry, not the security control.
  2. Until B0 is live, option C applies: the risk is documented here, and an alert watches the consumer count of the method queues.
  3. No method that carries a password (or another secret) may be enabled on a shared staging or production environment before B0 and the broker port restriction are rolled out. Development and tests use local and test environments with test accounts only.
  4. A per-call token (OD-10, option B1) is introduced only if the security review rejects the replay window of the forwarded token after B0. The window is the remaining lifetime of the token, 3600 seconds by default per client; the verdict for production stays an assumption until the real per-client lifetimes are read.

Owner: method security, platform operations.

OD-12 — Data permissions of a method call​

Decided on 2026-10-19: option A for Phase 3, option B as the end state, option C rejected.

A method call on a record that data permissions protect must not be a way around them. The question was where and how the engine's data-permission rules apply to methods.

Options:

  • A. Gateway check. The gateway of the asset repository checks the data permission of the target with the one evaluator of the engine, before the handler. Handlers keep running with the rights of the service.
  • B. A plus a caller-bound handler. The handler SDK also hands the handler a data access that is bound to the validated caller, so every read and write of the handler is filtered like the caller's own.
  • C. No data permissions for methods in Phase 3. Rejected: a method on a protected record (a user, a client) would bypass what the generic mutation enforces.

Decision: A for Phase 3 (a public check in the engine, the gateway step, the dataAccess declaration), B as the end state (not scheduled), C rejected. Sub-decisions:

  1. The default data action of a method is Write. The optional declaration authorization.dataAccess: Read is additive and lets a query-like method pass with a read grant.
  2. allowSelf does not exempt the data-permission check.
  3. A target that data permissions hide from the caller is answered with METHOD_TARGET_NOT_FOUND, not METHOD_FORBIDDEN. The declared authorization runs before and stays identical for an existing and a missing target.
  4. Order: authenticate, declared authorization, load the target, data-permission check, dispatch.

Unaffected: the declared authorization (roles, scopes, octo_api, default deny), the generic mutation and read paths, the stream data and file guards.

Consequence for models: a user whose role has no data grant on the user records cannot change their own password. The model has to grant it (worked example).

The rules as built, with the error answers: Data Permissions and the gateway gate.

Owner: method security.

Error codes​

The authoritative table (code, parameters, whether the method ran, caller action) is part of the wire contract of the platform; this table summarizes it for handler authors.

CodeMethod ran?Typical causeCaller action
METHOD_HANDLER_NOT_AVAILABLEnoNo live handler, handler expired, pipeline not deployed; or the idempotency store cannot decide (reason: idempotency-store-unavailable)Retry later (the same idempotency key may be used again)
METHOD_TIMEOUTunknownNo answer before the deadline, or the request arrived after it (parameters methodId, timeoutSeconds: the timeout of the method that the deadline was derived from)Read the state; do not retry a non-idempotent method blindly
METHOD_FORBIDDENnoCaller may not invoke the method, the data permission on the target does not allow the action (reason: data-permission), or a remote handler rejected the forwarded token (reason: token-missing, token-expired, token-invalid, tenant-not-allowed, audience-mismatch)Do not retry; after token-expired retry with a fresh token
METHOD_VALIDATION_FAILEDnoA parameter is missing or invalid (parameter field). For the idempotency key: field = idempotencyKey with reason parameters-changed, result-sensitive, caller-unknown, key-too-long, fingerprint-key-missing or fingerprint-unavailable (R8)Fix and call again; for parameters-changed use a new key
METHOD_TARGET_NOT_FOUNDnoTarget does not exist, or data permissions hide it from the caller (the answer is identical)Check the id
METHOD_HANDLER_FAILEDyes, failedUnexpected handler failure; details only in the handler log under correlationIdReport the id; read state before retrying
METHOD_OUTCOME_UNKNOWNunknownPossible duplicate delivery or unknown commit stateRead the state
METHOD_REQUEST_IN_PROGRESSnoA call with the same idempotency key (same tenant, method and caller) is still running; this call was not executedSend the same key again later; the first result is then returned
METHOD_NOT_SUPPORTED_BY_HANDLER_VERSIONnoReserved: handler does not support the method version—
METHOD_CONTRACT_VERSION_UNSUPPORTEDnoMessage uses a newer wire contract version than the receiver understandsWait for the rolling update

The prefix METHOD_ is reserved for the platform; a model declares its own codes without it (see method definitions).

See Also​