Method Metrics and Traces
An operator can see the rate, the duration and the failures of method calls per method, and can follow one call from the gateway over the event hub into the handler in a single trace. This page names the meter, its instruments and labels, and the spans. The consistency rules are in the Method Consistency Model, the handler side in Implementing Method Handlers.
Every service that calls AddObservability() subscribes to the meter and to the span sources Meshmakers.Octo.Methods and MassTransit.
Meter Meshmakers.Octo.Methods
| Instrument | Type | Labels | What it counts |
|---|---|---|---|
ck.method.calls | counter | method_id, transport, outcome | every finished call, once |
ck.method.duration | histogram (seconds, buckets 5 ms to 300 s) | method_id, transport, outcome | the duration of the same call as the caller sees it |
ck.method.no_handler | counter | method_id, transport | calls that nobody could take: METHOD_HANDLER_NOT_AVAILABLE because no handler is alive, the tenant is not served or the method is not registered. The same code answered because the idempotency store is down is not counted here. |
ck.method.duplicate_refused | counter | reason, method_id | deliveries that a handler host refused as possible duplicates and did not execute |
ck.method.audit.failures | counter | transport | audit events that could not be written |
ck.method.operations | counter | method_id, state | state changes of operations of asynchronous methods: accepted, running, succeeded, failed. Operations the sweep ends as lost (METHOD_OUTCOME_UNKNOWN, handler-lost) count as failed with method_id other. |
In Prometheus and Dash0 the names use underscores: ck_method_calls_total, ck_method_duration_seconds_bucket, ck_method_no_handler_total, ck_method_duplicate_refused_total, ck_method_operations_total.
Labels
| Label | Values |
|---|---|
method_id | the qualified method id without model version, for example System.Identity/User.ChangePassword-1. A process reports at most 1000 distinct ids; further ones appear as other, so a caller that invents ids cannot grow the series without limit. |
transport | bus (handler in another service, over the event hub), in-process (handler in the calling service), gateway (refused by the gateway before the call entered the pipeline; only in ck.method.calls and ck.method.duration), other |
outcome | SUCCESS, ACCEPTED (long-running operation started), CALLER_CANCELLED, a platform error code (METHOD_FORBIDDEN, METHOD_TIMEOUT, METHOD_HANDLER_FAILED, METHOD_HANDLER_NOT_AVAILABLE, METHOD_OUTCOME_UNKNOWN, ...) or an error code that the method declares; invalid_code for any other value. It is the same vocabulary as the audit event. |
state | accepted, running, succeeded, failed (ck.method.operations only) |
reason | broker_flag: the broker flagged the delivery as a possible duplicate (redelivery, bus retry). seen_id: the broker did not flag it, but this instance has already seen the invocation id. |
What a label never contains: parameter values, results, tokens, idempotency keys, entity ids, user names or subjects, and the tenant id. The tenant is not a label of this meter, so that the number of series depends on the number of methods and not on the number of tenants.
How a call is counted
- Every call that goes through the dispatcher or the in-process transport is counted once, around the whole invocation pipeline. A refusal by the authorization filter, an answer replayed from the idempotency store, an exception and a cancellation by the caller are counted as well, with their outcome. The metrics are not a filter of the pipeline and have no place in the order of the filters (audit 100, authorization 200, idempotency 300).
- A call that the gateway refuses before it reaches the pipeline (forbidden, unknown target, parameters too large, ...) is counted by the audit recorder that writes the audit event of the refusal, with
transport="gateway"and the refusal code as outcome. Such a call appears wherever the method audit is registered. ck.method.duplicate_refusedis counted by the shared duplicate-delivery check that every handler host uses, once per refused delivery. If both signs apply, the broker flag is counted. A growingseen_idwhilebroker_flagstays flat shows duplicates that the broker did not flag; that is the trigger for the open decision on duplicate deliveries (option B' of OD-3) in the Method Consistency Model.
Example queries
# calls per second, per method and outcome
sum by (method_id, outcome) (rate(ck_method_calls_total[5m]))
# failure ratio per method (anything but SUCCESS or ACCEPTED)
sum by (method_id) (rate(ck_method_calls_total{outcome!~"SUCCESS|ACCEPTED"}[5m]))
/ sum by (method_id) (rate(ck_method_calls_total[5m]))
# 95th percentile of the duration
histogram_quantile(0.95, sum by (le, method_id) (rate(ck_method_duration_seconds_bucket[5m])))
# calls that found no handler in the last 15 minutes
sum by (method_id) (increase(ck_method_no_handler_total[15m]))
# refused duplicates by reason in the last hour
sum by (reason) (increase(ck_method_duplicate_refused_total[1h]))
One trace for one call
The trace source Meshmakers.Octo.Methods provides two spans. They carry the tags ck.method.id and ck.method.transport, never parameters, results or tokens.
| Span | Where | Notes |
|---|---|---|
CK method call | the calling service, around the whole call | tag ck.method.outcome is set when the call ends; it is a child of the span of the request that made the call |
CK method handle | the handler: the bus consumer of the handler host, or the in-process handler | a child of the bus process span (bus) or of the call span (in-process) |
Over the event hub the chain is: request span of the gateway, CK method call, send, receive and process spans of the bus (source MassTransit), CK method handle. All of them have the same trace id, also when the handler runs in another service: the bus writes the W3C trace context into the message.
Where the injected auto-instrumentation exports the spans (the usual setup in a cluster), the two source names must also be listed in the pod setting OTEL_DOTNET_AUTO_TRACES_ADDITIONAL_SOURCES: MassTransit and Meshmakers.Octo.Methods. Without a listener on MassTransit the handler starts a new trace.