Observability integration

OpenTelemetry spans, Prometheus metrics, structlog / loguru context and Sentry breadcrumbs for every machine -- one line with instrument_all(), telemetry-hygienic by default.

This library contains failures by design: a raising action is logged and the transition still completes, a raising guard is False, an unhandled event is dropped. A long-lived machine should not die because one side effect had a bad day — which also means those failures are invisible unless you look. The [observability] extra is how you look: every plugin is a thin mapping over the existing hooks, attached to one interpreter with .use() or to every interpreter in the process with a single instrument_all() call at startup.

Install

pip install "xstate-statemachine[observability]"
pip install structlog loguru sentry-sdk   # optional -- soft imports, never pinned

[observability] pins exactly two packages: opentelemetry-api>=1.20 and prometheus-client>=0.17. You still configure the OpenTelemetry SDK (exporter, sampler) yourself — the library only talks to the API, as the OTel project recommends for libraries. structlog, loguru and sentry-sdk are detected when their plugin is constructed; a missing one raises MissingExtraError naming the package (pip install structlog). Tested versions are in the compatibility table.

For a complete, runnable app – PrometheusPlugin and OpenTelemetryPlugin over two charts in choreography with in-memory exporters, tests that no order id becomes a label, and a test suite – see the eda_fulfilment example.

Quick start

from prometheus_client import CollectorRegistry, generate_latest

from xstate_statemachine import SyncInterpreter, create_machine
from xstate_statemachine.contrib.observability import (
    PrometheusPlugin,
    instrument_all,
    uninstrument_all,
)

machine = create_machine(
    {
        "id": "door",
        "initial": "closed",
        "states": {
            "closed": {
                "on": {
                    "OPEN": "open",
                },
            },
            "open": {
                "on": {
                    "CLOSE": "closed",
                },
            },
        },
    }
)

registry = CollectorRegistry()
attached = instrument_all(
    otel=True, prometheus=PrometheusPlugin(registry=registry)
)

door = SyncInterpreter(machine).start()  # built AFTER the call -> instrumented
door.send("OPEN")
door.send("KNOCK")  # undeclared -> label "unknown"
door.stop()

text = generate_latest(registry).decode()
assert (
    'xstatemachine_transitions_total{event="OPEN",from_state="door.closed"'
    in text
)
assert 'disposition="unhandled"' in text and "KNOCK" not in text
uninstrument_all(attached)  # tests: leave the process as found

A Grafana-ready starting point:

# transitions per second, by machine and event
sum by (machine, event) (rate(xstatemachine_transitions_total[5m]))
# share of events nobody handled (misrouted traffic, stale clients)
sum by (machine) (rate(xstatemachine_events_received_total{disposition="unhandled"}[5m]))
  / sum by (machine) (rate(xstatemachine_events_received_total[5m]))
# p95 event-processing latency
histogram_quantile(0.95, sum by (le, machine) (rate(xstatemachine_transition_duration_seconds_bucket[5m])))
# action failures that were contained -- alert on any
sum by (machine, action) (increase(xstatemachine_action_errors_total[15m])) > 0

OTel collector note: the plugins emit through the global TracerProvider. Point the SDK’s OTLP exporter at your collector (OTEL_EXPORTER_OTLP_ENDPOINT), and let the collector do tail sampling — a busy machine emits one span per event.

Reference

instrument_all(interp_or_app=None, *, otel=False, prometheus=False, structlog=False, loguru=False, sentry=False, discovered=False, allow=None)

Each flag is False, True (a default-configured plugin) or a ready plugin instance. The target decides where they go:

discovered=True also attaches entry-point plugins through plugins.attach_discovered() (#296); allow= narrows them by name. Returns the plugins attached. Anything else raises TypeError.

uninstrument_all(attached)

Unregisters a global instrument_all() (new interpreters only).

OpenTelemetryPlugin(tracer=None, *, span_per="event", record_context=False, redact=())

One span named statechart.transition per processed event, opened in on_event_received and closed in on_event_processed — the hook that knows the outcome:

Guard evaluations are span events (guard_evaluated with guard.name, guard.result). Action / guard errors, service errors and runaway-chain trips are recorded with record_exception and set the span status to ERROR. Each invoked service gets a child span statechart.service (service.src, duration). A traceparent in the event payload (or payload["headers"]) becomes a span link to the remote context; an invalid one is ignored. span_per="transition" adds a statechart.microstep child span per transition taken.

📝 There is no official OpenTelemetry semantic convention for state machines; the statechart.* namespace is ours.

agent_span_exporter(tracer=None)

An on_span callable for the [agents] AgentTracePlugin: each trace record becomes a gen_ai.<operation> span with its gen_ai.* fields. AgentTracePlugin(on_span="otel") wires it for you.

PrometheusPlugin(registry=None, *, labels=("machine",), max_label_values=100, clock=None)

Metric objects are created once per CollectorRegistry and label set and shared, so several plugins on one registry never collide; a second plugin with a different labels= on the same registry is a ValueError. labels=() drops the machine label. queue_depth has no hook — it is a collector that reads interpreter.queue_depth from live interpreters at scrape time.

StructlogPlugin() / LoguruPlugin(logger=None)

Bind machine_id, state (sorted leaf ids), event and — when the event payload carries one — correlation_id from on_event_received until on_event_processed, so any log line your actions emit carries them. structlog: through structlog.contextvars (keep merge_contextvars in your processors); loguru: logger.contextualize() → record["extra"]. Unbound afterwards. The in-flight stack is per thread and per asyncio task, so several async interpreters on one loop never unwind each other’s fields. correlation_id is truncated to 128 characters. LoguruPlugin(logger=...) raises TypeError for an object without .contextualize().

SentryPlugin(level="info", *, capture_errors=False, sdk=None)

A statechart breadcrumb per transition (door.closed -> door.open (OPEN)). With capture_errors=True, action / guard / service errors and chain trips are sent with capture_exception and statechart.* tags. Works with sentry-sdk 1.x (push_scope) and 2.x (new_scope). Each failure is captured once (the action error, not also its on_error). Breadcrumb data is only machine_id and event – never payloads or context. If an sdk call raises, the plugin logs one warning and goes inert (same as OpenTelemetryPlugin with a raising tracer); the chart keeps running.

LabelGuard(max_label_values), event_label(machine, event_type), UNKNOWN, OTHER

The hygiene primitives the plugins share, exported for your own exporters.

Guarantees

What this does: observes every hook without changing behaviour — every plugin is wrapped by the engine’s _SafePlugin, so an exporter that raises is reported via on_plugin_error and the machine keeps running. instrument_all() covers every interpreter constructed after the call, on both engines, including children and restores. The hot path pays for on_event_processed only when a plugin overrides it (all of these do, by design: it carries the outcome).

What this does not do: configure an OTel SDK or exporter; retro-instrument interpreters that already exist; guarantee delivery of telemetry (exporters are best-effort); emit OTel messaging spans for brokers (the broker adapters do not trace; carry traceparent in the envelope). Per-event overhead was measured once, on a two-state chart (Windows, CPython 3.14): roughly 3× with PrometheusPlugin, 4–5× with OpenTelemetryPlugin (SDK, no exporter), about 40 % of it the engine’s per-event receipt. It is not a budgeted benchmark; measure on your chart.

See the programme-wide Guarantees and Security pages (#303).

Threat model

Who can call this: your own process; nothing here opens a port. The /metrics endpoint, if any, is yours to expose and protect.

What it exposes (X0.6 telemetry hygiene): a metrics backend is long-retention and priced by cardinality, so label values come from the chart, not from traffic — an event type the machine does not declare is unknown, engine events collapse to their family (after, done, error, xstate), and every label dimension is capped at max_label_values with the overflow bucket other (proven with 1,000 distinct event names). Never labels or attributes by default: event payloads, context, interpreter ids / store keys, correlation ids. Sentry tags and breadcrumbs carry names only. correlation_id is bound as log context (per line), never as a metric label.

You must configure: your OTel SDK and exporter; record_context=True only with a reviewed redact= list; capture_errors=True only if your Sentry project may receive exception messages (they can echo request data).

Compatibility

Troubleshooting