Recipe: WebSocket reconnect

disconnected ↔ reconnecting → connected with RetryPolicy jittered exponential backoff, and the live socket as a from_callback actor whose cleanup closes it. Both engines, fake client, simulated clock.

7 min read

Reconnect logic is where hand-written socket clients rot. There is the backoff counter, the retry that fires after the user already clicked Disconnect, and the socket that is never closed because the state changed while it was opening. As a chart, each of those is a transition, and the library supplies both hard parts:

RetryPolicy(...).logic() provides a named delay retryDelay (exponential backoff with jitter, computed from ctx["attempt"]) and retryCanRetry / retryBump / retryReset.
from_callback(listen) turns the live connection into an actor that exists exactly while connected is active. Messages and the close become events. Its cleanup closes the socket whenever the state is left, for any reason.

Files: examples/recipes/websocket_reconnect/.

stateDiagram-v2
  [*] --> disconnected
  disconnected --> connecting: CONNECT
  connecting --> connected: done / retryReset
  connecting --> reconnecting: error / retryBump
  connected --> reconnecting: DROPPED / retryBump
  reconnecting --> connecting: after retryDelay [retryCanRetry]
  reconnecting --> failed: after retryDelay
  connected --> disconnected: DISCONNECT
  reconnecting --> disconnected: DISCONNECT
  failed --> connecting: CONNECT

xsm simulate examples/recipes/websocket_reconnect/machine.json --events CONNECT,DROPPED,DISCONNECT
# -> socket.disconnected   (a drop, then the user leaves mid-backoff: no retry fires)

retryDelay is a named delay, which the simulator’s stub logic leaves unresolved. Supply the policy’s logic, as below, to step through the backoff.

The code

import random
from xstate_statemachine import (
    MachineLogic,
    SimulatedClock,
    SyncInterpreter,
    create_machine,
    from_callback,
)
from xstate_statemachine.patterns import RetryPolicy


class FakeClient:  # your adapter over `websockets`, `aiohttp`, ...
    fail = 0

    def connect(self):
        if FakeClient.fail:
            FakeClient.fail -= 1
            raise ConnectionRefusedError

    def close(self):
        self.closed = True


chart = {
    "id": "socket",
    "initial": "disconnected",
    "context": {
        "attempt": 0,
    },
    "states": {
        "disconnected": {
            "on": {
                "CONNECT": "connecting",
            },
        },
        "connecting": {
            "invoke": {
                "src": "openSocket",
                "onDone": {
                    "target": "connected",
                    "actions": "retryReset",
                },
                "onError": {
                    "target": "reconnecting",
                    "actions": "retryBump",
                },
            },
        },
        "connected": {
            "invoke": {
                "src": "listen",
            },
            "on": {
                "DROPPED": {
                    "target": "reconnecting",
                    "actions": "retryBump",
                },
                "DISCONNECT": "disconnected",
            },
        },
        "reconnecting": {
            "on": {
                "DISCONNECT": "disconnected",
            },
            "after": {
                "retryDelay": [
                    {
                        "target": "connecting",
                        "guard": "retryCanRetry",
                    },
                    {
                        "target": "failed",
                    },
                ],
            },
        },
        "failed": {},
    },
}

sockets = []


def open_socket(i, ctx, e):
    c = FakeClient()
    c.connect()
    sockets.append(c)  # raises -> onError


def listen(send_back, receive, ctx, e):
    c = sockets[-1]
    c.drop = lambda: send_back("DROPPED")  # wire your client's on_close here
    return c.close  # cleanup, exactly once


policy = RetryPolicy(
    max_attempts=3, base_ms=1000, jitter="full", rng=random.Random(7).random
)
logic = MachineLogic(
    services={
        "openSocket": open_socket,
        "listen": from_callback(listen),
    }
).merge(policy.logic())
clock = SimulatedClock()
s = SyncInterpreter(create_machine(chart, logic=logic), clock=clock).start()

s.send("CONNECT")
assert s.value == "connected"
sockets[-1].drop()
clock.increment(0)  # the server went away
assert (
    s.value == "reconnecting" and sockets[-1].closed
)  # old socket closed by the cleanup
clock.increment(1000)  # full jitter: at most base_ms
assert s.value == "connected" and s.context["attempt"] == 0

FakeClient.fail = 99  # server stays down
sockets[-1].drop()
clock.increment(0)
for _ in range(3):
    clock.increment(4000)
assert s.value == "failed"  # gave up after max_attempts

Why jitter

When a server restarts, every client drops at the same instant. With pure exponential backoff they all come back at the same instants too (1 s, 2 s, 4 s) and knock the server over again. jitter="full" spreads each retry uniformly over [0, delay]. Inject rng=random.Random(seed).random in tests and the delays become exact. The recipe’s test uses a seeded RNG and asserts that the machine stays reconnecting until one millisecond before the computed delay, then reconnects.

What the tests pin down

tests/recipes/test_websocket_reconnect.py runs every case on SyncInterpreter and Interpreter:

receive a message, then disconnect: the socket is closed by the from_callback cleanup;
a drop reconnects after exactly the jittered delay and resets attempt;
after max_attempts failed connects the machine is failed, having made 3 connects, not 4;
DISCONNECT during backoff cancels the pending retry, and no connect happens afterwards.

Troubleshooting

You see	Why	Fix
The machine sits in `failed`	`max_attempts` connects in a row failed (5 in `ws_reconnect.py`, 3 above). `failed` is deliberate: retrying forever hides an outage.	Send `CONNECT` to start over (it resets `attempt`), e.g. when the user returns or the network comes back.
Clients hammer the server in waves after a restart	Backoff without jitter.	Keep `jitter="full"`.
Two sockets open at once	The socket was opened outside the `listen` callback, so leaving `connected` cannot close it.	Open it in `openSocket` and return `client.close` from `listen`: the cleanup runs exactly once on exit.
A drop is never noticed	Your client’s close callback is not wired to `send_back("DROPPED")`.	Wire it inside `listen`.

What this does not do

No message is buffered or re-sent: anything you send while reconnecting is yours to queue. Reconnection state lives in memory, so a process restart starts disconnected.

On SyncInterpreter, socket callbacks are mailbox events: from_callback’s send_back posts through send_threadsafe(), so a MESSAGE the socket thread delivers is processed at the next send() / tick() on the owning thread, not the instant it arrives. A long-lived sync client should tick() on a schedule, or run on the async Interpreter, where the loop delivers them.

Edit this page on GitHub