ZyrisDocs
Protocol specification

Relayed hops

What the protocol requires of a multi-replica server that carries calls between replicas.

A multi-replica server terminates a node's websocket on exactly one replica — the owning replica — while calls may originate on any of them. The call has to cross a hop the protocol does not define. How that hop is carried is the deployment's business; what the protocol requires of it is not:

  • Ownership must be fenced. A replica that believes it owns a connection it has lost will answer for a node it cannot reach. Carry a per-connection epoch on every relayed message and drop traffic fenced by a newer one.
  • Ordering is per direction, per call. Control and data interleave meaningfully — a credit grant that overtakes the chunk it was issued for is a bug — so a relay must preserve the order of everything it carries in one direction of one call.
  • Loss must surface, not silently truncate. chunk_seq exists for exactly this hop. An at-most-once transport can drop a chunk, and the receiver MUST fail the stream with stream_lagged rather than deliver bytes past the gap. Credit is the only bound on relay buffering, since there is no end-to-end TCP backpressure across the hop.
  • The 256 KiB chunk cap keeps a relayed frame inside the message-size limit of most brokers.

A deployment with no relay routes node calls on the owning replica only; a call that lands elsewhere fails with node_offline, which is retriable.

Attacca carries this hop over NATS, with epoch fencing against the node's row.