Skip to content

Heartbeats and RPC

Redis carries service liveness, game and player request-response calls, and player-to-player messages. Each contract uses a separate key or channel pattern.

Current implementation

Maintainers can diagnose the runtime by inspecting these keys, channels, and payloads. The timeout table defines how long clients wait. None of these contracts is a public transport protocol for external services.

flowchart LR
    service["Game or player service"] -->|"heartbeat hash"| redis[(Redis)]
    service -->|"shutdown tombstone"| redis
    registry["Service registry"] -->|"scan and read"| redis
    em["Experiment runner client"] -->|"RPC request"| redis
    redis -->|"command subscriber"| service
    playerA["Player service"] -->|"session message"| redis
    redis --> playerB["Other role subscriber"]

Heartbeat and tombstone keys

Key Writer Reader Lifetime
heartbeat:<service_name>:<uuid> Game or player heartbeat broadcaster Service registry Refreshed every 3 seconds; expires after 10 seconds
tombstone:<service_name>:<uuid> Broadcaster during graceful shutdown Registry expiry diagnostics Expires after 120 seconds

A heartbeat includes service UUID and name, timestamp, readiness, monotonically increasing sequence, uptime, PID, and hostname. Player and game heartbeat variants add their service state; players also report capabilities.

Graceful shutdown changes readiness to not_ready, writes a final heartbeat, then writes a tombstone. The tombstone includes the final readiness, failure reason, heartbeat count, and uptime.

Expiry diagnosis

ServiceExpiredContext records:

  • whether a tombstone exists;
  • whether the heartbeat key still exists and its remaining TTL;
  • any remaining hash fields;
  • the last sequence, uptime, PID, and hostname seen by the registry.
Failure category Condition
graceful_shutdown A tombstone records normal service shutdown.
partial_hash No tombstone exists, but the heartbeat key and some fields remain.
unexpected_disappearance No tombstone or usable remaining heartbeat fields exist.
never_connected Declared enum value that the current context property does not return.

Generated heartbeat contracts

These generated models expose fields used by the current service implementation. They do not define a supported transport or Python extension interface.

BaseHeartbeat pydantic-model

Bases: BaseEvent

Event for a service to indicate that it is alive and well.

This is also used as connect events.

Attributes

timestamp class-attribute instance-attribute 

timestamp: Instant = Field(default_factory=Instant.now)

Wall-clock instant at which this heartbeat was created.

ready_state instance-attribute 

ready_state: ReadyState

Service eligibility for work, independent of the phase-specific state.

is_idle property 

is_idle: bool

Check if the service is in an idle state.

PlayerHeartbeat pydantic-model

Bases: BaseHeartbeat

Event for a player service to indicate that it is alive and well.

Attributes

capabilities instance-attribute 

capabilities: PlayerCapabilities

Capabilities advertised at registration and serialised in the Redis heartbeat hash.

is_idle property 

is_idle: bool

Check if the player is in an idle state.

Tombstone pydantic-model

Bases: BaseEvent

Tombstone written on shutdown to indicate a service has stopped.

Attributes

uptime_seconds instance-attribute 

uptime_seconds: float

Seconds from heartbeat-broadcaster creation until service shutdown.

ServiceExpiredContext pydantic-model

Bases: BaseModel

Diagnostic context gathered by the registry when a service expires.

This bundles all the Redis-probed information (tombstone, key state) into one object that gets passed to the EM's _handle_expired_service for rich logging.

Attributes

tombstone class-attribute instance-attribute 

tombstone: Tombstone | None = None

Shutdown record retained in Redis when expiration was diagnosed.

heartbeat_key_ttl class-attribute instance-attribute 

heartbeat_key_ttl: int = -2

Raw Redis TTL at diagnosis.

A value of -2 means that the key was absent.

remaining_heartbeat_fields class-attribute instance-attribute 

remaining_heartbeat_fields: dict[str, Any] = Field(default_factory=dict)

Hash fields still present in the heartbeat key when expiration was diagnosed.

last_heartbeat_seq class-attribute instance-attribute 

last_heartbeat_seq: int | None = None

Sequence number from the last heartbeat accepted into the registry manifest.

failure_category property 

failure_category: FailureCategory

Determine the failure category from the gathered diagnostics.

RPC channels

Channel Client Service
game:<uuid>:commands:<command> GameClient GameService
player:<uuid>:commands:<command> PlayerClient PlayerService

BaseRPCClient sends a payload through FastStream's request-response operation and decodes the reply. Service handlers register one subscriber per command. The response decoder reconstructs transported exceptions where the service returns one.

A timeout does not identify one cause

The default Redis RPC timeout is 600 seconds. Individual bomb-state, observation, and game control calls can use shorter limits. A timeout can mean the subscriber is absent, the service expired, Redis is unreachable, or the operation itself exceeded its limit. Check heartbeats, the registry, and the corresponding process log before changing a timeout.

Player messages

Player communication uses:

Message channel
session:<session_id>:player:<role>:messages

A player publishes to the other role's channel. IncomingMessageHandler subscribes when the player is configured for the session and stores received messages. It supplies the messages to the next forward pass. These messages are asynchronous events rather than RPC commands.