Skip to content

Suite configuration

Suite YAML defines missions, protocols, pairing policy, manual profile, modalities, and revision. suite freeze records a frozen revision in configs/suites/suites.lock. Specification generation reads that frozen revision rather than the current YAML.

Configuration shape

Files under configs/suites/ use Hydra's _target_ keys to construct Suite, SuiteMatchup, PlayerProtocol, and ManualProfile. Start from configs/suites/_template.yaml.

PairingType accepts:

  • with_self for the same configured player in both roles;
  • pairwise for every ordered Defuser and Expert pairing;
  • with_best_defuser or with_best_expert for an anchor from the run manifest; and
  • no_expert for a solo Defuser protocol.

The defuser_protocol role must be defuser. An expert_protocol, when present, must use expert. A solo Defuser cannot have an Expert.

Increase the revision when measured content changes

suite freeze rejects a changed suite digest at an existing name and revision. Increase the revision, then freeze the new configuration and mission snapshot.

modality is sorted and deduplicated before digest calculation. Current v2 scheduling does not enforce the value against player capabilities.

Generated models

Suite pydantic-model

Bases: BaseModel

One benchmark-suite configuration that defines a comparable set of results.

It records the mission set, per-role interaction protocol, player matchup, required modalities, and revision that define what is measured. A SuiteLockEntry records a frozen revision of this configuration.

digest fingerprints the config and the mission files together, so a change without a revision bump is caught.

Attributes

mission_set property 

mission_set: str

The mission-set name (the missions_path basename), grouping attempts and records.

digest property 

digest: str

A stable digest of the suite config and the current mission files.

The full fingerprint of what the suite measures. Frozen lock entries store it alongside the suite revision.

SuiteMatchup pydantic-model

Bases: BaseModel

The pairing that turns a run's roster into (defuser, expert) games.

Attributes

pairing_type instance-attribute 

pairing_type: PairingType

SuiteIdentity pydantic-model

Bases: BaseModel

The frozen suite the interactive results were measured against.

Attributes

target property 

target: str

What was measured, with its pin, the bundle dir's leaf name.

PlayerProtocol pydantic-model

Bases: BaseModel

Protocol that a player has for some experiment.

Attributes

role instance-attribute 

role: PlayerRole

The role of the player in the experiment.

This also determines what they will have access to.

communication_style instance-attribute 

communication_style: CommunicationStyle

The style of communication the player will use.

Either async (all players communicate at once) or sync (players take turns).

is_playing_alone instance-attribute 

is_playing_alone: bool

Whether the player is playing alone or with others.

include_manual instance-attribute 

include_manual: bool

Whether the manual should be included in the prompt.

receive_feedback_after_action class-attribute instance-attribute 

receive_feedback_after_action: bool = False

Whether or not a player should receive feedback after each action.

allow_magic_actions class-attribute instance-attribute 

allow_magic_actions: bool = False

Whether the player is allowed to perform magic actions.

allow_lottery_actions class-attribute instance-attribute 

allow_lottery_actions: bool = False

Whether the player is allowed to perform lottery actions.

is_solo_player property 

is_solo_player: bool

Whether the player is a solo player.

This is used to determine whether the player is playing alone or with others.

allow_message_output property 

allow_message_output: bool

Whether to allow the players to send messages to each other.

This only makes sense if the player is not playing alone.

Methods:

check_expert_is_not_playing_alone 

check_expert_is_not_playing_alone() -> Self

Return an expert cannot play alone.

It doesn't make sense for them to be alone.

digest covers the materialised mission bodies, manual profile and rule seed, role protocols, matchup, and modalities. It excludes the suite name, revision, configuration path, and freeze provenance. The lock version defines the digest recipe. The exact frozen lock structure is documented with submission bundles and comparability is explained in Suites, revisions, and comparability.