Skip to content

Experiment generation API

These objects form the supported Python boundary for constructing suite identities, mission and player pairings, experiment specifications, and frozen suite snapshots. CLI workflows should use generate-missions, suite freeze, and generate instead.

Suites and pairings

Suite pydantic-model

Bases: BaseModel

One benchmark-suite configuration that defines a comparable set of results.

It records the mission set, per-role interaction protocol, player matchup, required modalities, and revision that define what is measured. A SuiteLockEntry records a frozen revision of this configuration.

digest fingerprints the config and the mission files together, so a change without a revision bump is caught.

Attributes

digest property 

digest: str

A stable digest of the suite config and the current mission files.

The full fingerprint of what the suite measures. Frozen lock entries store it alongside the suite revision.

mission_set property 

mission_set: str

The mission-set name (the missions_path basename), grouping attempts and records.

Methods:

digest_for 

digest_for(missions: Sequence[KtaneMissionSpec]) -> str

Return the suite digest for an explicit mission snapshot.

SuiteMatchup pydantic-model

Bases: BaseModel

The pairing that turns a run's roster into (defuser, expert) games.

Attributes

pairing_type instance-attribute 

pairing_type: PairingType

SuiteIdentity pydantic-model

Bases: BaseModel

The frozen suite the interactive results were measured against.

Attributes

target property 

target: str

What was measured, with its pin, the bundle dir's leaf name.

Pairing

Bases: NamedTuple

Pairing for a game.

Attributes

defuser instance-attribute 

defuser: str

expert instance-attribute 

expert: str | None

PairingGenerator

Generate all the possible pairings for a game.

This covers the different types of experiments we are going to run. Note that if you want to generate with_best_defuser or with_best_expert, then you need to give the best_model.

Methods:

generate 

generate() -> Iterator[Pairing]

Generate all the pairings.

Missions and specifications

MissionGeneratorConfig pydantic-model

Bases: BaseModel

Config to generate missions for KTANE from.

Attributes

time_limit instance-attribute 

time_limit: NonNegativeInt | None

Time limit for the game, in seconds.

If None, the time limit will be calculated based on the modules.

allow_back_placement instance-attribute 

allow_back_placement: bool

Are any modules allowed to be placed on the back?

n_modules_min class-attribute instance-attribute 

n_modules_min: NonNegativeInt = Field(ge=1)

Inclusive lower bound for the number of components generated in each mission.

n_modules_max class-attribute instance-attribute 

n_modules_max: NonNegativeInt = Field(ge=1)

Inclusive upper bound for the number of components generated in each mission.

sample_from_modules instance-attribute 

sample_from_modules: bool

Selects one sampled component set or one mission per available module for each seed.

allow_repeat_module instance-attribute 

allow_repeat_module: bool

Whether random multi-module sampling selects components with replacement.

min_optional_widgets class-attribute instance-attribute 

min_optional_widgets: NonNegativeInt = Field(ge=1, default=1)

Inclusive lower bound for the randomly generated optional-widget count.

max_optional_widgets class-attribute instance-attribute 

max_optional_widgets: NonNegativeInt = Field(ge=1, default=5)

Inclusive upper bound for the randomly generated optional-widget count.

excluded_modules class-attribute instance-attribute 

excluded_modules: set[KtaneModuleId] = Field(default_factory=lambda: {'NeedyVentGas', 'NeedyCapacitor', 'NeedyKnob'})

available_modules property 

available_modules: list[KtaneModuleId]

Get a list of available modules.

expected_num_missions property 

expected_num_missions: int

Get the expected number of missions.

MissionGenerator

Generate missions for KTANE from the config.

Includes RNGs and seeds for reproducibility.

Methods:

generate 

generate() -> Iterator[KtaneMissionSpec]

Generate fresh mission specs from the configured seeds.

This is the authoring path that materialises a mission set. The run path never generates. It loads the materialised files with load_missions.

ExperimentGenerator pydantic-model

Bases: BaseModel

Generate experiments from the missions and pairings.

Methods:

generate 

generate(missions: Iterator[KtaneMissionSpec], pairings: Iterator[Pairing]) -> Iterator[ExperimentSpec]

Generate all possible experiments to be run from the inputs.

ExperimentSpec pydantic-model

Bases: BaseModel

The mission and player setup for one experiment attempt.

A spec says which frozen suite mission and manual profile to run, which player protocols and model names to use, and which attempt this is. It does not contain runtime service identities, resolved capabilities, or the result.

Attributes

mission_spec instance-attribute 

mission_spec: KtaneMissionSpec

Exact bomb configuration played by this attempt.

mission_set instance-attribute 

mission_set: str

The mission set this came from (the missions_path basename), e.g. single_module.

attempt class-attribute instance-attribute 

attempt: int = 1

Generator-assigned, one-based repeat index for the same mission and player pairing.

suite_name instance-attribute 

suite_name: str

suite_revision instance-attribute 

suite_revision: int

Frozen suite revision that defines comparability for this specification.

suite_digest instance-attribute 

suite_digest: str

Digest of the frozen suite configuration and mission snapshot.

manual_profile instance-attribute 

manual_profile: ManualProfile

Ordered manual profile prepared for this experiment.

defuser_protocol instance-attribute 

defuser_protocol: PlayerProtocol

defuser_name instance-attribute 

defuser_name: str

player_name assigned to the defuser role, not the player configuration name.

expert_protocol instance-attribute 

expert_protocol: PlayerProtocol | None

expert_name instance-attribute 

expert_name: str | None

player_name assigned to the expert role, not the player configuration name.

fingerprint property 

fingerprint: str

Identify experiments that use the same mission, frozen suite, and player protocols.

The attempt number and player names are excluded because changing them does not change what the experiment measures.

is_single_player property 

is_single_player: bool

Check if the experiment is one player experiment.

communication_style property 

communication_style: CommunicationStyle

Get the communication style for the experiment.

pairing property 

pairing: str

Get the names of the pair.

Just to be consistent with the old way of doing it.

suite_version property 

suite_version: str

Get the suite version for the experiment.

experiment_name property 

experiment_name: str

Get the name for the experiment.

attempt_name property 

attempt_name: str

Get the name for the attempt.

some_player_wants_feedback property 

some_player_wants_feedback: bool

Check if any player wants feedback.

Methods:

verify_expert_has_both_spec_and_name 

verify_expert_has_both_spec_and_name() -> Self

Verify that the expert has both a spec and a name if they are set.

verify_no_expert_for_single_player 

verify_no_expert_for_single_player() -> Self

Verify that the expert is None if the defuser is a solo player.

get_player_protocol 

get_player_protocol(role: PlayerRole) -> PlayerProtocol | None

Get the player protocol for the role.

Frozen snapshots

SuiteLock pydantic-model

Bases: BaseModel

A self-contained, append-only snapshot of all the frozen suites.

We store the full suite config and the detailed mission specs in the lock file so that we can reconstruct a suite and its missions without reading the original configs.

Methods:

from_lock_path classmethod 

from_lock_path(path: Path | None = None) -> Self

Load a lock from disk, or raise if the file is missing or malformed.

select_entry 

select_entry(name: str, revision: int | None) -> SuiteLockEntry

Get the requested entry, or the latest revision when revision is None.

load_suite 

load_suite(name: str, revision: int | None = None) -> tuple[Suite, list[KtaneMissionSpec]]

Rebuild a frozen suite and its missions from this lock.

revision defaults to the latest frozen revision of name. Raises SuiteNotFrozenError when the suite or the requested revision is absent.

Note: A solo suite omits its expert. Since TOML has no null, a missing optional reconstructs as None.

snapshot 

snapshot(name: str, revision: int) -> Self

Return the bundle suite snapshot for one frozen suite revision.

The generator assigns attempts starting at one. ExperimentSpec requires expert protocol and expert name to be either both present or both absent, and forbids an Expert for a solo Defuser.