Skip to content

Player interfaces

The specification models define recorded player identity and behaviour. Prediction interfaces connect a configured Pydantic AI agent to the player loop.

Specifications

Classes

PlayerIdentity pydantic-model

Bases: BaseModel

Presentation metadata for a model/player.

Attributes
display_name instance-attribute 
display_name: str

Name used on the leaderboard.

organisation instance-attribute 
organisation: str

The organisation that develops the player.

is_os_model instance-attribute 
is_os_model: bool

Whether the player is an open-source model.

url instance-attribute 
url: str | None

Link to the player's model page, or None if there is none.

PlayerCapabilities pydantic-model

Bases: BaseModel

The capabilities of a player, that is set once on instantiation.

This tells the EM what the player is and what they can do for the matchmaking.

Attributes
player_name instance-attribute 
player_name: str

The name of the player.

player_type instance-attribute 
player_type: PlayerType

The type of player (AI or human).

thinking_method class-attribute instance-attribute 
thinking_method: ThinkingMethod = 'inner-monologue'

The thinking method of the player.

structured_output_mode class-attribute instance-attribute 
structured_output_mode: StructuredOutputMode | None = 'prompted'

Which structured output mode to use, as per pydantic-ai.

include_schema_in_instructions class-attribute instance-attribute 
include_schema_in_instructions: bool = True

Should we manually include the output schema in the instructions.

max_observations_per_request class-attribute instance-attribute 
max_observations_per_request: int = 16

The maximum number of observations per request for the player.

We default this to 16.

usage_limits class-attribute instance-attribute 
usage_limits: UsageLimits = Field(default_factory=UsageLimits)

Pydantic AI request limits, including the input-token limit used for history truncation.

image_dimensions class-attribute instance-attribute 
image_dimensions: ImageDimensions = Field(default=KtaneSettings().image_dimensions)

Width and height for the player's input images.

Default to KTANE settings.

tokens_per_image class-attribute instance-attribute 
tokens_per_image: int = Field(default=0, ge=0)

The number of tokens used to represent one image at image_dimensions.

Truncation counts each image as this many tokens when sizing the prompt. Left at zero, images add nothing to that estimate, so a vision player's prompt can be sized under its real length.

interaction_location_method class-attribute instance-attribute 
interaction_location_method: InteractionLocationMethod = 'set-of-marks'

Whether interaction locations are predicted as set-of-marks or coordinates.

coordinate_mode class-attribute instance-attribute 
coordinate_mode: CoordinateMode = 'absolute'

The flavour of coordinates that the model supports.

Normalised coordinates use coordinate_scale as their per-axis denominator, while absolute coordinates are pixel values based on the image dimensions.

coordinate_scale class-attribute instance-attribute 
coordinate_scale: PositiveInt | None = None

Per-axis denominator for normalised coordinates, otherwise None.

Every normalised model declares its native scale explicitly. Absolute coordinates have no scale, so combining coordinate_mode="absolute" with a value here is invalid.

preserve_last_frame_for_n_turns class-attribute instance-attribute 
preserve_last_frame_for_n_turns: int = 0

Number of previous turns from which to keep the last frame in the observation window.

enable_nobf_generation class-attribute instance-attribute 
enable_nobf_generation: bool = True

Whether to generate Naughty Output Behaviour Feedback for each action.

model_settings class-attribute instance-attribute 
model_settings: dict[str, JsonValue] | None = Field(default=None, exclude_if=lambda settings: settings is None)

Declared model settings included in recorded benchmark identity.

normalised_coordinate_scale property 
normalised_coordinate_scale: PositiveInt

Return the validated scale for normalised-coordinate code paths.

interact_location_type property 
interact_location_type: type[SingleAlphabetLetter] | type[PixelLocation] | type[ScaledLocation]

The type used for interaction locations.

This is based on the interaction location method so that we can create the output type for the protocol without needing to have a whole different if-statement.

fingerprint property 
fingerprint: str

A stable digest of this exact model setup.

We exclude several fields to ensure that the fingerprint of the model is only represented by its capabilities and not by any other fields that may change over time, or that are not relevant.

Methods:
validate_no_duplicate_schema_inclusion 
validate_no_duplicate_schema_inclusion() -> Self

Ensure the schema only appears at maximum once.

validate_coordinate_scale_matches_mode 
validate_coordinate_scale_matches_mode() -> Self

Require a scale exactly when the coordinate mode is normalised.

validate_thinking_mode_output_compatibility 
validate_thinking_mode_output_compatibility() -> Self

If the thinking mode is out-loud, ensure that structured outputs are not used.

PlayerProtocol pydantic-model

Bases: BaseModel

Protocol that a player has for some experiment.

Attributes
role instance-attribute 
role: PlayerRole

The role of the player in the experiment.

This also determines what they will have access to.

communication_style instance-attribute 
communication_style: CommunicationStyle

The style of communication the player will use.

Either async (all players communicate at once) or sync (players take turns).

is_playing_alone instance-attribute 
is_playing_alone: bool

Whether the player is playing alone or with others.

include_manual instance-attribute 
include_manual: bool

Whether the manual should be included in the prompt.

receive_feedback_after_action class-attribute instance-attribute 
receive_feedback_after_action: bool = False

Whether or not a player should receive feedback after each action.

allow_magic_actions class-attribute instance-attribute 
allow_magic_actions: bool = False

Whether the player is allowed to perform magic actions.

allow_lottery_actions class-attribute instance-attribute 
allow_lottery_actions: bool = False

Whether the player is allowed to perform lottery actions.

is_solo_player property 
is_solo_player: bool

Whether the player is a solo player.

This is used to determine whether the player is playing alone or with others.

allow_message_output property 
allow_message_output: bool

Whether to allow the players to send messages to each other.

This only makes sense if the player is not playing alone.

Methods:
check_expert_is_not_playing_alone 
check_expert_is_not_playing_alone() -> Self

Return an expert cannot play alone.

It doesn't make sense for them to be alone.

PlayerSpec pydantic-model

Bases: BaseModel

One player in a roster, with a player config, an optional provider override, and a count.

Attributes
player instance-attribute 
player: str

A configs/player/<player>.yaml config name.

provider class-attribute instance-attribute 
provider: str | None = None

A configs/player/provider/<provider>.yaml config name, or None to use the default.

count class-attribute instance-attribute 
count: int = Field(default=1, ge=1)

How many copies of this player to spawn.

PlayerIdentity is submission attribution. PlayerCapabilities affects prompts, parsing, image handling, and the participant fingerprint. PlayerProtocol assigns a role and communication contract for an experiment. PlayerSpec is the resolved process specification consumed by a player service.

Prediction

Classes

ActionPredictor dataclass

Predict actions to perform using AI agents/models to do so.

Attributes
agent instance-attribute 
agent: Agent[PlayerDeps | None, PlayerOutputType | str]

The PydanticAI agent that the AI player uses.

capabilities instance-attribute 
capabilities: PlayerCapabilities
reasoning_parser class-attribute instance-attribute 
reasoning_parser: ReasoningParser[Any, Any] = field(init=False, repr=False)
protocol class-attribute instance-attribute 
protocol: PlayerProtocol = field(init=False, repr=False)
conversation class-attribute instance-attribute 
conversation: Conversation = field(init=False, repr=False)
exception_recovery class-attribute instance-attribute 
exception_recovery: ExceptionRecoveryChain = field(default_factory=ExceptionRecoveryChain.with_default_strategies, repr=False)
reflection_exception_recovery class-attribute instance-attribute 
reflection_exception_recovery: ExceptionRecoveryChain = field(default_factory=ExceptionRecoveryChain.with_reflection_recovery, repr=False)
model_name property 
model_name: str

Get the name of the model.

Methods:
configure_for_experiment 
configure_for_experiment(*, protocol: PlayerProtocol, conversation: Conversation) -> None

Setup the agent for the current experiment.

send_request_to_agent async 
send_request_to_agent(*, message_input: AgentMessageInput) -> AgentCallResult[PlayerOutputType]

Send a message to the AI.

This will be the main way to send messages to the AI.

send_reflection_request async 
send_reflection_request(*, reflection_message: str, max_tokens_override: int = 5000) -> AgentCallResult[SendMessageAction]

Send a reflection message to the agent.

Importantly, we do not care if the reflection fails, we just want to log it and move on.

This request uses a larger output-token limit because reasoning models can exceed the regular limit.

For handling structured outputs

If the player supports structured output, then we give it the chance to use the structured output for send message directly, otherwise we just let it be a string and hope for the best with parsing it.

ActionPredictor makes the configured agent call and returns a GPTNT result wrapper. Provider model objects, model settings, usage limits, and output modes follow the Pydantic AI API.

Call results

Classes

AgentCallResult pydantic-model

Bases: BaseModel, Generic[ModelOutputT_co]

Result of an agent call.

Attributes
output instance-attribute 
output: ModelOutputT_co
thoughts instance-attribute 
thoughts: str | None
usage instance-attribute 
usage: RunUsage
new_messages instance-attribute 
new_messages: list[ModelMessage]

Tool-free request and response messages ending in the call's final model response.

ai_response_error class-attribute instance-attribute 
ai_response_error: list[AIResponseErrorType] = Field(default_factory=list)

Response-error classifications retained after parsing or recovery.

raw_output class-attribute instance-attribute 
raw_output: str | None = None
Methods:
check_no_tools_in_messages classmethod 
check_no_tools_in_messages(messages: list[ModelMessage]) -> list[ModelMessage]

Ensure there are no tool parts in the new messages.

We do this just to make life easier right now. But that also means we are double-ing down on "no using function tools to play the game" aspect of the benchmark.

Also Pydantic says to use ValueError and not TypeError, hence the noqa.

check_final_message_is_model_response classmethod 
check_final_message_is_model_response(messages: list[ModelMessage]) -> list[ModelMessage]

Ensure the final message is a ModelResponse.

DispatchedAgentCallResult pydantic-model

Bases: AgentCallResult[ModelOutputT_co], Generic[ModelOutputT_co]

Agent result stamped when dispatch of its output begins.

Attributes
dispatched_at instance-attribute 
dispatched_at: Instant

Instant when output dispatch starts and the origin for the recorded step timestamp.

Methods:
from_agent_call classmethod 
from_agent_call(agent_call_result: AgentCallResult[ModelOutputT_co]) -> Self

Copy an agent result and stamp it with the current time.

Since we are copying an existing AgentCallResult, we use model_construct to avoid re-validating the object.

AgentCallResult pairs parsed model output with response metadata. The dispatched variant also stores the action dispatch result.

For configuration choices, use Add a model. For the relationship between these models, use Roles, protocols, and capabilities.