Skip to content

Actions and observations

Player output models represent the choices a language model can return. Before each model call, GPTNT prepares game frames and converts a selected location into the game input contract.

Player output actions

Type Aliases

GameInteractionActionType 

GameInteractionActionType = MagicGameAction | LotteryGameAction | InteractGameAction[InteractableLocation]

Action types representing only game interaction actions.

PlayerOutputType 

PlayerOutputType = DoNothingAction | SendMessageAction | InteractGameAction[InteractableLocation] | MagicGameAction | LotteryGameAction

Any possible output from a player.

Classes

DoNothingAction pydantic-model

Bases: ModelOutputDumpsMixin

Create a 'do nothing' action.

SendMessageAction pydantic-model

Bases: ModelOutputDumpsMixin

Create a 'send message' action.

Attributes
message instance-attribute 
message: str

InteractGameAction pydantic-model

Bases: KtaneBaseAction[GameActionType, LocationDataT_co], ModelOutputDumpsMixin, Generic[LocationDataT_co]

Interaction action for the player to take in the game.

MagicGameAction pydantic-model

Bases: KtaneBaseAction[Literal['magic'], InteractableLocation], ModelOutputDumpsMixin

Magic action for the player to take in the game.

LotteryGameAction pydantic-model

Bases: KtaneBaseAction[Literal['lottery'], InteractableLocation], ModelOutputDumpsMixin

Lottery action for the player to take in the game.

PlayerOutputType is the complete supported model-output union. GameInteractionActionType excludes messaging and no-op output.

Interaction locations

Type Aliases

SingleAlphabetLetter 

SingleAlphabetLetter = str

SetOfMarksLocation 

SetOfMarksLocation = NonNegativeInt | SingleAlphabetLetter

Set of marks location to interact with; must be an int >= 0, or one letter A-Z.

InteractionLocationMethod 

InteractionLocationMethod = Literal['set-of-marks', 'coordinates']

Whether interaction locations are predicted as set-of-marks or coordinates.

CoordinateMode 

CoordinateMode = Literal['absolute', 'normalised']

The flavour of coordinates that the model supports.

The normalised scale is configured by PlayerCapabilities.coordinate_scale; absolute coordinates are pixel values based on the image dimensions.

Classes

PixelLocation pydantic-model

Bases: BaseModel

Absolute pixel coordinate to interact with in the game.

Attributes
x instance-attribute 
x: NonNegativeInt

Absolute x-coordinate from the left.

y instance-attribute 
y: NonNegativeInt

Absolute y-coordinate from the top.

ScaledLocation pydantic-model

Bases: BaseModel

Normalised coordinate to interact with in the game.

Attributes
x instance-attribute 
x: NonNegativeInt

Normalised x-coordinate from the left.

y instance-attribute 
y: NonNegativeInt

Normalised y-coordinate from the top.

Set-of-marks locations are a non-negative mark number or one letter. Coordinate output is either absolute pixels or a normalised integer scale selected by player capabilities.

Game inputs

Attributes

KtaneGameplayInput module-attribute 

KtaneGameplayInput = KtaneBaseAction[GameActionTypeWithExtras, RelativeCoordinate]

Everything the game actually accepts as input for gameplay actions.

Classes

GameActionType

Bases: Enum

Actions that can be performed in the game.

Attributes
rotate_left class-attribute instance-attribute 
rotate_left = 'left'

Only 90deg rotations are allowed.

rotate_right class-attribute instance-attribute 
rotate_right = 'right'

Only 90deg rotations are allowed.

flip class-attribute instance-attribute 
flip = 'flip'

Rotate the bomb 180 degrees.

roll_up class-attribute instance-attribute 
roll_up = 'up'

Roll the bomb up 90 degrees.

roll_down class-attribute instance-attribute 
roll_down = 'down'

Roll the bomb down 90 degrees.

zoom_out class-attribute instance-attribute 
zoom_out = 'out'

Zoom out of the current depth (i.e. right-clicking).

click_release class-attribute instance-attribute 
click_release = 'click'

Click (and immediate release) on a point.

hold class-attribute instance-attribute 
hold = 'hold'

Hold (and do not release) on a point.

release class-attribute instance-attribute 
release = 'release'

Release the hold (does not use a location).

Methods:
require_location classmethod 
require_location() -> set[GameActionType]

Return the set of actions that require a location to interact on.

RelativeCoordinate pydantic-model

Bases: BaseModel

Coordinates for location-based actions.

The top-left of the screen is (0, 0) and the bottom-right is (1, 1).

Attributes
x_pos instance-attribute 
x_pos: float

Relative x-coordinate from the left.

y_pos instance-attribute 
y_pos: float

Relative y-coordinate from the top.

KtaneBaseAction pydantic-model

Bases: BaseModel, Generic[KtaneActionT, LocationDataT_co]

Interaction action for the player to take in the game.

Attributes
action instance-attribute 
action: KtaneActionT
location class-attribute instance-attribute 
location: LocationDataT_co | None = None

Location to interact with, if needed.

is_clicking_action property 
is_clicking_action: bool

Check if the action is a clicking action.

Methods:
check_actions_align_with_location_use 
check_actions_align_with_location_use() -> Self

Some actions require a location, so reject mismatched location data.

try_validate_action_by_name classmethod 
try_validate_action_by_name(action: Any) -> GameActionType | Any

Validate the action using the GameActionType enum name.

to_query_params 
to_query_params() -> QueryParams

Convert the action to query parameters for the API.

Observations and conversion

Classes

Observation pydantic-model

Bases: BaseModel

Observation from the game.

Attributes
frames instance-attribute 
frames: list[PNGBytes]

Ordered PNG game frames before the final frame is replaced by its set-of-marks version.

segm_mask instance-attribute 
segm_mask: PNGBytes | None

PNG segmentation mask aligned with the final game frame.

som_image instance-attribute 
som_image: PNGBytes

Final frame after optional set-of-marks processing and resizing, sent to the defuser model.

ObservationHandler dataclass

Handle observations from the game client.

This deals with set of marks, relative coordinates, and all of that to ensure that it is coming and going in the right format.

Attributes
interaction_location_method instance-attribute 
interaction_location_method: InteractionLocationMethod
coordinate_scale class-attribute instance-attribute 
coordinate_scale: int | None = None
image_resizer class-attribute instance-attribute 
image_resizer: ImageResizer | None = None
set_of_marks_painter class-attribute instance-attribute 
set_of_marks_painter: SetOfMarksHandler | None = None
Methods:
reset 
reset() -> None

Reset the observation handler.

This is called when the player is reset, and it should reset any internal state.

handle_new_observation 
handle_new_observation(*, frame_buffer: FrameBuffer, bomb_state: BombState, num_frames_to_use: int = 1) -> Observation

Handle a new observation from the game.

convert_to_game_action 
convert_to_game_action(*, action: GameInteractionActionType) -> KtaneGameplayInput

Convert the action to the game action.

An Observation carries recent PNG frames, an optional segmentation mask, and the processed image used for set-of-marks output. ObservationHandler applies the configured resizing and location method, then converts interaction output to KtaneGameplayInput.

The runtime does not compare suite modality declarations with player capabilities. Validate the chosen model's image and interaction support before a benchmark run.