Skip to content

Static evaluation outputs

Dataset-backed static tasks write a model-specific directory with run metadata, one JSON file per prediction, and task-dependent metrics. These formats have no independent format version.

Dataset task layout
output/
  <task>_predictions/
    <provider-model>/
      run_meta.json
      prediction_0.json
      prediction_1.json
      ...
      metrics.json

Run metadata

StaticsIdentity pydantic-model

Bases: BaseModel

What was measured: the task plus its dataset pin (repo, split, revisions).

Attributes

is_pinned property 

is_pinned: bool

Whether the dataset is pinned to a concrete commit (a resolved sha exists).

revision_label property 

revision_label: str

A short label for the dataset revision, always the resolved commit sha.

A HuggingFace dataset is only reproducibly pinned by its commit sha; a requested tag or branch can move, so it never forms the label. With no resolved sha (offline/private repo), return the unpinned mark.

target property 

target: str

What was measured, with its pin, a submission bundle dir's leaf name.

StaticsRunMetadata pydantic-model

Bases: BaseModel

Everything a submission needs beyond the predictions.

Attributes

model_name instance-attribute 

model_name: str

run_date instance-attribute 

run_date: Instant

Output-set start instant bound before the first prediction and preserved on resume.

statics instance-attribute 

statics: StaticsIdentity

capabilities instance-attribute 

capabilities: PlayerCapabilities

provenance instance-attribute 

provenance: Provenance

GPTNT release provenance captured before prediction.

requested_revision is the branch, tag, or commit supplied to the dataset loader. resolved_revision is the concrete dataset commit resolved before prediction. A resumed output set keeps its original run date and resolved commit. Prediction files without metadata cannot be resumed because current provenance cannot be assigned to earlier calls.

Prediction files

Each prediction_<index>.json adds the dataset row index to ModelOutput:

ModelOutput

Bases: TypedDict

Output of the model predict.

Attributes

usage instance-attribute 

usage: dict[str, int]

Non-zero Pydantic AI token-usage counts flattened for this prediction.

model instance-attribute 

model: str

Provider model name resolved for this prediction.

output instance-attribute 

output: str

Parsed task output before task-specific score normalisation.

scored_output instance-attribute 

scored_output: str

Canonical task answer consumed by scorers.

thoughts instance-attribute 

thoughts: str | None

raw_output instance-attribute 

raw_output: str | None

error instance-attribute 

error: str | None

Model-response validation classifications produced by parsing or recovery.

exception instance-attribute 

exception: Any | None

Structured traceback for an exception that prevented prediction.

output is the parsed task output before score normalisation. scored_output is the canonical task answer used by current scorers. error records response-validation classifications; exception records a structured traceback when prediction did not complete. usage omits zero token counts.

Metrics

metrics.json is a task-dependent dict[str, dict[str, value]] written after local scoring. It is not a cross-task schema. Static submission copies this file verbatim and uses run_meta.json for the player, capabilities, task, dataset, run date, and provenance.

Limit changes the output boundary

--limit-instances writes and scores only the loaded prefix. Do not treat a limited diagnosis run as complete task coverage.

how-do-you

output/how_do_you/<provider-model>.json maps each supported module to a list of attempts. Each attempt contains prompt and response. An optional prefix changes the filename. This command writes no run_meta.json or metrics.json and is not a dataset-backed submission target.