Run static evaluations¶

Run a configured player against a static dataset and write predictions, metrics, dataset identity, capabilities, and benchmark provenance without starting KTANE.

Before you start¶

Complete the player and provider checks described in Add a model. Choose a task from gptnt statics --help.

Static evaluations are optional submission targets. Run one when you need its dataset-specific measurement or want to prepare a static bundle.

Preview a task¶

Every dataset-backed command requires --player. --provider selects an optional provider override.

Run in your shell

gptnt statics expert-vqa-no-manual --player <player-name>

Without --throw, the command constructs the evaluation but does not execute predictions. Add a small limit while checking a player:

Run in your shell

gptnt statics expert-vqa-no-manual \
  --player <player-name> \
  --limit-instances 3 \
  --throw

The option is --player

Static commands select configs/player/<name>.yaml with --player. They do not accept the older --model spelling.

Pin and run the dataset¶

Supply a Hugging Face branch, tag, or commit with --dataset-revision. GPTNT resolves the request to a concrete dataset commit before the first prediction and records both values.

Run in your shell

gptnt statics expert-vqa-no-manual \
  --player <player-name> \
  --dataset-revision <commit-or-tag> \
  --throw

The output directory is output/<task>_predictions/<provider-model>/. Before the first prediction, GPTNT writes run_meta.json. Each completed instance writes prediction_<index>.json; scoring writes metrics.json.

If a prediction file already exists, a resumed run skips that index. The stored run date and resolved dataset commit remain bound to the output set. GPTNT rejects a resume when existing predictions have no run_meta.json, or when the stored metadata conflicts with the current player, capabilities, task, dataset, or provenance.

Keep dataset and benchmark identity with the output

A requested tag or branch can move. Comparability uses the resolved dataset commit recorded in run_meta.json. Running with --force when release provenance cannot be verified omits that provenance from the static output and prevents submission.

Check the output¶

Confirm that the directory contains run_meta.json, the expected prediction indices, and metrics.json. The submission builder copies the metrics and reads the metadata; it does not infer identity from the directory name.

See Statics command reference for every task and option, and Static evaluation outputs for the stored fields.