Skip to content

GPTNT Documentation

Static evaluation commands

Static evaluation commands¶

gptnt statics runs fixed evaluation tasks against Hugging Face datasets. Dataset-backed tasks share one option set and always select a local player profile with --player.

Dataset-backed tasks¶

Command	Task-specific input
`defuser-grounding-coordinates`	Dataset split `test_coordinates`.
`defuser-grounding-som`	Dataset split `test_som`.
`defuser-vqa-oe`	Open-ended VQA.
`defuser-vqa-mcq`	Multiple-choice VQA.
`defuser-state-recognition-vqa-mcq`	`--state-split`: `state-change`, `solved`, or `strikes`. Defaults to `state-change`.
`expert-vqa`	Expert VQA with the configured manual input.
`expert-vqa-no-manual`	Expert VQA without manual input.
`expert-ocr`	Expert element OCR.
`expert-ocr-with-text`	Requires `--manual-artifact PATH`.
`expert-element-grounding`	Expert manual-element grounding.

The common form is:

Command syntax

gptnt statics TASK --player NAME [--provider NAME] [--download] [--throw]
                    [--upload] [--limit-instances N]
                    [--dataset-revision REF]
                    [--allow-thinking | --no-thinking]
                    [--force]

Option	Default and effect
`--player`	Required name under `configs/player/`.
`--provider`	Optional name under `configs/player/provider/`.
`--download`	Downloads the dataset before the run for diagnostic use.
`--throw`	Executes predictions and local scoring. Disabled by default.
`--upload`	Uploads an evaluation to Weave. Disabled by default.
`--limit-instances`	Restricts the loaded instances.
`--dataset-revision`	Requests a Hugging Face branch, tag, or commit.
`--allow-thinking`, `--no-thinking`	Enables thinking by default for dataset tasks.
`--force`	Continues when release provenance cannot be verified. Forced outputs cannot be submitted.

`how-do-you`¶

Command syntax

gptnt statics how-do-you --player NAME [--provider NAME]
                         [--attempts N] [--allow-thinking | --no-thinking]
                         [--output-file-prefix PREFIX]

--attempts is a positive count and defaults to 1. Thinking is disabled by default on this command. It writes a module-to-attempt mapping under output/how_do_you/; it has no dataset, download, throw, upload, or benchmark-integrity option.