Suites, revisions, and comparability¶

A result is comparable only when the inputs that define the measurement agree. The GPTNT package version alone does not establish that agreement.

Suite identity¶

A suite revision is selected by its name@revision, such as multi-self-sync@2. The revision is the human-managed boundary for comparable results. Increase it whenever measured content changes.

suite_digest identifies the measured content for one materialised suite: mission bodies, manual_profile, manual_rule_seed, role protocols, matchup, and modalities. It excludes the suite name, revision, configuration path, and freeze provenance. The lock version defines both the TOML layout and this digest recipe.

flowchart LR
    C["Suite configuration"] --> F["suite freeze"]
    M["Materialised mission bodies"] --> F
    F --> R["suite registry: suites.lock"]
    R --> S["Frozen suite revision\nname@revision + suite_digest"]
    S --> E["Experiment specification"]
    E --> X["Comparable result identity"]
    P["Player capabilities and fingerprint"] --> X
    B["Release tag, commit, and protected state"] --> X

gptnt suite freeze records each frozen suite revision in configs/suites/suites.lock. It rejects measured content whose digest differs at an existing name and revision. Generation reads the frozen entry rather than the current suite YAML and copies its revision and digest into every specification.

The suite's modality set participates in suite_digest. Current v2 execution does not enforce that set against player capabilities, so successful scheduling does not prove modality compatibility.

Player and protocol identity¶

Specifications store player names and role protocols. Runtime records add the resolved PlayerCapabilities for each role. Submission groups a Defuser by the capability fingerprint, not only by its display name. The same model name with different reasoning, output, coordinate, image, or usage-limit capabilities belongs to a different group.

The experiment fingerprint identifies a frozen mission, suite, manual profile, and role protocols. It excludes the attempt number and player names. Those excluded values still matter when deciding which execution and attribution a row represents.

Manuals, missions, and datasets¶

The suite registry stores each materialised mission body once by content digest. A frozen suite revision refers to a body by digest and retains its readable mission_key. An interactive bundle contains a bundle suite snapshot: one frozen suite revision and exactly the mission bodies it references.

The manual profile is also copied into each specification and result. Manual preparation and source provenance are explained in Manuals.

Static identity uses the task name, Hugging Face repository and split, requested revision, and resolved commit. Only the resolved commit pins a dataset. An unavailable commit is recorded as unpinned and produces a validation warning.

Benchmark provenance and submission validation¶

Every recorded output includes the installed GPTNT version, exact annotated release tag, release commit, release protected-content digest, checkout protected-content digest, and protected_content_modified state. The digest covers protected source, prompts, base configuration, and manual inputs. It is independent of Git branch names, HEAD being exactly at the tag, the index, ignore rules, and permitted runner inputs such as player, suite, mission, and run files.

The release tag selects the baseline commit. Both digests must be equal for submission, and protected_content_modified must equal their comparison. This permits commits after the release and local changes outside the protected paths without treating the benchmark as changed. The installed GPTNT version remains descriptive; validation does not require the validator's installed version to equal it.

Submission schema 4 binds these fields to protected-content digest policy v1. Its root set, exclusions, path encoding, content normalization, and serialization remain fixed so a later validator can recompute a schema 4 release digest. The policy normalizes CRLF in UTF-8 files, preserves other file bytes, and excludes executable bits because checkout modes differ across operating systems. A future policy change requires a separate versioned policy and submission-schema transition. It must not alter v1.

Local gptnt submission validate checks that a bundle is internally consistent. Submissions CI can recompute protected content from the bundle's declared release and compare it with the recorded release digest. It then requires an exact release-lock match: the bundle suite snapshot must equal the matching snapshot in that release's suite registry. A changed digest or changed freeze provenance fails this check. Rebuild the bundle from the declared release. Do not edit suite.lock.

Every frozen suite revision in a verified published GPTNT release is eligible for leaderboard submission. There is no separate acceptance catalog.

The current submission workflow requires multi-self-async and multi-self-sync. Static targets are optional and retain their own dataset identity.