Skip to content

Completion and provenance API

These package APIs define experiment completion and benchmark provenance. A completion source can change where status is read, but not the rule for a valid terminal outcome.

Completion ledgers

CompletionLedger

Bases: Protocol

Answers 'which of these experiments are already done?' for a set of attempt names.

Methods:

status_for 

status_for(attempt_names: Iterable[str]) -> dict[str, ExperimentStatus]

Map every attempt name to its status.

completed 

completed(attempt_names: Iterable[str]) -> set[str]

Return the subset of attempt names that are done and valid (safe to skip re-running).

Implementations MAY prune stale state while answering (the W&B ledger marks invalid remote runs old so a run left in a bad state doesn't block a re-run); the local ledger is a pure read. Both decide "done & valid" through the same is_valid_outcome, so the answer itself never depends on the source.

LocalLedger dataclass

Bases: CompletionLedger

Completion ledger backed by the recorder's on-disk parquet outputs.

Reads completion straight from the recorded outputs, the same attempt_name key W&B uses, so a local lookup is a drop-in for the W&B query. Does not need a DB build: it groups the experiment-*.parquet files by the attempt name in their footer (canonical, independent of the filename) and reuses the DB layer's footer-based validity check.

Methods:

status_for 

status_for(attempt_names: Iterable[str]) -> dict[str, ExperimentStatus]

Map each attempt name to done/failed/not_attempted from disk.

completed 

completed(attempt_names: Iterable[str]) -> set[str]

Return the attempt names with a valid, completed set of outputs on disk.

Source

Bases: StrEnum

Where to read experiment-completion truth from.

local (the default) reads the on-disk experiment outputs. wandb is a cross-machine aggregator and requires WANDB_ENTITY/WANDB_PROJECT in the environment and the wandb extra installed.

Attributes

local class-attribute instance-attribute 

local = 'local'

wandb class-attribute instance-attribute 

wandb = 'wandb'

resolve_ledger 

resolve_ledger(source: Source, *, output_dir: Path) -> CompletionLedger

Build the ledger for the chosen source (local by default).

filter_experiments 

filter_experiments(specs: list[ExperimentSpec], *, source: Source, output_dir: Path) -> list[ExperimentSpec]

Drop the specs that are already done, per the chosen completion source.

The entry point used by both submit and the run/doctor resume check, so the two can never disagree about what counts as 'already done'.

ExperimentStatus is done, failed, not_attempted, or running. The local ledger does not report running by itself. The CLI status command adds that state from the experiment manager.

Provenance and integrity

Provenance pydantic-model

Bases: BaseModel

Release provenance captured when a record is created.

capture reads the release tag and release commit from the checkout at that point. Stored records must contain every field and are never completed from the checkout that later reads them.

Methods:

capture classmethod 

capture(repository: Path = _MODULE_DIR, *, force: bool = False) -> Provenance

Capture release provenance, or record its absence for a forced execution.

BenchmarkIntegrityError

Bases: RuntimeError

The repository cannot produce a benchmark integrity result.

check_benchmark_integrity 

check_benchmark_integrity(repository: Path) -> _BenchmarkIntegrityResult

Compare protected checkout content with its reachable release baseline.

Select the highest semantic annotated release reachable from HEAD. Build the v1 protected tree from that commit and from the checkout, then report their identities and changed paths. Raise BenchmarkIntegrityError when the repository or release baseline is unavailable.

gptnt_version cached 

gptnt_version() -> str

Resolved gptnt version, e.g. 0.13.2 or 0.13.2.dev3+g<sha> between releases.

git_sha cached 

git_sha() -> str | None

Current commit for the checkout containing the installed package, if available.

is_valid_version 

is_valid_version(recorded: str | None) -> bool

Return whether a recorded version is resolvable.

A valid version must parse as a version (PEP 440/SemVer) AND not be the UNKNOWN_VERSION fallback we stamp when the package metadata can't be resolved.

Stored provenance is complete at record creation. A later reader never fills missing values from its checkout. Benchmark integrity requires one exact annotated vMAJOR.MINOR.PATCH tag on the release commit; an absent or ambiguous tag raises BenchmarkIntegrityError.