Start

Core concepts

The objects and defaults that connect a dataset, model output, split, evaluator, and result.

A run combines a dataset, a model output, a split, and an evaluator. Results can be returned in memory or saved to the dataset's output directory.

Dataset

BenchmarkDataset

A processed dataframe plus metadata describing tasks, targets, species, split defaults, and compatible evaluation routes.

Model

EmbeddingModel

The adapter used to produce embeddings, likelihood scores, or predicted tracks.

Evaluator

LinearProbe

A simple downstream estimator used to measure what is accessible in a frozen representation.

Analysis

Results and scripts

Persisted metrics and seed summaries keyed by task, target, split, and model.

Dataset metadata defines evaluation defaults

Every registered dataset stores:

its stable identifier and species;
one or more task and target columns;
a default split strategy;
whether it belongs to the core or extended benchmark;
supported linear-probe and variant-effect routes;
an optional variant region and VEP target column.

Calling LinearProbeBuilder(dataset).build() uses the first task and target, the metadata split, and the evaluator matching that task. Explicit builder methods override those defaults.

Evaluation routes

Dataset metadata stores these evaluation route keys inevaluations. embedding_vep andlikelihood_vep are the two variant-effect prediction (VEP) routes.

Route	What is evaluated	Typical output
`linear_probe`	A simple estimator fitted on frozen sequence embeddings or paired embedding differences.	Regression, classification, or multilabel metrics.
`embedding_vep`	A scalar function of the pooled alternate-minus-reference embedding difference.	The default L2 score is an unsigned zero-shot magnitude. Signed regression uses a supervised delta-embedding probe or a justified custom `scoring_fn`.
`likelihood_vep`	The difference between normalized reference and alternate log-likelihoods.	Reference minus alternate for classification; alternate minus reference for regression.

Paired variant datasets implement get_vep_pairs() to align each alternate sequence with its reference sequence and optional tracks.

Frozen probing and fine-tuning answer different questions

A linear probe asks whether the representation already organizes useful task information. Fine-tuning asks how well the model can adapt when its parameters or added low-rank adaptation (LoRA) weights are trained for that task. The paper's central model comparison uses frozen representations.