Run

Results and analysis

Find persisted outputs, query run metadata, combine seeds, and interpret each metric.

Storage layout

text

<data_path>/
  <dataset-name>/
    data_df.parquet
    raw_data/
    embeddings/
      <dataset>_<model-short-name>.npz
    results.db
    results.db.lock
    ft_results/  # created when FineTunePersister is used
      result_ft_....json

Pooled embeddings use compressed NPZ files with anembedding key. Ragged token embeddings use an HDF5embeddings group. Floating tensors with another dtype are converted to float32 before they are saved.

Query linear-probe results

Load all persisted rows through LinearProbePersister for normal analysis:

python

import mrna_bench as mb

from mrna_bench.linear_probe.persister import (
    LinearProbePersister,
)

dataset = mb.load_dataset("go-mf")
results = LinearProbePersister.load_all_results(
    dataset.dataset_path
)
for result in results[:3]:
    print(result)

Each results.db row records the model, task, target, split, seed, and metrics. Rerunning the same model, task, target, split, and seed replaces that row.

Use SQL for targeted queries

python

import json
import sqlite3
import mrna_bench as mb

dataset = mb.load_dataset("go-mf")
connection = sqlite3.connect(
    f"{dataset.dataset_path}/results.db"
)
connection.row_factory = sqlite3.Row

rows = connection.execute(
    """
    SELECT model, task, target_col, split_type, seed, metrics
    FROM lp_results
    WHERE model = ? AND target_col = ?
    ORDER BY task, seed
    """,
    ("orthrus-large-6", "target"),
).fetchall()
connection.close()

results = [
    {
        **dict(row),
        "metrics": json.loads(row["metrics"]),
    }
    for row in rows
]

Find incomplete seeded runs

python

from collections import defaultdict

expected_seeds = {2541, 413, 411, 412, 2547}
seeds_by_run = defaultdict(set)

for result in results:
    seed = str(result["seed"])
    if not seed.isdecimal():
        continue
    key = (
        result["model"],
        result["task"],
        result["target_col"],
        result["split_type"],
    )
    seeds_by_run[key].add(int(seed))

incomplete_runs = {
    key: sorted(expected_seeds - observed)
    for key, observed in seeds_by_run.items()
    if expected_seeds - observed
}
print(incomplete_runs)

Combine completed seeds

python

# probe is the LinearProbe built in the benchmarking guide.
seed_metrics = probe.linear_probe_multirun(
    random_seeds=[2541, 413, 411, 412, 2547],
    persist=True,
)
summary = probe.compute_multirun_results(
    seed_metrics,
    print_output=True,
    persist=True,
)
print(summary)

The summary writes an all seed row containing the mean and a normal-approximation interval:mean ± 1.96 * np.std(values, ddof=0) / sqrt(n).

Metric reference

Task	Metrics written by the package
Regression	MSE, Pearson r, Spearman rho (stored under the legacy key `p`)
Binary classification	MCC, balanced accuracy, AUROC, AUPRC
Multiclass	MCC, balanced accuracy, accuracy, macro F1, micro/macro AUROC and AUPRC
Multilabel	Micro MCC and micro/macro AUROC and AUPRC
Zero-shot variant-effect classification	AUROC and AUPRC

Inspect embedding inventory

python

from pathlib import Path
import mrna_bench as mb

root = Path(mb.get_data_path())
for embedding in sorted(root.glob("*/embeddings*/*")):
    if embedding.suffix in {".npz", ".h5"}:
        size_mb = embedding.stat().st_size / 1024**2
        print(f"{embedding.relative_to(root)}\t{size_mb:.1f} MB")

Use this before scheduling probes to check which model and dataset combinations are already present and how much storage they use.