GitHub

Benchmark overview

A benchmark centered on mature-mRNA biology

mRNABench evaluates linearly accessible information in frozen nucleotide-model representations across translation, stability, localization, molecular interactions, transcript lifecycle, and UTR SNV classification.

Public preprintbioRxiv v1

mRNABench: A curated benchmark for mature mRNA property and function prediction

Ruian (Ian) Shi†1,2,3, Taykhoom Dalal†3, Philip Fradkin†1,2, Divya Koyyalagunta3, Simran Chhabria3, Andrew Jung4, Cyrus Tam3, Defne Ceyhan3, Jessica Lin3, Kaitlin U. Laverty2,3, Ilyes Baali3, Bo Wang1,2,5, Quaid Morris3

  1. 1Department of Computer Science, University of Toronto
  2. 2Vector Institute
  3. 3Computational and Systems Biology Program, Sloan Kettering Institute
  4. 4Department of Electrical and Computer Engineering, University of Toronto
  5. 5Peter Munk Cardiac Center, University Health Network

† Equal contributiondoi:10.1101/2025.07.05.662870

Why mRNA

Why mature mRNA needs separate evaluation

Mature mRNA is produced when selected exons are spliced from a pre-mRNA into a complete transcript. Alternative splicing can generate multiple isoforms from one gene, each with a different combination of untranslated regions, coding sequence, and exon junctions, and therefore different regulatory properties.

Those properties determine how long a transcript persists, where it localizes, and how efficiently it is translated. They are central to normal gene regulation, disease, and the design of mRNA therapeutics. A useful representation must therefore capture both short regulatory elements and information distributed across transcripts that often span several kilobases.

DNA and non-coding RNA benchmarks ask different biological questions and often use much shorter sequence windows. mRNABench brings mature-mRNA tasks into one evaluation so that models can be compared on the sequence scales and functions they are meant to represent.

Comparison

What is held consistent across models

01

Biological target

Every task measures a property or function of mature mRNA, including translation, stability, localization, and molecular interactions.

02

Sequence context

Inputs range from short regulatory regions to complete transcripts containing UTRs, coding sequence, and splice context.

03

Task breadth

The same frozen model is evaluated across multiple assays and biological outcomes.

04

Split strategy

Each dataset records a split suited to its rows, including random, k-mer, chromosome, and homology grouping.

Evaluation

Frozen representations and linear probes

Models provide per-token or per-nucleotide representations. When a sequence exceeds a model's context limit, token-level representations from its chunks are concatenated along sequence position and then mean-pooled over the reconstructed sequence.

The model weights remain frozen throughout evaluation. Lightweight linear probes are trained on the same dataset splits, making the comparison a test of what biological information is accessible in each pretrained representation.

UTR SNV tasks compare alternate and reference transcripts. The benchmark can evaluate paired embedding differences or normalized log-likelihood differences for models that support them.

Inputs
Endogenous transcripts or synthetic constructs, with codon-position and splice-site tracks supplied only to compatible adapters.
Representations
Frozen model embeddings, pooled over sequence position.
Classification metric
AUPRC, with micro-averaging for multilabel tasks. Paired SNV scoring also reports AUROC.
Regression metric
Pearson correlation.

AUPRC changes with the frequency of positive labels. Compare models within the same dataset rather than comparing raw AUPRC values across datasets.

Task scale

Local and global mRNA tasks

The paper groups tasks by how the sequence signal giving rise to each label is distributed. Local tasks are dominated by short motifs or targeted perturbations, while global tasks reflect the combined effects of regulatory elements across the transcript. Some assays draw on both scales.

Local tasks

Motifs, UTR perturbations, and binding events

Reporter MRL varies the 5' UTR while keeping the CDS fixed; eCLIP and miRNA tasks reflect short binding motifs; variant tasks alter a single UTR nucleotide.

Global tasks

Properties shaped across the full transcript

RNA half-life, translation efficiency, localization, and transcript lifecycle integrate regulatory effects distributed across UTRs and coding sequence.

Split design

Four ways to separate train and test data

Each dataset uses a default split suited to its rows and labels. The dataset catalog lists that choice.

01

Random

Assigns rows at random. This is used for synthetic constructs and matched variant sets when gene-level grouping does not fit the study design.

02

k-mer

Clusters unnormalized raw k-mer counts before assigning groups to splits, so both motif counts and sequence length affect grouping.

03

Chromosome

Holds out complete chromosomes, while recognizing that homology can still cross chromosomes.

04

Homology

Keeps genes from the same precomputed paralog group in one split.

Findings

What the benchmark reveals

Taken together, the results show that model scale alone does not explain performance. Pretraining objective, sequence domain, and evaluation split each change what a model appears to learn.

Scale is only part of the picture

Model size was only weakly correlated with overall performance. Evo2 led the aggregate comparison, but much smaller models such as Orthrus remained competitive on several tasks, showing that architecture, training data, and pretraining objective matter alongside parameter count.

Joint MLM and contrastive pretraining

Orthrus was strongest on global tasks and weaker on tasks driven by local sequence features. Holding its architecture and pretraining data fixed, we varied the balance between masked language modeling and contrastive learning. The joint objective covered the benchmark more evenly than either objective alone, and the selected Orthrus+MLM model matched or exceeded the strongest foundation model on seven of 11 datasets.

Training data should match the sequence domain

Models pretrained on mature mRNA generally outperformed models trained on genomic DNA or non-coding RNA. The compression analysis shows that coding regions, UTRs, introns, and intergenic DNA have measurably different sequence structure. Together, these results are consistent with a benefit from domain-matched pretraining, although the model groups also differ in architecture and objective.

Split choice changes the result

Random splits produced the most optimistic estimates of generalization. Grouping transcripts by k-mer composition caused the largest performance drops, while homology-based splits produced smaller but systematic gaps. The effect was especially clear for eCLIP and miRNA tasks, where short binding motifs can otherwise appear on both sides of a random split.

Limits

Limits of the benchmark

Linear probes. A high score means that a linear model can recover a label from the frozen representation. This is a useful measure of representation quality, but it does not identify the regulatory mechanism behind a prediction or measure how a model will respond to end-to-end fine-tuning.

Dataset resolution. mRNABench follows the resolution available in each source study. Gene-level measurements are paired with one principal transcript, while binding events and compartment measurements are summarized as transcript-level targets. This gives every label a defined sequence and a consistent prediction unit, but these tasks assess transcript-level outcomes rather than isoform-specific regulation or individual binding sites.

Experimental coverage. The benchmark brings together a broad set of assays, but every score still belongs to a particular species, cell type, and experimental system. Results from synthetic reporter constructs and endogenous transcripts should therefore be interpreted in their own biological settings.

Cite mRNABench

bioRxiv v1 (2025)

Ruian Shi, Taykhoom Dalal, Philip Fradkin, Divya Koyyalagunta, Simran Chhabria, Andrew Jung, Cyrus Tam, Defne Ceyhan, Jessica Lin, Kaitlin U. Laverty, Ilyes Baali, Bo Wang, and Quaid Morris.

bioRxiv v1 DOI: 10.1101/2025.07.05.662870

Open documentation
BibTeX
@article{shi_dalal_fradkin_2025_mrnabench,
    author = {Shi, Ruian and Dalal, Taykhoom and Fradkin, Philip and Koyyalagunta, Divya and Chhabria, Simran and Jung, Andrew and Tam, Cyrus and Ceyhan, Defne and Lin, Jessica and Laverty, Kaitlin U. and Baali, Ilyes and Wang, Bo and Morris, Quaid},
    title = {mRNABench: A curated benchmark for mature mRNA property and function prediction},
    elocation-id = {2025.07.05.662870},
    year = {2025},
    doi = {10.1101/2025.07.05.662870},
    publisher = {Cold Spring Harbor Laboratory},
    URL = {https://www.biorxiv.org/content/early/2025/07/08/2025.07.05.662870},
    eprint = {https://www.biorxiv.org/content/early/2025/07/08/2025.07.05.662870.full.pdf},
    journal = {bioRxiv}
}