Introducing Helix-mRNA-v0

• A small model: 5M parameter.
• Through our sophisticated two-step pretraining approach, we pretrain on a diverse set of 26M mRNA sequences and a total training size of 312B tokens achieving SOTA performance while offering more flexibility.
• Our hybrid architecture integrates attention mechanisms with state-space models, leveraging the unique advantages of each approach.
• The model demonstrates remarkable efficiency through its streamlined design, maintaining high performance at a smaller scale.
• Our implementation of single-nucleotide tokenization enables fine-grained insights at the molecular level.

The intrinsic properties of mRNA molecules, such as translation efficiency, stability, and half-life, play an important role in determining the performance of mRNA-based therapies. Optimizing these properties is therefore essential in preclinical development. However, experimental optimization is often constrained by high costs and extensive timelines. This underscores the importance of robust in silico approaches to accelerate innovation while minimizing costs.

With Helix-mRNA we can optimize mRNA properties for specific functional requirements. It can be fine-tuned to improve translation efficiency, stability, or half-life, depending on the desired application. The model can evaluate existing mRNA sequences, assisting in the prioritization of candidates. Beyond that, Helix-mRNA can be applied for generative mRNA design, enabling the creation of diverse mRNA sequences for diverse applications like designing mRNA payloads for vaccines such as SARS-CoV-2.

Helix-mRNA was pretrained on a curated genomic dataset consisting of 26million mRNAs from diverse eukaryotic species and 238 clinically relevant viruses (resulting in 312 Billion tokens). Eukaryotic sequences included representative taxa from animals (mammals, birds, reptiles, amphibians, and fish), fungi, protists, and other animal clades, while plant sequences were excluded. The viral component encompassed major human pathogens including respiratory viruses (SARS-CoV-2, influenza A/B/C, RSV), retroviruses (HIV-1/2, HTLV-1/2), hepatotropic viruses (HBV, HCV), herpesviruses (HSV-1/2, EBV, VZV), and emerging arboviruses (Zika, Dengue 1-4). This taxonomically diverse pretraining dataset was specifically curated to capture both deep evolutionary conservation patterns and the distinct nucleotide compositions and structures characteristic of viral genomes. The broad phylogenetic coverage enables the model to learn robust representations of sequence features across different evolutionary distances and genomic architectures.

‍