ProtModernBERT：省80-90%预训练算力的蛋白质语言模型
ProtModernBERT: Compute-Efficient Protein Language Modeling
算力紧张也能训：ProtModernBERT 用一两成算力追平 ProtBert，架构比参数量更关键。
研究者提出 ProtModernBERT，一个在氨基酸序列上预训练的现代 BERT 架构模型，直接对标 ProtBert、Amplify-120M 和 ESM2。在二级结构预测、无序区预测、亚细胞定位等监督任务和 ProteinGym 零样本功能基准上，它以最多减少 80-90% 的预训练 FLOPs 达到或接近 Amplify-120M 与 ProtBert 的成绩，并在控制预训练条件后缩小与 ESM2 的差距。作者采用基于聚类的验证协议挑选超参数，再用随机初始化模型做对照，区分架构与数据各自的贡献。结论是现代架构组件带来更强的归纳偏置，在低算力场景下比参数量更关键。
ProtModernBERT: Compute-Efficient Protein Language Modeling
Large protein language models provide a powerful framework for learning sequence-function relationships and guiding experimental design. However, architectural improvements are underexplored, especially in low-compute regimes of relevance for resource-constrained research laboratories. We show that the training cost for encoder-only transformer architectures can be reduced by leveraging modern architectural refinements, offering an encouraging performance / compute-efficiency trade-off. We introduce ProtModernBERT, a modernized BERT-style model pre-trained on amino acid sequences, and compare it with ProtBert, Amplify, and ESM2 across a set of supervised structural tasks -encompassing secondary structure prediction, disorder prediction, and subcellular localization- and ProteinGym's zero-shot functional benchmark. We find that ProtModernBERT matches or closely tracks the performance of Amplify-120M and ProtBert with up to 80-90\% reduction in pre-training FLOPs, and narrows the gap to ESM2 when controlling for pre-training. Our analysis using a cluster-based validation protocol for hyperparameter selection and randomly initialized models disentangles the contributions of architecture and data, showing that modern components provide stronger inductive biases that are more consequential than parameter count in low-compute regimes.
