分层策略:用多个 LoRA 适配器优化推理时 pass@k 采样
Learning Stratified Policies for Inference-Time Objectives
做推理时采样或 best-of-n 的朋友可以看看这篇:不重复采一个模型,而是挂多个 LoRA 各出一个答案,理论证明了误差能降为零,pass@k 在多个推理任务上稳定跑赢单策略。想挤榨采样预算的话值得照着实现试试。
arXiv 论文(编号 2611.23154)提出 stratified policies(分层策略),将推理时的 k 次采样预算分配给 k 个不同的学习策略,而不是反复从单一策略采样。实现方式是在共享的语言模型主干上挂载 k 个独立的 LoRA 适配器,每个策略贡献一个回答,并以 pass@k 为目标联合优化。理论层面,作者在线性设置下证明:在单个 LoRA 都存在严格正预测误差的场景中,一组 LoRA 可以把 best-of-k 近似误差降到零。
实验显示,在多个推理任务上,分层策略的 pass@k 一致优于单策略基线。核心结论是:把采样预算分散到互补的策略上,比从单一策略做 i.i.d. 重复采样更能提升推理时扩展的效率。
Learning Stratified Policies for Inference-Time Objectives
Inference-time scaling improves the performance of large language models by sampling multiple responses and evaluating their collective utility. However, many existing approaches obtain these samples by repeatedly sampling from a single policy. In this work, we study if the sample budget at inference time can instead be allocated across a population of policies. We introduce stratified policies, where each of $k$ learned policies contributes one response to a $k$-sample inference-time objective. We realize this policy population using separate LoRA adapters on a shared language model backbone and optimize it for pass@$k$. We theoretically motivate the approach by analyzing the linear setting; a population of LoRAs can achieve zero best-of-$k$ approximation error in settings where every single LoRA has strictly positive prediction error. Empirically, we demonstrate that stratified policies consistently improve pass@$k$ over a single-policy baseline across multiple reasoning tasks. Overall, our results highlight how distributing samples across complementary policies can be a more effective use of sample budget than repeated i.i.d. sampling from a single policy at inference time.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:LoRA、pass@k、推理模型、采样策略,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.AI」(@Mridul Mahajan, Xuezhou Zhang, Aldo Pacchiano),经 AITOP 多维评分管线结构化提炼。