CA-ZODPO:零阶偏好优化方法将 GPU 峰值显存降低约 59%
Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment
做 LLM 对齐但显存不够的朋友可以看看这篇。它把扰动限制在 chosen/rejected 激活差异的低秩子空间里,在 Qwen2.5-1.5B 上跑 HH-RLHF,效果接近 DPO,峰值显存省 59%。想低成本做偏好训练的团队值得复现试试。
一篇 arXiv 论文提出 CA-ZODPO(Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation),针对 DPO 依赖反向传播导致激活显存开销大的问题,改用零阶优化并构建低秩搜索子空间。该子空间由每个偏好对中 chosen 与 rejected 响应的激活差异构造,扰动被限制在该子空间内,同时在低秩坐标上保留动量,从而避免各向同性扰动带来的噪声更新。在 Qwen2.5-1.5B 与 Anthropic HH-RLHF 数据集上的实验显示,CA-ZODPO 超过常见 ZO 基线,成对偏好评估中与一阶 DPO 表现相当,序列长度 2048 时峰值 GPU 显存降低约 59%,且保持 ZO 训练随序列长度扩展的特性。
技术上的核心是用激活差异而非随机方向来引导扰动:偏好目标的梯度天然来自 chosen 与 rejected 的差值,把搜索范围压到这个差异对应的低秩子空间,等价于先验地过滤掉与目标无关的噪声方向。这一改动让零阶方法在无需反向传播的前提下逼近一阶方法的对齐质量。
对显存受限、无法做全参数反向传播的团队来说,这条路线意味着在单卡或低显存环境下也能做偏好对齐训练,代价是收敛速度可能仍受零阶采样效率影响,论文未给出训练时长对比数据。
Contrastive Activation-Guided Zeroth-Order Preference Optimisation for Memory-Efficient LLM Alignment
Direct Preference Optimisation (DPO) simplifies reinforcement learning from human feedback, but still relies on traditional backpropagation and its associated activation-memory cost. Zeroth-order (ZO) optimisation avoids backpropagation, yet isotropic perturbations can produce noisy updates for preference objectives defined by differences between chosen and rejected responses. We introduce Contrastive Activation-guided Zeroth-Order Direct Preference Optimisation (CA-ZODPO), which constructs a low-rank search subspace from the activation discrepancy between the chosen and rejected responses in each preference pair. Perturbations are restricted to this subspace, while momentum is maintained in low-rank coordinates to preserve the memory advantages of ZO training. On Qwen2.5-1.5B with Anthropic HH-RLHF, CA-ZODPO outperforms common ZO baselines and remains competitive with first-order DPO in pairwise preference evaluation. It also reduces peak GPU memory by about 59\% at a sequence length of 2048 while retaining the sequence-length scaling behaviour of zeroth-order training.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:DPO、Qwen2.5、Anthropic HH-RLHF、零阶优化,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始信息发布自一手权威信源「arXiv: Anthropic」(Tier-1 一手发布),具备高确定性参考价值。