ScoreCLR 用去噪得分匹配估计对比学习斥力项,替代负样本批次
Contrastive latent representation learning with denoising score matching
对比学习里估斥力要靠大批次,这篇论文换了个思路:用去噪得分匹配学一个斥力网络,成本从批大小平方降到线性。在 CIFAR-10 和 ImageNet-100 上和 SimCLR 打平,显存紧张的团队值得看看这套做法。
论文提出 ScoreCLR,一种新的对比表征学习方法,核心变化是用去噪得分匹配训练的轻量网络来估计斥力项,替代显式负样本批次。方法在最大化样本两个潜视图互信息的框架下,推导香农熵对编码器参数的梯度,得到涉及潜表征得分的等价目标,再联合训练得分估计网络,并配合多噪声水平正则化与潜空间高斯先验。
成本上,ScoreCLR 的斥力估计对批大小是线性复杂度,而对比方法中显式负样本对批大小是二次复杂度,因此摆脱了对大批次训练的依赖。在 CIFAR-10 和 ImageNet-100 上,与 SimCLR 共享同一训练配方时,线性探测准确率损失极小,同时投影器输出的有效秩更高,更接近高斯先验。
Contrastive latent representation learning with denoising score matching
Contrastive representation learning methods operate by pulling different views of the same data sample together while pushing them away from views of different samples. A fundamental challenge is the estimation of the repulsive force, which typically requires large batch sizes, can exhibit high variance, and is susceptible to the curse of dimensionality. We introduce ScoreCLR, a contrastive representation learning method that offers a novel way to estimate this repulsion term. We consider the framework of representation learning by maximizing the mutual information between two latent views of a sample. By deriving the gradient of the Shannon entropy with respect to the encoder parameters, we obtain an equivalent objective that involves the score of the latent representation. We then use denoising score matching techniques to estimate this score with a lightweight network that is trained jointly with the encoder. We further propose regularization techniques using different noise levels and a Gaussian prior on the latent representation. ScoreCLR replaces the batch of explicit negative samples in contrastive methods with a learned repulsion whose cost is linear in the batch size rather than quadratic. On CIFAR-10 and ImageNet-100 it does so with minimal loss of linear-probe accuracy compared to SimCLR under a shared recipe, while producing projector outputs of higher effective rank, closer to the Gaussian prior.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:ScoreCLR、SimCLR、对比学习、去噪得分匹配,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Christopher Marouani, Grigory Bartosh, Christian A. Naesseth),经 AITOP 多维评分管线结构化提炼。