研究分析锥形分类头在 softmax 交叉熵下能否约束特征
Asymptotic Margins and Feature Confinement in Conic Classification Heads under Softmax Cross-Entropy
这篇理论工作回答了开放式分类问题里一个实际困惑:有界类区域并不等于特征有界。做开集识别、新颖类拒绝的人值得看,里面给出可用 SOCP 直接计算的渐近裕度判定,还在 CIFAR-10/100 上验证了 1210 个单元格。
arXiv 论文(编号 2611.18507)研究锥形分类头在 softmax 交叉熵训练下能否将特征限制在有界区域内,这类有界区域被用于拒绝未见类样本。作者针对线性项与欧氏范数组合的打分头证明:每个类的区域由一个旋转二次曲面界定,其离心率等于某个学习到的比率的倒数。
论文进一步给每个类定义了渐近裕度(asymptotic margin),比较该类得分与其他所有得分的增长速度,可通过二阶锥规划(SOCP)计算。裕度为正时损失下确界为零且仅沿发散特征趋近;裕度为负时损失是 coercive 的,极小值点落在显式给出的球内。
由于 softmax 只依赖得分差,区域有界的类仍可有正裕度,因此有界区域并不约束特征。该结论可推广到包括多面体锥函数在内的其他范数。实验上,CIFAR-10 与 CIFAR-100 上 67 个训练好的锥形头共 1210 个单元格全部裕度为正,而 prototype 头如理论预测地保留了 10 个单元格中的 9 个有界。
Asymptotic Margins and Feature Confinement in Conic Classification Heads under Softmax Cross-Entropy
Classification heads built on conic functions give each class a bounded region of the feature space, and such regions are used to reject samples of unseen classes. We ask whether these regions also keep the features bounded when the head is trained with softmax cross-entropy. For a head whose class scores combine a linear term with a Euclidean norm, we first show that the region of each class is bounded by a quadric of revolution, whose eccentricity is the reciprocal of one learned ratio. We then attach to each class an asymptotic margin, which compares the growth of its score with the growth of all the other scores and can be computed by a second-order cone program. The loss of a class has infimum zero exactly when its margin is positive, and it then approaches zero only along diverging features. When the margin is negative, the loss is coercive and its minimizers lie in an explicit ball. Since softmax depends only on the differences between scores, a class with a bounded region can still have a positive margin, so bounded regions do not confine the features. The result extends to other norms, including the polyhedral conic functions. On CIFAR-10 and CIFAR-100, all 1210 cells of 67 trained conic heads have positive margins, whereas a prototype head keeps nine of its ten cells bounded, as the theory predicts.
本条情报归属 AITOP「论文」核心追踪栏目。涉及核心实体与技术议题包括:softmax交叉熵、开集识别、CIFAR-10、分类头,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Omer Nezih Gerek),经 AITOP 多维评分管线结构化提炼。