论文

arXiv 研究:grokking 现象不会改变神经网络训练能效曲线

Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?

精选理由

训练模型烧电,能量效率能不能预测?这篇论文用 modular arithmetic 和 MNIST 实测后发现:即便模型出现 grokking 式的延迟泛化,能耗效率曲线照样平滑下降。做训练成本估算、能耗优化的工程和研究人员可以看看他们的实验设计。

一篇 arXiv 论文研究了 grokking(模型在明显过拟合之后延迟出现的分布外泛化能力)是否会影响深度神经网络的训练效率,效率定义为单位能耗下的准确率。实验涵盖 modular arithmetic 和 MNIST 两个经典任务,测试 16、32、64-bit 三种精度,训练轮数从 500 到 20000 epochs。

正文

论文引入 Orthograd 正则化方法做干预:短训练(≤500 epochs)中即使施加更强的 Orthograd 也不会诱发 grokking,效率曲线平滑下降;长训练(≤20000 epochs)中 Orthograd 成功诱导 grokking,但测试准确率跃升点前后,效率曲线仍近似单调,没有局部抬升。

作者据此判断,在其实验设定下 grokking 只是训练后期的泛化事件,并不实质性改变整体训练能效动态。对关注训练能耗核算的人来说,这意味着不需要为可能出现的 grokking 现象预留额外的能效波动预算。

原文 · arXiv cs.LG

Deep Neural Networks Training Efficiency: Does Grokking Modify its Dynamics?

We study whether grokking, the delayed emergence of out-of-distribution generalization after apparent overfitting, alters training efficiency in deep neural networks, where efficiency is measured as accuracy per unit energy. Using two canonical settings, modular arithmetic and MNIST, we analyze two neural network models under 16-, 32-, and 64-bit precision and use Orthograd to test whether grokking produces noticeable deviations from the typically decreasing dynamics of training efficiency. We consider both short training runs of up to 500 epochs and long training runs of up to 20,000 epochs, and in both cases identify grokking as a sharp improvement in test accuracy after a sustained train-test gap. In the short-horizon experiments, applying Orthograd at progressively stronger levels during early training does not induce grokking, and efficiency remains smoothly decreasing. In the long-horizon experiments, Orthograd induces grokking, but the efficiency curves remain approximately monotonic and show no appreciable local increase at the grokking transition. These results suggest that, in our setting, grokking does not materially alter overall training-efficiency dynamics.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:grokking、Orthograd、MNIST、训练效率,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Eduardo Cueto-Mendoza, John D. Kelleher),经 AITOP 多维评分管线结构化提炼。