JD 提出 ROAD MOPD：用双教师一致性信号抑制蒸馏中的梯度冲突
JD 在 MOPD 上加了个双教师一致性过滤，教师意见统一才加强监督，搞蒸馏的同学可以看看这个思路。
JD 团队针对 MOPD 蒸馏方法提出改进版 ROAD MOPD，核心是 Cross-Teacher Guarding 机制。一名 assigned teacher 和一名 auxiliary teacher 对学生模型的同一个响应打分，并相对于一个冻结的 reference model 比较两者预测。当两位教师判断一致时加强该 token 的监督权重，不一致时降低权重，以此抑制教师分歧造成的梯度冲突。作者同时指出该设计受限于 reference model 的判断方式，认为基于 KL 散度的方案可能更优。
from JD, a refinement on MOPD to further suppress gradient conflict due to [mostly superficial] teacher divergence: Cross-Teacher Guarding for MOPD (ROAD MOPD) -assigned teacher and an auxiliary teacher evaluate a student response. - Their predictions are compared relative to a frozen reference model - When teachers agree, strengthen the token's supervision; when disagree, reduce
I see the utility, but it's obviously hobbled by the reference model. Eg if it assigns lower prob than both teachers, then they "agree", and if it's in between, then not. The conceit here is that agreement *relative to the student's initialization* is a viable proxy for cross-domain compatibility. OK but I suspect that some KL-based approach would be better
