论文

论文提出教师-图灵测试范式:以"被误认为教师"作为强化学习奖励

Uncovering the Turing Test's Latent Signal: Indistinguishability as a Reward

精选理由

让小模型靠"骗过"大模型来学本领,不用标准答案,Qwen2.5-1.5B 在四个基准上都有提升,训练思路挺新颖。

arXiv 论文提出 domain-based teacher-Turing-Test 方法:让 Qwen2.5-1.5B 学生模型模仿 DeepSeek V4 Flash 教师模型,以教师"把回答归因于自己"的概率作为强化学习奖励。训练过程中不提供标准答案,也不需要教师生成回复。该方法在 CommonsenseQA、GSM8K、KodCode、Spider 四个域的 held-out 评测中均带来提升。分析显示自我归因概率与答案正确性相关,且能奖励部分正确的回答。

原文 · arXiv: DeepSeek

Uncovering the Turing Test's Latent Signal: Indistinguishability as a Reward

Can a model learn by trying to fool another model? We show that it can. Inspired by the Turing Test and its recent generalization to arbitrary distinguishers, we introduce a domain-based teacher-Turing-Test, in which a stronger teacher model serves as both the target of imitation and the distinguisher, and a weaker student model is trained to make the teacher attribute responses to itself. We use this self-attribution (or fooling) probability as a reinforcement learning reward, without ever providing the student ground-truth answers, nor requiring the teacher to generate responses. We test our approach independently in four domains with verifiable answers: CommonsenseQA (reasoning), GSM8K (mathematics), KodCode (programming), and Spider (SQL), training a Qwen2.5-1.5B student/imitator model using DeepSeek V4 Flash as teacher/distinguisher. Beyond improving at passing the Turing Test, the student improves on independent, held-out evaluations in all domains and runs. To probe why being mistaken for the teacher can improve capability, our analyses reveal that self-attribution is associated with correctness -- substantially more than formatting changes -- and that it rewards partial correctness. Our results demonstrate a new, simple teacher-student training paradigm with a private and/or high-cost teacher. They also suggest that indistinguishability need not merely measure behavior -- it can also be a mechanism for learning.