CAPS：从日志 bandit 反馈学习校准决策头
Learning a calibrated decision head from logged bandit feedback
训练分类头时只有“选了但对不对”这种反馈怎么办？这篇论文给出 CAPS 方法，比只学确认样本强不少，还把代码和适配器全开源了。做 RLHF 之外反馈学习或部署校准的工程师值得仔细看实验设计。
一篇 arXiv 论文提出 Chosen-Action Proper Scoring（CAPS），让决策头仅凭日志 bandit 反馈（只能观察到所选选项是否正确）完成训练并保持校准。作者在 Qwen3-0.6B-Base 和 1.7B-Base 上用 LoRA 适配，实验覆盖 Banking77 和情感数据集。
论文先证明把采样动作自身概率的 Brier/log score 当作 REINFORCE 奖励会导致崩溃：在二选项情形信号为零，一次实验中有序刻度问题准确率从 .636 跌到 .111。改用 pathwise 梯度按损失处理概率，或使用 r 与 r−p 奖励可避免崩溃。在无标签场景，CAPS 在 Banking77 上比只训练确认答案的基线高 2.6 个准确点、ECE 低 .053，情感数据上单种子高 21.7 点。
与全标签训练相比，N=5000 时 CAPS 在 0.6B 模型上仍差 1.5 个点、1.7B 上差 .6 个点；在全标签条件下，四种 bandit 方法都不如交叉熵加一次温度缩放。在已知翻转率的噪声反馈下，CAPS 置信度平均追踪噪声信道，backward correction 可恢复大部分干净校准。代码、适配器和逐条结果已公开。
Learning a calibrated decision head from logged bandit feedback
A decision head reads a state and a list of options and returns a probability per option in one forward pass. Once deployed, it usually observes only whether the option it chose was right: logged bandit feedback. We adapt such a head, LoRA on Qwen3-0.6B-Base and 1.7B-Base, from that feedback and report three results. First, the Brier or log score of the sampled action's own probability is not a usable REINFORCE reward: every wrong action earns at least gold's reward, with a mean baseline the lowest-probability wrong action gets the largest advantage, and at two options the signal is zero; in one run it cut accuracy on ordered-scale questions from .636 to .111. Scoring the probability as a loss by the pathwise gradient avoids the collapse, as do the REINFORCE rewards r and r minus p. Second, with full labels, none of four tested bandit arms beat cross-entropy plus one temperature on mean expected calibration error (ECE) over four unseen label schemas. Third, from logged feedback alone, Chosen-Action Proper Scoring (CAPS), the direct method with a proper score on the chosen action's probability, beats fine-tuning on confirmed answers only, which discards negatives: on Banking77 by 2.6 accuracy points, .053 lower ECE and lower selective risk (three seeds against one baseline seed), and by 21.7 points on one seed on emotion data with an often-wrong logger. At N 5000, CAPS stays 1.5 points below full-label training at 0.6B and .6 points at 1.7B against one full-label seed, and propensity weights add nothing on this log. Under feedback flipped at a known rate, one seed per rate, CAPS's confidence tracks the noisy channel on average, backward correction recovers most of the clean calibration, and early stopping on held-out logged outcomes prevented memorising noisy draws in all three runs. Code, adapters, logs and per-record results are public.
本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：CAPS、Qwen3、LoRA、校准，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.LG」（@Ye Yuan），经 AITOP 多维评分管线结构化提炼。