Optimal Sequential Annotations for Off-Policy Evaluation：有限标注预算下的双稳健估计方法
Optimal Sequential Annotations for Off-Policy Evaluation
标注预算不够又想评估策略？这篇 arXiv 论文告诉你怎么分配人工标注最省钱，RMSE 能降一半以上。
该论文研究离线强化学习中的 off-policy evaluation 问题：当状态和奖励是文本或图像时，可用 LLM-as-a-judge 或廉价分类器标注，但存在未知偏差，专家标注则更贵。作者提出在双稳健 OPE 框架下优化序列标注概率，给出 batch-adaptive 的可行实现。在无家可归服务非营利机构的 casenotes 数据集上，以 40% 全量标注预算实现 RMSE 降低 34-65%；在 LMArena 人类偏好投票数据集上，所有预算下 RMSE 降低 55-62%。
Optimal Sequential Annotations for Off-Policy Evaluation
Offline reinforcement learning and off-policy evaluation evaluates dynamic treatment rules based on retrospectively collected data prior to deployment. In recent AI applications, state and reward information is recorded as complex text or image, which recent AI advancements such as LLM-as-a-judge can label with unknown bias. Expert annotation may be available but at a higher cost. For example, safety classification via cheap but imperfect classifiers vs. expensive expert review. We show how a limited budget for ground-truth data-annotation can be used via doubly-robust OPE with missing rewards, and we optimize variance-optimal annotation probabilities for sequential off-policy evaluation, where the target policy value is estimated from annotated data. We characterize the optimal annotation probabilities for sequential forward-monotone annotation protocols, and provide a feasible batch-adaptive implementation. Our work is motivated by a collaboration with a homelessness services nonprofit that writes casenotes for individuals over time. Our method can be used to unlock trustworthy inference from casenote data and answer new inferential questions such as: how does expanding outreach effort over time affect progress towards a housing application and improvement in housing placement? In simulations and on two real datasets - casenotes from the nonprofit and human-preference votes from LMArena - we see reductions in RMSE of 34-65% for housing placement and 17-68% for progress towards a housing application at budgets of 40% of full annotation and above, and by 55-62% at every budget on LMArena.
