arXiv 论文提出 Pairwise Bellman Validation 简化离线策略评估中的模型选择
Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
做强化学习离线评估的人可以看看，PBV 不用额外训练验证模型就能在候选里挑出最好的 Q 函数，论文里 Hopper 和 Deep OPE 都有实验。
论文针对离线策略评估（OPE）中拟合 Q 函数与占用率比估计难以用留出数据直接验证的问题，提出免调参的 minimax 方法 Pairwise Bellman Validation（PBV）。PBV 用候选模型两两差值构造 critic，只需有限次经验矩计算，无需辅助回归或单独优化 critic 类。作者给出有限样本 oracle 不等式，即使候选类存在误设也能保证误差控制。方法包括针对 Q 函数选择的 unweighted PBV 与 TA-PBV，以及针对占用率比选择的 A-PBV；在 Hopper 和 Deep OPE 基准上，unweighted PBV 表现有竞争力。
Model Selection for Off-Policy Evaluation via Pairwise Bellman Validation
Model selection and hyperparameter tuning pose a central challenge in off-policy evaluation (OPE) because errors of fitted Q-functions and occupancy-ratio estimates cannot be directly evaluated from held-out transitions. Bellman-residual and minimax validation methods often require auxiliary regressions or optimization over separate critic classes, adding computational and tuning costs. We introduce pairwise Bellman validation (PBV), a tuning-free minimax procedure for selecting among finite classes of fitted Q-functions and occupancy-ratio estimators using held-out transitions. PBV constructs critics from pairwise candidate differences, requiring only finitely many empirical moment calculations without auxiliary validation models or separate critic optimization. We establish finite-sample oracle inequalities relative to the best available candidate, even under candidate-class misspecification. Our key insight is that Q-function errors and log ratios between candidate and true occupancy ratios provide oracle critics whose Bellman and adjoint Bellman moments control estimation error. Pairwise candidate differences approximate these unknown critics, while penalties account for comparator approximation error. For Q-function selection, we first develop unweighted PBV, which evaluates pairwise Bellman moments under the offline distribution and admits oracle guarantees under a pairwise coercivity condition relating the offline distribution to the target-policy dynamics. We then develop target-aligned PBV (TA-PBV), which uses marginalized importance weighting to provide error control without pairwise coercivity. For occupancy-ratio selection, we develop adjoint PBV (A-PBV), which evaluates adjoint Bellman moments against log-ratio critics. On Hopper and Deep OPE, unweighted PBV performs competitively, while target-occupancy weighting can improve performance in settings with controlled overlap.