OwlSight 论文提出 Factor 策略监控思维链推理
OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning
一篇讲怎么监控推理模型思维链的论文，一个监控器管多任务，还在 OpenAI 的基准上拿了第一。
OwlSight 是一个思维链（CoT）监控器，通过 Factor 策略让用户在提示词中自定义监控要求，单个监控器即可覆盖不同任务。Factor 分为细粒度和粗粒度两类，例如监控特定邮箱是否泄露与监控隐私泄露。基于 15,971 条推理轨迹的实验显示，OwlSight 在细、粗粒度上分别超过通用 LLM 1.32-30.57 和 11.71-27.81 分，并在 OpenAI 的 CoT monitorability 基准上取得 SOTA。它还能泛化到未见过的模型 CoT，并对 Factor 改写、CoT 压缩和混淆保持鲁棒。
OwlSight: A Panoramic Monitor for Chain-of-Thought Reasoning
Chain-of-Thought (CoT) monitoring offers actionable evidence for oversight AI systems, with the potential to enable early detection and intervention. However, existing approaches are focused on specific tasks, whereas the regulation of AI, particularly reasoning models, requires monitoring a broader range of tasks. We introduce Factor for CoT monitoring, a strategy that supports customizing monitoring requirements in the prompt, which enables a single monitor to cover different tasks by relying on different factors. Furthermore, we introduce two types of factors, fine-grained and coarse-grained (e.g., monitoring whether a specific email address has been leaked versus monitoring privacy leakage), to accommodate diverse user monitoring needs. Based on this, we develop a CoT monitor named OwlSight that relies solely on factor to monitor behaviors related to multiple dimensions, including capability, safety, and trustworthiness. Experiments across 15,971 reasoning trajectories show that OwlSight surpasses general-purpose LLMs by 1.32-30.57 and 11.71-27.81 points at fine and coarse granularity, respectively, and achieves state-of-the-art results on OpenAI's CoT monitorability benchmark. It also generalizes to unseen models' CoTs and remains robust to Factor rephrasings, CoT compression, and obfuscation, offering a promising paradigm for CoT monitoring that can contribute to the oversight of frontier AI.
