Pinocchio：为黑盒 API 模型做不确定性估计的外部校准器
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
调用不返回 logprobs 的 API 模型时，可以用 Pinocchio 判断回答靠不靠谱，两行代码就能接入。
研究者提出 Pinocchio，一个外部校准器，用于预测黑盒 API 模型的回答是否正确，不需要访问目标模型的 logits、权重或内部状态。该方法在 7 个 LLM 的回答上联合训练，对留出回答的正确性预测达到 0.862 AUROC，并能零样本迁移到来自 8 个机构的 13 个未见模型。只需单次前向传播即可输出不确定性估计，纯文本的 0.8B 版本 AUROC 与最大模型持平。团队开源了代码，在现有仓库中加入不确定性估计只需额外两行代码。
Pinocchio: Fast Uncertainty Estimates for Black-Box Language Models
In high-stakes decision-making applications of large language models (LLMs), practitioners require not only accurate LLMs but also uncertainty estimates for their predictions. Existing approaches to uncertainty estimation for LLMs require access to log-probabilities output by the model or require fine-tuning access. However, many industrial LLM products use closed-source API models, and many such API models like GPT do not return log-probabilities and may not allow fine-tuning. We introduce Pinocchio, an external calibrator that estimates the correctness of responses from black-box API models. Trained jointly on responses from seven LLMs, it achieves 0.862 AUROC predicting the correctness of held-out responses from those same models, and shows zero-shot transfer to thirteen unseen models across eight organizations. Our model needs only a single forward pass to generate an uncertainty estimate and requires no access to the target model's logits, weights, or internal states. A lightweight text only 0.8B checkpoint matches our largest model's AUROC. We release code for adding uncertainty estimation to existing repos in only two additional lines of code.