论文

Typed Decision Model 能否取代微调分类器?团队在 24 篇 ACL 论文上实测

Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers

精选理由

做文本分类的朋友看过来:研究者拿 Jev 换掉了 24 篇 ACL 论文里的微调分类器,零样本就能赢下 12/46 个测试集。加上 500 条标注和提示优化(中位成本 1.66 美元)能追回 72% 差距,比 LoRA 微调 Qwen3.8-27B 的 97% 还差一截,但省掉了管 GPU 的麻烦。想快速上分类基线的团队可以直接抄这套流程。

一项发表于 arXiv 的研究(编号 2611.20191)用 TypeSafe 的 Typed Decision Model Jev 替换了 24 篇 ACL 论文中的任务分类器,保留每篇论文的数据切分、标签定义和评测指标,并借助 Cloudflare 的 Clef、Microsoft-Decision-1 和 OpenAI 的 GPT-6 Luna Decisions 交叉核验结果,最终形成包含 46 个英文测试集的 TDM-Gauntlet 基准。

正文

零样本提示下,Jev 直接在 46 个测试集中的 12 个上达到或超过原论文方法,主要集中在标签含义直白的单一判断任务;表现落后的场景多为原方法在自有标签上训练过、且标注约定未体现在标签名称中的情况。

在给 500 条训练标签加验证集、不管理 GPU 的条件下,校准加 GEPA 风格提示优化能弥合与原论文方法差距的中位数 72%,约为现成 GEPA 效果的两倍,单次搜索成本中位数 1.66 美元。

同一批标签下,RoBERTa 和 DeBERTa 等微调编码器弥合 59%,对开源模型 Qwen3.8-27B 做 LoRA 微调则达 97%。

结论是微调仍是精度上限最高的方案,但标签语义清晰时零样本提示已够用,提示优化能覆盖大部分剩余差距。

原文 · arXiv: OpenAI

Do We Still Need Fine-Tuned Classifiers? Swapping Typed Decision Models into 24 ACL-Venue Papers

Typed Decision Models such as TypeSafe's Jev, also called System One or System 1 models, answer typed questions about an input with a probability for each option, and they need no training. Many NLP papers still fine-tune a task-specific classifier, and prompted large language models (LLMs) trail such classifiers on most text classification benchmarks. We ask how many of these fine-tuned classifiers a Typed Decision Model can replace under each paper's own evaluation, and what is needed where it cannot. We swap the predictors of 24 ACL-venue papers for Jev, keep each paper's splits, label definitions and metric, and check the results with Cloudflare's Clef, Microsoft-Decision-1 and OpenAI's GPT-6 Luna Decisions. The resulting 46 English test sets, with their questions and published scores, form the TDM-Gauntlet benchmark. With a zero-shot prompt, Jev directly replaced the paper's predictor on 12 of the 46 test sets, mostly single judgments with plain labels, but rarely where the paper's method was trained on its labels. Where Jev scored lower, we most often attributed the shortfall to an annotation convention that the label name does not state. With 500 training labels plus validation and no user-managed GPU, calibration and GEPA-style prompt optimization closed a median 72% of the gap to the paper's method, about twice what off-the-shelf GEPA closed, for a median $1.66 per search. On the same labels, fine-tuned encoders such as RoBERTa and DeBERTa closed 59%, and LoRA fine-tuning of the open LLM Qwen3.8-27B closed 97%. Fine-tuning remains the most accurate option, but a zero-shot prompt is often enough for plain labels, and prompt optimization closes most of the remaining gap.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Jev、GEPA、RoBERTa、Qwen3.8-27B,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始信息发布自一手权威信源「arXiv: OpenAI」(Tier-1 一手发布),具备高确定性参考价值。