成本感知弃权框架用于 LLM 工单分类：校准后成本降低 38.1%
Cost-Aware Abstention for LLM-Based IT Ticket Classification
这篇论文把 IT 工单自动路由的隐性成本算清楚了，微调的 DistilBERT 比 GPT-4o-mini few-shot 准确率高出近一倍，用成本模型选策略能省 38% 成本。
论文提出一种成本感知弃权评估框架，先在校准数据上以显式成本选择路由策略，再在测试数据上一次性评估冻结策略。在 47,837 条工单、8 个类别的公开语料上，TF-IDF 加逻辑回归配合 0.60 阈值将测试成本从 0.763 降至 0.472，相对降低 38.1%，错误路由率从 15.3% 降到 3.5%。对比显示 GPT-4o-mini few-shot 准确率仅 0.477，而微调后的 DistilBERT 达 0.879。在 0.80 的盈亏平衡概率下，七个提示词配置选择全部弃权，Haiku few-shot 仅路由 1,000 条中的 19 条。
Cost-Aware Abstention for LLM-Based IT Ticket Classification
Wrong automatic routing of an IT support ticket can cost more than human triage. We evaluate cost-aware abstention as an offline, cost-conditional readiness assessment: select a routing policy on calibration data under explicit costs, then evaluate that frozen policy once on held-out test data. On a public 47,837-ticket, eight-class corpus, we compare four trained non-LLM classifiers with four large language models under zero- and few-shot prompting. Under the illustrative cost model (c_ok, c_triage, c_wrong) = (0, 1, 5), TF-IDF with logistic regression and a threshold of 0.60 selected on calibration reduces test cost from 0.763 to 0.472, a 38.1% relative reduction (paired ratio-bootstrap 95% CI [35.7%, 40.4%]), while lowering wrong routes from 15.3% to 3.5%. Frozen calibration-selected policies also beat both always-route and always-defer on held-out test data in 14 of 16 settings in a post-hoc cost-parameter analysis. The strongest prompted configuration reaches 0.477 accuracy on a stratified 1,000-ticket test subset, versus 0.879 for fine-tuned DistilBERT on the same tickets. Isotonic lowers expected calibration error without reliably improving discrimination: GPT-4o-mini few-shot has one confidence value before and after recalibration and correctness AUROC 0.500. We therefore report correctness AUROC and tie-aware risk-coverage area alongside ECE. At the analytically derived break-even probability 0.80, seven prompted configurations defer all tickets; Haiku few-shot routes only 19/1,000. An LLM-only agreement control and margin/entropy rejection baselines further separate calibration from discrimination. The findings are scoped to this dataset, prompt design, model endpoints, and illustrative costs; the work is an evaluation framework, not a new abstention algorithm or universal deployment certification.