无参考场景下用多模型一致性评估 LLM 代码摘要可信度
Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization
写 COBOL 这种没人愿意人工标注的老代码文档时，这篇给了个思路：让几个模型各写摘要互相投票，不用人也能估准不准。
论文针对 LLM 生成的 COBOL 代码摘要，提出一种无需人工参考答案的正确性估计方法。方法分三步：由多个不同模型生成候选摘要、用 LLM-as-judge 对摘要做两两结构化对比评分、再用校准聚合规则合并证据并只惩罚有相互印证的矛盾。核心假设是模型间在语义层面的一致性可以指示真伪，且 LLM 评审比字符串或嵌入相似度指标更可靠。
Ensemble Methodology for Reference-Free Confidence Estimation in LLM-based Code Summarization
This paper describes an approach and use case to estimate the correctness of LLM-generated COBOL summaries without access to human-authored references. The core intuition is that inter-model agreement on semantics is informative about truth, and that an LLM-as-judge can provide higher-fidelity pairwise assessments than traditional string- or embedding-based metrics. We operationalise this intuition with (i) controlled summary generation from diverse models; (ii) a LLM evaluator that compares pairs of summaries and emits structured judgments; and (iii) a calibrated aggregation rule that combines supportive evidence while penalising only corroborated contradictions.
