RTLRepairBench：考察 RTL 修复基准究竟在测什么
What Do RTL Repair Benchmarks Actually Measure?
搞硬件/RTL 生成的朋友看下：这篇用 459 个测试用例说明现有修复基准可能没测到你以为的东西，还开源了整套数据和评估框架。
论文构建了 RTLRepairDataset，包含 5,784 对（损坏、修复）训练样本，以及 459 个测试用例的 RTLRepairBench（374 个语法突变、85 个行为突变），并配有支持仿真与形式化等价验证的评估框架。作者发现两个测量混淆因素：在一个评估队列中，仿真与形式化评分对“提供规格说明”的效果给出相反符号，reset 感知的重新评分将几乎全部分歧归因于复位和首周期比较语义。在 274 个模型失败案例上，六个模型的规格说明条件化恢复点估计在 -8.5 到 +18.3 个百分点之间波动。论文提出评估控制建议：报告初始化、复位、首周期与未知值语义，披露 oracle 覆盖范围，并纳入仅规格说明的基线。
What Do RTL Repair Benchmarks Actually Measure?
RTL repair benchmarks should measure whether a model can fix broken RTL given a specification. To this end, we construct and hash-bind RTLRepairDataset, 5,784 (broken, fixed) training pairs; RTLRepairBench, 459 held-out cases with 374 syntactic and 85 behavioral mutations; and an evaluation harness supporting simulation and formal equivalence. We then use this infrastructure to identify two measurement confounds. First, simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of the simulation-formal disagreement to reset and first-cycle comparison semantics. Second, on 274 mined model failures, the specification-only minus specification-conditioned recovery point estimate ranges from -8.5 to +18.3 percentage points across six models. Thus, a conventional repair score need not isolate whether the model used the faulty RTL. We translate these findings into evaluation controls: report initialization, reset, first-cycle, and unknown-value semantics; disclose oracle coverage and undecided cases; and include a specification-only baseline. Together, the resources and audit provide both a benchmark for behavioral repair and a protocol for diagnosing what contributes to its measured score. We release the benchmark, the training set, construction code, sanitized aggregate evidence, and reproduction materials at https://github.com/Evaaaz/RTLRepairBench.