研究揭示基于 Prefill 的解码长度预测存在理论上限
On the Fundamental Limits of Prefill-Based Decoding Length Prediction
这篇论文解释了为什么解码长度预测器换个模型就失效，还给出了理论证明，做推理调度的开发者可以看看。
一篇 arXiv 论文指出，仅用 prefill 阶段隐藏状态预测 LLM 解码 token 数的方法，在更换目标模型、数据集或采样配置后准确率会大幅下降。作者将解码建模为停时随机过程，推导出每步停止概率在数百步内逐步上升，解码长度因此服从中心极限定理，并满足方差仿射律 σ=αμ+b。实验显示 MLP、EGTP、FiLM、soft-label、MoE 等多种预测器在固定条件下性能接近理论上界，说明系统误差主导模型误差。论文提出固定模型、数据与采样参数并报告相对误差指标的基准测试协议。
On the Fundamental Limits of Prefill-Based Decoding Length Prediction
Predicting how many tokens a large language model (LLM) will decode, using only prefill-phase hidden states, is a key primitive for inference scheduling. We show that current benchmarks conflate predictor quality with task hardness: when the target model, dataset, or sampling configuration changes, reported accuracy collapses even though the predictor is unchanged. We trace this instability to an overlooked error source---the inherent stochasticity of autoregressive decoding---and derive bounds that every prefill-side method must obey. Our analysis treats decoding as a stopped stochastic process. The per-step stopping probability of a response is not constant but ramps up over hundreds of steps, so the decode length inherits a central limit theorem from the many local decisions that determine where a response ends; this yields per-prompt normality with an explicit error term, together with an affine variance law $σ=αμ+b$ whose slope is the coefficient of variation of the number of response segments. Empirically, the normal law reproduces the per-prompt MAE and the central mass of the decode-length distribution to within a few percent, and the residual departures from normality (mild right skew, slight excess kurtosis) are the ones predicted by the sampling-noise correction. Under fixed model, data, and sampling conditions, a broad design space of predictors (MLP, EGTP, FiLM, soft-label, and Mixture-of-Experts variants) converges to similar performance close to the bound, confirming that \emph{system error dominates model error}. We argue for a benchmarking protocol that fixes the model, data, and sampling parameters and that reports relative-error metrics alongside absolute ones. Code and analysis tools are available at https://github.com/ShwStone/DecodingPredict.