研究发现 LLM 可直接编程表格基础模型权重与缓存
Weights & Caches: LLMs Can Program Tabular Foundation Models
GPT-6 Astra 不训练就能把 nanoTabPFN 从零权重编到 ROC AUC 0.849,还能靠改 KV-cache 让 TabPFN-3.5 反转趋势。做表格模型或可解释性研究的人应该看看这篇,思路和微调完全是两条路。
arXiv 论文(编号 2611.20174)展示了 LLM 能在不做梯度优化的前提下,直接对表格基础模型(TFM)进行权重编程。
实验使用 GPT-6 Astra,在三种设定下验证:一是从零初始化权重直接编程 nanoTabPFN,在 20 个 TabArena 数据集上取得平均 ROC AUC 0.849,接近预训练基线的 0.
876;二是诊断并修复了预训练 TabPFN-3.
5 中符号乘积回归的失败点,修复后仅带来 0.08% 的平均相对误差增加;
三是通过编写 KV-cache 编辑程序改变 TabPFN-3.5 的预测行为,例如反转线性趋势、把正弦波转成近似方波,全程不改动模型权重。
论文认为 LLM 驱动的编程可以补充基于梯度的优化方式,也为观察和干预 TFM 内部计算提供了新途径。
Weights & Caches: LLMs Can Program Tabular Foundation Models
Neural networks are typically programmed through optimization, rather than by explicitly specifying their computations. Recent work on weight-space programming suggests that large language models (LLMs) can instead reason about and manipulate neural-network weights directly. In this work, we show that LLMs can directly program statistical computations in tabular foundation models (TFMs), whose parameters enable inference across different datasets. We investigate three complementary settings using GPT-6 Astra, without gradient-based optimization. First, Astra directly programs nanoTabPFN from zero-initialized weights, achieving a mean ROC AUC of 0.849 across 20 TabArena datasets, compared to 0.876 for a pretrained baseline. Second, Astra diagnoses and repairs a signed-product regression failure in pretrained TabPFN-3.5, restoring near-perfect predictions on held-out synthetic datasets while preserving performance on TabArena-Lite, with only a 0.08% average relative increase in error. Third, it writes KV-cache-editing programs that steer TabPFN-3.5's predictions on unseen synthetic datasets, demonstrating transformations such as reversing linear trends and converting sine waves into approximate square waves, without modifying model weights. Together, our findings suggest that LLM-driven programming could complement gradient-based optimization across TFMs' development and usage, while offering a new way to investigate and intervene in their internal computations.
本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:TabPFN、GPT-6 Astra、KV cache、TabArena,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Shi Bin Hoo, Magnus Buehler, Omar Swelam, Frank Hutter, Lennart Purucker),经 AITOP 多维评分管线结构化提炼。