Inside-JEPA：用逆动力学损失让潜空间世界模型理解实体关系
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
JEPA 潜空间一直像黑盒，这篇论文加 16 个关系通道再用逆动力学损失，不用关系标注就能让模型理解谁对谁做了什么，实验数据给得挺扎实。
论文提出 Inside-JEPA，在 JEPA 潜在状态内加入有序实体对的关系通道。在 Something-Something V2 上，仅用潜空间预测的关系判别不优于随机头，而逆动力学监督在十个设置中将关系 AUROC 提升 .065-.103。在 RH20T 机器人数据集的 held-out 任务上提升 .081，使用现成检测与跟踪在五个关系上提升 .050-.061。单步关系目标下，16 通道的监督读出在 65 个动作中把正确动作排进前五的比例为 43%，高于仅预测潜表示的 24% 和频率先验的 17%。
Inside-JEPA: From Prediction to Relational Grounding in Latent World Models
Joint-embedding predictive architectures (JEPAs) learn world models in latent space, enabling efficient planning but leaving their states opaque. We introduce Inside-JEPA, which places relational channels for ordered entity pairs inside the JEPA latent state and shows that an inverse-dynamics loss can ground these channels without relation labels when its head reads them, either alone or as part of the full state. On Something-Something V2, latent prediction alone performs no better than a random relation head, while inverse-dynamics supervision that recovers actions from channel changes improves relational AUROC by .065-.103 across ten settings with annotated boxes, by .081 on RH20T robot episodes with held-out tasks, and by .050-.061 across five relations using off-the-shelf detection and tracking. An inverse head on entity latents decodes actions more accurately but barely improves relational grounding; shuffled labels, two label-free objectives, and action labels on only 5% of the pairs also fail to ground the channels. Given an action, predicted relational changes approach the quality of observed future relations (AUROC .599 vs .621). For one-step relational goals, a supervised readout of the 16 channels ranks the correct action among the top five of 65 in 43% of cases, compared with 24% for prediction-only latents and 17% for a frequency prior.
