StreamEval 实时视频基准发布：38 模型、300 条测试用例
清华、NVIDIA、CMU 等机构做了个实时视频基准，300 条测试用例测了 38 个模型，做视频模型的可以提交自己跑分。
StreamEval 是一个针对实时视频 AI 的基准，覆盖 Avatar、世界生成、视频编辑三个类别。基准包含 300 条脚本化测试用例，单个会话时长从 15 秒到 5 分钟，首轮报告了 38 个模型的成绩。评测横跨 5 项任务和 10 个指标，考察模型是否遵循指令、维持画面质量，以及在下一条指令到来时保留此前的编辑结果。代码、数据集和评分规则均已开源，视频模型团队可通过 API 端点提交评测。
StreamEval is worth a look if you’re following real-time video AI.
The paper reports results from 38 models. The benchmark contains 300 scripted test cases, each representing a session lasting from 15 seconds to five minutes.
The leaderboard covers three categories:
- Avatar: dialogue, performance and character consistency. - World generation: changing scenes while keeping them coherent. - Video editing: applying edits while preserving unaffected content.
Across five tasks and ten metrics, it examines whether models follow requests, maintain visual quality and keep the scene consistent. That includes preserving an earlier edit when the next instruction arrives.
The code, dataset, evaluation protocols and scoring rules are public. Researchers can inspect the test cases and run their own evaluations with the open-source implementation.
The project brings together researchers affiliated with Tsinghua University, NVIDIA, Carnegie Mellon University, UC Berkeley, Princeton University, the National University of Singapore, and Reactor.
If you’re building a video model, you can submit an API endpoint through “Evaluate your model”. The team reviews submissions for future evaluations.
https://t.co/8AKklGVE2W
Paper, code and dataset links below.
#StreamEval #StreamEvalBenchmark #streamingvideo
