论文

Social Signal Bench:47小时音视频社交信号评测基准发布

Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

精选理由

做对话分析、客服质检或者情感计算的同学可以看看这个基准。它把兴趣、犹豫这类容易混淆的共现信号单独拎出来评,还配了个比普通 F1 更讲道理的 EW-F1 指标。7B 开源模型微调后能打过前沿闭源模型,说明这个方向自己练小模型完全可行。

研究团队发布 Social Signal Bench,一个总计 47 小时、稠密标注的社交信号识别评测基准,包含 25 小时的视频数据集(覆盖 10 种社交信号)和 22 小时的音频数据集(覆盖 6 种信号)。

正文

训练数据来自在线对话视频,其中 10 小时经专家标注,其余为伪标签,测试集由专家标注并覆盖四个对话领域,且不公开。论文同时提出 Evidence-Weighted F1(EW-F1)指标,按标注者证据为参考标签加权,并通过逐片段最优匹配给语义相关的预测赋予梯度得分,扰动实验显示它能区分语义相关错误与无关错误。

在统一评测协议下,基于该基准后训练的 7B 开源模型在视频任务上超过通用前沿模型,在音频任务上与之持平。

原文 · arXiv cs.LG

Social Signal Bench: Evidence-Weighted Evaluation of Co-occurring Social Signals in Video and Audio

Recognizing social signals such as interest and hesitation requires interpreting context-dependent behaviors that may co-occur in conversation. Existing affective computing benchmarks rarely evaluate such co-occurring signals, and conventional multi-label F1 overlooks semantic relationships and annotator disagreement. We introduce Social Signal Bench, a 47-hour densely annotated benchmark for the evaluation of social signal recognition and engagement, comprising two datasets: Social Signal Bench Video (25 hours), covering ten social signals in audiovisual recordings, and Social Signal Bench Audio (22 hours), covering six signals in audio-only recordings. Each dataset pairs a public training set of online conversational video, of which 10 hours are expert-annotated and the remainder pseudo-labeled, with a private expert-labeled test set spanning four conversational domains. We also propose Evidence-Weighted F1 (EW-F1), which weights reference labels by annotator evidence and assigns graded credit to semantically related predictions through optimal per-clip matching. Controlled perturbations demonstrate that EW-F1 distinguishes semantically related errors from unrelated ones, unlike conventional F1. Under a shared evaluation protocol, 7B open models post-trained on Social Signal Bench outperform general-purpose frontier models on video and match them on audio.

产业坐标与信源核验

本条情报归属 AITOP「论文」核心栏目,重点聚焦北美前沿实验室与硅谷科技巨头。在跨区域产业观察坐标中,该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线,反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括:Social Signal Bench、EW-F1、情感计算、多模态,读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明:本文原始线索采编自「arXiv cs.LG」(@Siddharth Ravi, Anders R. Bargum, Hugo Anselme, Patrizio Cugia di Sant'Orsola, Alberto Cardaci, Flora Kincso Parragi, Vincent Borowiec, Line H. Clemmensen),经 AITOP 多维评分管线结构化提炼。