研究揭示语音分词器语义层脆弱性，提出 T-SemAttack 攻击与 ROSETok 防御
Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer
做语音识别或音频大模型的人该看看这篇：论文演示了小扰动就能让 9 个 ASR 系统和 4 个 LALM 出错，还公开了攻击代码和演示。如果你在用开源语音分词器搭 SpeechLLM，ROSETok 的训练思路可以直接参考，比泛泛的安全讨论实用得多。
arXiv 论文（编号 2611.14929）指出语音分词器在语义层面存在脆弱性：微小扰动即可干扰语义编码过程，导致下游 ASR 与大型音频语言模型（LALM）性能严重下降。作者提出可迁移语义攻击 T-SemAttack，基于语义编码器集成（ensemble），通过联合漂移多个语义编码器的表示空间来破坏语音的语义内容，同时保持感知质量。
技术机制上，论文分析了逐层表示漂移与跨模型梯度对齐：扰动可使深层语义表示发生大幅偏移，不同代理编码器的输入梯度呈现方向对齐，说明输入空间中共享的对抗敏感方向可能是攻击跨模型迁移的关键。防御方面，作者提出 ROSETok，将噪声训练与鲁棒语义蒸馏结合，兼顾重建保真度与下游 SpeechLLM 鲁棒性。
实验覆盖多个数据集、9 个开源与黑盒 ASR 系统以及 4 个 LALM，验证了 T-SemAttack 的迁移攻击效果和 ROSETok 的防御能力。代码与演示已在项目页公开。
Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer
Speech tokenizers serve as speech front ends, encoding input speech signals into representations for downstream models, such as automatic speech recognition (ASR) systems and large audio-language models (LALMs). In this paper, we find that speech tokenizers are fragile at the semantic level: small perturbations can disrupt their semantic encoding process and consequently cause severe degradation in downstream ASR and LALM performance. To study this vulnerability, we propose T-SemAttack, a transferable semantic attack based on a semantic encoder ensemble. By jointly drifting the representation spaces of multiple semantic encoders, T-SemAttack effectively disrupts the semantic content of speech while preserving perceptual quality, thereby inducing error features. We further analyze layer-wise representation drift and cross-model gradient alignment. Our findings show that perturbations can induce substantial shifts in deep semantic representations and that the input gradients of different surrogate encoders exhibit directional alignment. These observations suggest that shared adversarially sensitive directions in the input space may be a key factor underlying cross-model attack transferability. Building on these, we introduce ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation to improve reconstruction fidelity and downstream SpeechLLM robustness. Extensive experiments on multiple datasets, 9 open-source and black-box ASR systems, and 4 LALMs demonstrate that T-SemAttack achieves strong transferable attack performance, while the proposed Robust Speech Tokenizer exhibits robustness under high-fidelity reconstruction and downstream tasks. Our code and demo are available at https://wxzyd123.github.io/t-semattack.
本条情报归属 AITOP「论文」核心栏目，重点聚焦中国 AI 产业与国产大模型生态。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」「中美大模型技术代差与商业化对照」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：语音分词器、ASR、对抗攻击、SpeechLLM，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。
信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Zhisheng Zhang, Yifan Mi, Yixuan Zhou, Haiyun Li, Shengbo Cai, Dongfu Song, Ziang Xu, Jie Hao, Zhiyong Wu），经 AITOP 多维评分管线结构化提炼。