研究发现 Qwen 单个神经元可编码多个无关概念
一个神经元只对 Tornado 框架有反应，对龙卷风无感，这条推把 Qwen 可解释性新进展讲得挺生动。
这条推文讨论一篇关于 Qwen 神经元可解释性的论文，作者 Nick 观察到某些 Qwen 神经元存在第二个无关概念的较小峰值。文中举例一个神经元对 Tornado web 框架反应强烈，但对真实龙卷风、虚构同名框架、任意 web 框架均无反应。作者认为这比此前“神经元是任意概念叠加”的认知更容易解读，并将这种状态称为 polysemantic（多义）。
this is definitely “polysemantic”. it’s important to note this so critics don’t accuse you of drawing schizophrenic connections to support any theory. but if you read Nick’s paper he claims that some qwen neurons have a second unrelated smaller peak; that is still a much better interp world than we lived in last week, where we thought neurons could be an arbitrary mismash of superposed concepts. hell even if there are 3 or 5 we can read the stack of meanings like Lyra’s alethiometer
few interesting things about this neuron: - it really likes the Tornado web framework but doesn’t care about tornadoes - it responds weakly even to fictional software frameworks called Tornado - it gets somewhat less excited by arbitrary web frameworks - there is no connection to Ipython, which was invented in Colorado - there is no connection to Apache, a software license but also a tribe that lives in the American southwest - -ado connection doesn’t exist, “el dorado” does nothing
