a few thoughts：预训练科学与可解释性关系的随想
OpenAI 前研究员 roon 发推聊可解释性，说基础突破反而来自公司外部，OpenAI 也在其中，观点挺扎心。
a few thoughts 用蒸汽机先于热力学的类比，指出预训练科学发展路径与热力学相似，方向指向可解释性。作者提到可解释性研究者 Nick，称与他的相遇促成自己加入 OpenAI 的经历。文中估算全球真正观察神经网络成长的研究者很少，多数人转向行为层面的奖励函数与测试分数分析。作者认为像这类基础突破来自大模型公司外部，对 OpenAI 等公司不是好信号。
a few thoughts:
1. pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". my guess would be that major breakthroughs in interp also advance capabilities
2. nick is a good friend of mine, safe to say my origin story at openai wouldn't have happened if i hadn't met him, and it's fascinating watching him operate and seeing the early drafts evolve into this. he's the most profoundly happy guy I know- not driven by guilt, activism, or religious terror, but by fascination. he of course understands the stakes of understanding neural networks, but it doesn't seem like the core reason he does it: he just loves the neural nets. afaict he's posting this from some beautiful hideaway in tokyo. there's a lesson in there for all the +EV maxers. i feel very grateful to have met him
3. i do think it's crazy how few people across the world were staring at neural nets growing: most have moved to a different level of behaviorist analysis, thinking about reward functions and test scores and whatnot. it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out
