研究：推理模型写故事的心理深度未必更强
Does Reasoning Improve Psychological Depth in Large Language Models?
有人测了推理模型写故事到底好不好，GPT-5 略胜但 DeepSeek-R1 反而输给自家旧款，还发现 LLM 评委偏爱推理模型，结论挺有意思。
一项研究通过60次盲测对比了推理模型在短篇小说写作上的表现：GPT-5 略优于 GPT-4o（60.0-62.9%），而 DeepSeek-R1 反而落后于 DeepSeek-V3（42.9%）。人类标注者的 Krippendorff's α 仅 0.070，说明评价高度主观。LLM-as-a-Judge 则表现出强烈的推理偏好偏差，98.3% 的对子倾向推理模型，远高于人类的 55.2%。
Does Reasoning Improve Psychological Depth in Large Language Models?
Benchmark gains from reasoning-enhanced large language models encourage the assumption that these models should broadly perform better on complex generative tasks, but this remains underexplored in open-ended, creative domains shaped by nuances of reader experience. We tested this assumption through the lens of psychological depth in short stories. We compared DeepSeek-R1 vs. DeepSeek-V3, our closest available contrast for reasoning-oriented post-training, and GPT-5 vs. GPT-4o, through 60 blinded pairwise evaluations in which seven human annotators and an ensemble LLM-as-a-Judge rated prompt-matched story outputs. Human evaluations showed no universal reasoning advantage: GPT-5 was slightly preferred over GPT-4o (60.0-62.9% across two reasoning-effort settings), whereas DeepSeek-R1 fell behind DeepSeek-V3 (42.9%). Although this pattern was marked by low inter-annotator agreement (Krippendorff's $α= 0.070$), our annotation-quality checks suggested genuine evaluative subjectivity, plausibly reflecting distinct annotator decision schemes rather than annotation failure. In contrast, the LLM-as-a-Judge showed a strong pro-reasoning bias (98.3% of pairs vs. 55.2% of human votes), consistent across its five configurations and partly tracking sentence length. These findings suggest that, at the frontier, reasoning-related quality differences are small enough to be dominated by reader subjectivity, and that LLM-as-a-Judge evaluators can obscure this subjectivity in nuanced creative-generation tasks.