RealEmbedBench：评测 LLM 在应用层嵌入式软件任务上的能力

研究团队对六款代表性 LLM 进行评测，发现项目上下文（尤其是局部代码窗口）对代码生成与补全的成败影响显著；模型在状态机建模上表现尚可，但把模型转成可执行代码时能力明显不足。跨 OS 翻译的成绩高度依赖方向，其中从 Harmony 翻译到 RT-Thread 最难，平均通过率仅 0.1667。

作者据此给出实践建议：使用 LLM 做代码生成时应提供足够的局部代码上下文，并按翻译方向挑选模型；同时论文主张在训练数据中加入更多真实状态机实现样本，以缩小建模与落地代码之间的差距。

RealEmbedBench: Benchmarking LLMs on Real-World Application-Layer Embedded Software Tasks

Large language models (LLMs) perform well on general coding tasks, but their capabilities in application-layer embedded software development remain underexplored. Existing embedded benchmarks primarily focus on hardware-in-the-loop validation in simulated environments. We introduce RealEmbedBench, a benchmark of 5,142 samples covering three application-layer tasks: code generation and completion, state machine modeling and implementation, and cross-OS code translation. We evaluate six representative LLMs and find substantial performance differences across tasks and models. Project context, especially the local code window, significantly affects code generation and completion. Models perform reasonably well at state machine modeling but struggle to translate these models into executable code. Cross-OS translation performance varies substantially by direction. Translation from Harmony to RT-Thread is particularly challenging, with an average pass rate of 0.1667. These findings suggest that developers should consider local code context when using LLMs for code generation and completion and select models according to the translation direction. They also motivate the use of more real-world state machine implementation data in LLM training to narrow the gap between modeling and executable code.

本条情报归属 AITOP「论文」核心追踪栏目。在跨区域产业观察坐标中，该动态直接关联「AI 开发者工具链、Coding Agent 与工程化实践」研究主线，反映出当前技术演进与商业落地的最新风向。涉及核心实体与技术议题包括：RealEmbedBench、LLM、嵌入式开发、代码生成，读者可通过下方标签进入对应专题页查阅历史脉络与关联对比。

信源核验说明：本文原始线索采编自「arXiv cs.AI」（@Yujun Huang, Yuwei Zhang, Yi Zhou, Shanshan Wu, Xiang Chen, Lina Gong），经 AITOP 多维评分管线结构化提炼。