手机端本地AI功能或受新型模型加速
curious if Jev-like models might accelerate us towards on-device AI native functionality on mobile d...
Andrew Chen觉得手机本地AI功能可能受新型模型加速，值得看看。
手机运行强大LLM面临内存带宽慢、需高度量化MoE模型、功耗发热等问题。为解决此，人们正开发针对较小LLM的下一代移动NPU。许多移动端UX（如通知、输入、应用内体验）可受益于快速廉价AI决策模型。
curious if Jev-like models might accelerate us towards on-device AI native functionality on mobile d...
curious if Jev-like models might accelerate us towards on-device AI native functionality on mobile devices strong LLMs are a long way from running on phones: - slow memory bandwidth - only highly quantized MoE models will fit - lots of issues with power/heat/etc Folks are building NPUs for the next gen of mobile but targeting pretty modestly sized LLMs There’s lots of mobile UX that would benefit from fast/cheap AI decision models - from notifications, typing/texting, to in-app experiences like inboxes/calendars/etc I always figured the other solution would be a model-on-a-chip that would encode an older-yet-useful model into the actual phone hardware itself, but maybe this will move faster? 💬 0 🔄 0 ❤️ 0 ⚡
