ROSE模型引擎统一LLM与嵌入计算
ROSE is the model engine. It reuses the same kernels for LLMs and embeddings. For embeddings, it sk...
精选理由
Perplexity发布的ROSE引擎，用一套内核同时处理LLM和嵌入，节省计算资源。
ROSE是新型模型引擎，复用相同内核处理LLM和嵌入任务。嵌入任务中，ROSE跳过KV缓存，使用不规则注意力替代分页注意力机制。该引擎支持多种注意力后端，内核选择取决于模型形状和序列长度。
原文 · Perplexity
ROSE is the model engine. It reuses the same kernels for LLMs and embeddings. For embeddings, it sk...
ROSE is the model engine. It reuses the same kernels for LLMs and embeddings. For embeddings, it skips the KV cache and uses ragged attention instead of paged attention. ROSE supports multiple attention backends, so kernel choice depends on model shape and sequence length. 💬 1 🔄 0 ❤️ 4 👀 294 📊 1 ⚡
