Groq 下一代推理芯片计划堆叠 DRAM 突破 SRAM 容量瓶颈
Nvidia 花了 200 亿美元买 Groq 技术，这块新芯片要用堆叠 DRAM 解决内存太小的问题，做推理的值得看看。
据行业消息，Groq 计划在下一代推理芯片中堆叠 DRAM，以解决片上 SRAM 容量过小的问题。当前 Groq 3 LPU 的 500MB SRAM 提供 150TB/s 带宽，约为 Rubin GPU HBM 带宽的 7 倍，但一个 256 芯片机柜总容量仅 128GB，跑万亿参数模型约需 2000 颗芯片。报告中的 3DS 设计将 DRAM 芯片垂直连线但保持标准 DRAM 接口，只提升容量；混合键合方案则可实现约 10 倍于 HBM 的带宽密度，每 bit 能耗降低 77% 至 83%。Nvidia 去年 12 月以 200 亿美元授权 Groq 技术并雇用其高管，Cerebras 也计划在 CS-6 晶圆上堆叠 DRAM。
Groq plans to stack DRAM in its next inference chip to get around SRAM’s small capacity, industry sources say.
Nvidia paid $20B last December to license Groq’s technology and hire its leaders, so this chip is effectively Nvidia’s next LPU.
The big deal here is that, each generated token requires reading the model’s weights from memory, so bandwidth sets speed.
The big deal here is that each generated token requires reading the model’s weights from memory, so bandwidth sets speed.
Today’s Groq 3 LPU reads its 500MB of on-chip SRAM at 150TB/s, about 7 times a Rubin GPU’s HBM bandwidth, which is fast enough for thousands of tokens per second.
That memory is so small, though, that a full 256-chip rack holds 128GB, less than one Rubin GPU’s 288GB, and an 8-bit trillion-parameter model needs roughly 2,000 chips just for its weights.
Stacking DRAM on the logic die can put far more memory beside the compute, and hybrid-bonded versions promise roughly 10 times HBM’s bandwidth density with 77% to 83% less energy per bit.
That would let fast inference on large models run on far fewer chips.
The reported design, known as 3DS, is more modest: it wires DRAM dies vertically through the silicon and keeps a standard DRAM interface, adding capacity mainly at ordinary DRAM speeds.
Groq hasn’t announced it, and earlier supply-chain reporting described another route: SK Hynix DRAM fused directly onto the logic die by hybrid bonding, alongside SRAM.
Cerebras plans to stack DRAM on its CS-6 wafer too, a sign the capacity ceiling is binding across SRAM-first designs.
So the bonding method decides the outcome: 3DS mostly adds capacity at DRAM speeds, while hybrid bonding promises bandwidth much closer to SRAM.
