Nvidia has detailed the Groq 3 LPX architecture for the first time at Hot Chips 2026, revealing a specialized inference accelerator built around the LP30 chip acquired in the company’s $20 billion Groq deal last December. The Groq 3 LPX architecture ditches traditional GPU memory hierarchies in favor of on-die SRAM and a fully deterministic pipeline, delivering token generation rates that outpace competing cloud endpoints by a significant margin in early benchmarks.
Performance and the Third-Party Benchmark
Artificial Analysis, an independent benchmarking firm, measured the LPX rack at 3,431 output tokens per second on a 100K-context Gemma 4 31B reasoning workload—roughly four times faster than the next-fastest public endpoint at 870 tokens per second. The test ran on a private, pre-release Gemma 4 31B endpoint served through Google Cloud, taking the median of 50 sequential requests at a concurrency of one. Nvidia’s own on-stage demo showed a higher figure of 10,996 tokens per second on the same model, which Igor Arsovski, Nvidia‘s VP of hardware and the former chief architect at Groq, flagged as “self-reported” before emphasizing the company’s commitment to “third-party verified independent benchmarks that you guys can trust.”
The caveat matters: Artificial Analysis tested against shared production serverless endpoints, while the LPX ran in single-request mode—the highest per-user token rate the hardware can post. This setup isn’t directly comparable to the multi-tenant conditions other providers operate under. As covered earlier, NVIDIA Launches DLSS 4.5 Ray Reconstruction for RTX Remix and New Titles showed how specialized hardware can reshape performance metrics in narrow use cases, and the same principle applies here. The benchmark also tested only a dense 31-billion-parameter model small enough to fit inside a single LPX rack; trillion-parameter mixture-of-experts models, where memory capacity becomes the main constraint, remain unaddressed.
Architecture and Design Trade-offs of the Groq 3 LPX Architecture
Each LP30 chip carries roughly 500MB of on-die SRAM and no HBM. A full LPX rack of 256 chips holds 128GB of memory with 40 PB/s of aggregate bandwidth and 315 PFLOPS of FP8 compute. The Groq 3 LPX architecture keeps model weights resident in SRAM rather than streaming them from HBM, eliminating the memory-access latency that dominates single-token decode. The design drops caches, branch prediction, and out-of-order execution in favor of a fully deterministic pipeline that the compiler schedules at clock-cycle granularity—a direct descendant of the Tensor Streaming Processor Groq described in a 2020 ISCA paper.
Capacity is the cost of this SRAM-only approach. A 31-billion-parameter model at FP8 needs roughly 62 LPUs to hold its weights, and large mixture-of-experts models run into four figures of chips across several racks. Nvidia is positioning the LPU as a decode co-processor paired with Vera Rubin NVL72 GPUs, which handle the compute-heavy prefill phase and build the KV cache while the LPUs generate tokens. The company showed three ways to divide the work: disaggregated prefill and decode; attention-FFN disaggregation, which keeps attention on GPU HBM while the LPU runs feed-forward layers; and external-draft speculative decoding, where a small model on the LPU proposes tokens that the GPU verifies in parallel. Nvidia measured gains of roughly three-to-five-times over Rubin alone on a two-trillion-parameter workload with a 400K-token cached context.
Across racks, Nvidia synchronizes chips to a single virtual clock in a plesiosynchronous network, with each chip acting as both processor and router. An FPGA bridges the synchronous LPU domain and the asynchronous world of host I/O and GPU hand-offs, while Nvidia‘s Dynamo runtime and an LPU extension to CUDA orchestrate the split. In a related development, Google Pixel 11 Pro Fold Review: A Solid Effort Facing Tough Competition examined how specialized hardware and software integration shape real-world performance in consumer devices, a principle that extends to data center accelerators as well.
المصدر: Tom’s Hardware
