NVIDIA × Groq 3 LPX Rack: 35× Tokens per Watt

Quick Answer

At GTC 2026, NVIDIA and Groq announced the Groq 3 LPX rack — 256 LPU inference accelerators designed to sit beside Vera Rubin racks, delivering 35× higher tokens-per-watt on LLM inference (NVIDIA newsroom, 2026). The partnership combines NVIDIA's Rubin serving as prefill/context engine and Groq's LPU handling the decode phase.

Data last verified September 9, 2026 from NVIDIA newsroom and Groq.

Why a hybrid LPU + GPU rack

The LLM inference pipeline has two phases:

• Prefill — processing the input prompt (parallel-friendly, GPU-native).
• Decode — generating tokens one at a time (sequential, latency-sensitive, low-utilization on GPU).

GPUs excel at prefill but waste 70-90% of FLOPs during decode (memory-bound, not compute-bound). Groq's LPU is purpose-built for decode — deterministic dataflow architecture with on-chip SRAM, no scheduler stalls (NVIDIA, 2026).

The LPX rack sits beside a Vera Rubin rack: Vera Rubin handles prefill (high-throughput), Groq LPX handles decode (low-latency). The combined system delivers 35× the tokens-per-watt of a pure GPU rack on real workloads.

Architecture

Source: Groq + NVIDIA GTC 2026 (2026).

Deployment model

The LPX is sold as an add-on to existing Vera Rubin deployments. Customers buy:

1. Vera Rubin NVL72 rack(s) — for training and prefill inference.
2. Groq LPX rack(s) — for high-QPS decode inference.
3. NVIDIA NIM software stack — unified orchestration across GPU + LPU.

This is a software-defined hybrid: NIM routes requests dynamically based on token phase, latency targets, and cost optimisation.

Customer impact

For real-world workloads:

• Chatbot serving — 100ms time-to-first-token vs 400ms on pure GPU.
• Cost — 35× lower per-token cost on decode-heavy workloads.
• Throughput — 50k+ tokens/sec per rack on Llama-3.1 70B.
• Energy — 10 kW per rack vs 50+ kW for equivalent GPU-only inference.

Competitive context

The inference accelerator market has split into three camps:

1. GPUs — NVIDIA dominant, AMD MI series for budget.
2. Dataflow accelerators — Groq, Cerebras, SambaNova, Tenstorrent.
3. Custom silicon — Google TPU, AWS Trainium, Microsoft Maia, Meta MTIA.

Groq's partnership with NVIDIA is unique — instead of competing head-on, Groq becomes a co-processor in the NVIDIA stack. For NVIDIA, this widens the moat: customers get the best of both architectures under one procurement relationship.

Why this matters

AI inference is the dominant compute workload in 2026 — Jensen Huang said at GTC that inference now exceeds training in NVIDIA's compute mix. The hybrid LPU + GPU rack is NVIDIA's bet on the inference-economics inflection: when tokens become the unit of revenue, the cost-per-token matters more than peak FLOPs.

For the broader NVIDIA stack, see Jensen Huang's AI factories / token economy thesis.

Why NVIDIA partnered with a competitor

NVIDIA + Groq is unusual - Groq competes with NVIDIA in inference. The partnership makes sense because:

1. Different architecture - Groq is dataflow (deterministic, sequential), NVIDIA is SIMD (parallel). Different problem profiles.
2. NVIDIA needs LPU complement - decode inference is GPU-weak; NVIDIA doesn't have a competitive LPU today.
3. Groq gets NVIDIA distribution - Groq cannot match NVIDIA's hyperscaler relationships alone.

Why not Cerebras or SambaNova

Cerebras and SambaNova compete more directly with NVIDIA in training + inference. Cerebras WSE-3 targets large model training, where NVIDIA Vera Rubin is strongest. SambaNova RDU focuses on enterprise/government, a market NVIDIA addresses through DGX. Groq is the cleanest complement.

Customer economics

For an AI startup running 100M tokens per day on Llama-3.1 70B:

Source: vendor pricing + industry benchmarks (2026).

What it means for the inference market

By 2027, expect most production AI deployments to be hybrid GPU + LPU. NVIDIA's bet on Groq positions it as the platform of record for both architectures. AMD has no equivalent partnership - a competitive gap.

Why this matters for the broader AI ecosystem

This announcement fits into a larger pattern of the 2026 AI industry consolidation wave. Frontier labs (OpenAI, Anthropic, Google DeepMind, NVIDIA, xAI) are racing to capture the next platform shift while regulators, open-source competitors, and enterprise customers apply pressure from all sides. The three forces shaping the industry in 2026-2028 are: (1) inference cost compression (Vera Rubin driving 35x token cost reduction), (2) agent capability maturity (GPT-6 Astra, Claude Opus 4.5, Gemini 3.8 Flash all shipped in 2026), and (3) sovereign AI deployment (US Stargate, Saudi HUMAIN, UAE G42, India IndiaAI collectively committing over $200B).

For developers and businesses, the practical implications are concrete. Enterprise AI deployments are moving from pilot (2024-2025) to production (2026-2027). The key questions for any CTO evaluating AI in late 2026: which model(s) for which workload, how to handle data residency, how to manage agent risk, and how to measure ROI. The answers vary by industry - financial services prioritises compliance and auditability, healthcare prioritises privacy and FDA pathways, retail prioritises personalisation and unit economics.

What to watch next

Three upcoming events will validate or revise this analysis:

1. NVIDIA GTC Berlin (Oct 20-22, 2026) - European AI sovereignty + Vera Rubin EU rollout.
2. Made by Google October 2026 - Pixel 11, Gemini Spark 2, Android XR 2 launch.
3. AWS re:Invent (Nov 30 - Dec 4, 2026) - Trainium 4 announcement + AI infrastructure roadmap.

Cross-references

For related TutorsBot coverage, see our guides on Jensen Huang's $1T order outlook, Vera Rubin shipping Q3 2026, and the 2026 AI chip war landscape. For the broader market context, our analysis of NVDA's $4.5T market-cap trajectory and the AI factory / token economy thesis provide the strategic context.