Fitting a model's weights into VRAM says very little about how it will run. Qwen 3.8 27B is roughly 17GB as a four-bit quant, which makes it look like an easy fit for any 24GB or 32GB card, and testing it across discrete GeForce cards and the current unified-memory boxes puts delivered performance much closer to the inference engine than to the hardware. The same model on the same RTX 5090 moves between a half-hour wait and an interactive session depending only on which runner loads it.
One card, three runners, and a thirty-minute wait
llama.cpp will allocate Qwen 3.8 27B's full 262K context on a single RTX 5090 and then take roughly 30 minutes to return a first token at long context lengths, with throughput far below what 1.8 TB/s of GDDR7 should deliver, according to Tom's Hardware's sweep across local AI hardware. The same collapse shows up on the RTX 4090 and, oddly, not on the RTX 3090, which points at a software bug rather than a memory or bandwidth ceiling.
vLLM avoids the cliff and is not a desktop-weight stack in exchange: the test system needed another 64GB of swap on top of 64GB of DDR5 just to load the model the first time. Its published single-card recipe caps context at 32K, leaves no room for the model's built-in multi-token prediction, and holds decode near 20 tokens per second, which is a poor return on the fastest consumer card on sale. SGLang ran close to three times faster than that recipe on the same 5090 and reached 37,740 tokens of context.
Reaching the full 262K context took a second card. Two 5090s under tensor parallelism produced 70 to 80 tokens per second of decode across the context sweep, and 100 to 110 with multi-token prediction enabled, while time to first token stayed interactive out to the maximum context length. Replicating that pair costs over $13,000 at the moment, which is the honest shape of the result: the model fits on one card, the configuration that runs it properly does not.
The 24GB cards, and the boxes that trade bandwidth for capacity
The 3090 and 4090 hold the Q4_K_M weights, but only with the KV cache quantized to Q8_0, and context tops out near 112K instead of the model's native 262K. Neither card has memory left for a desktop session alongside the workload, so both want a separate display GPU or a headless machine, which brings PCIe bifurcation and primary-device enumeration problems of its own.
Unified memory inverts the usual ranking. DGX Spark's 273 GB/s is poor for a dense model, yet its prefill is quick enough that time to first token stays interactive out to the full context window, with decode around 20 tokens per second once multi-token prediction is on, and a Spark still sells for about $5,000. The M4 Max Mac Studio has more bandwidth for decode and still spends longer per turn, because prompt processing dominates it, and multi-token prediction there costs performance at short context rather than adding it. AMD's Ryzen AI Halo gets both halves wrong, with low bandwidth and slow prefill, and is the worst case of the group for a dense model this size. None of that ordering is predictable from a spec sheet.
AMD's tower answers the capacity half
AMD used its IFA 2026 keynote to introduce the Threadripper Halo Station, a liquid-cooled tower built to hold large models on the machine rather than call a cloud endpoint. The CPU is the Threadripper PRO 9995WX, 96 Zen 5 cores and 192 threads with eight DDR5 channels, 128 lanes of PCIe 5.0 and support for up to 2TB of system memory. The accelerators are Instinct MI350P cards, the PCIe cut-down of the CDNA 4 MI350X, each carrying 144GB of HBM3E at 4 TB/s and rated at 2.3 PFLOPS of MXFP8, as ServeTheHome recorded from the keynote.
The figure being quoted around this machine, 576GB of GPU memory, needs a qualifier. The system demonstrated at the show had two cards and 288GB. AMD described a path to four, without saying whether it intends to sell that configuration or whether reaching it means a customer adding cards to a PCIe chassis, and the trillion-parameter claim attached to the product rests on the larger number. Pricing, availability, storage, networking and chassis options are all still unstated, along with whether the system goes out direct or through integrators, as TechSpot notes of the same announcement. Each MI350P draws up to 600W, and the demo cooled every processor with its own closed loop instead of one shared circuit, which is what a workstation assembled from off-the-shelf datacenter parts looks like.
For anyone already running a 24 or 32GB card, the useful work is not shopping. It is measuring time to first token at the context length actually in use, with more than one runner, because the gap between llama.cpp and SGLang on identical silicon was wider than the gap between two generations of card.