Local LLMs: GPU speed and cost

People run local LLMs on all kinds of GPUs. But performance varies considerably, and so do hardware prices and cloud rental rates.

A faster GPU is useful, but it may be unnecessary if a cheaper option already meets your needs. On the other hand, saving money on hardware is less appealing if every response leaves you waiting.

What we wanted to understand was straightforward: How quickly does a particular model respond on different hardware, and what does that performance cost?

Individual benchmark results are helpful, but comparing them is difficult when prompt lengths, output lengths, and concurrency differ. We therefore tested two Qwen models across a range of GPU configurations, using the same measurement procedure.

For each model, we selected up to eight distinct GPU configurations from Dorbit’s available options priced at $2 per hour or less, starting with the cheapest. We rented the machines through cloud providers, including configurations with consumer GPUs that could also inform a local hardware purchase.

The prices below are Dorbit’s quoted prices at the time of testing in October 2026. Availability and pricing can change.

We focused on individual use: one request at a time, with input lengths of 1,024, 4,096, and 16,384 tokens and an output length of 256 tokens. Each condition included two warm-up requests followed by five measured requests using GuideLLM 0.7.4.

The tables show median results for 4,096-token inputs. Time to first token measures how long the response takes to begin. Generation speed measures output after that first token. Measurements ran within each GPU node, excluding internet latency.

We started with Qwen3.8-27B, a dense model.

Unlike a mixture-of-experts model, a dense model uses the main weights in each layer rather than selecting a subset of experts. We ran its unquantized BF16 version using vLLM 0.27.1. This is a different model and precision from the quantized MoE tested later.

Similar prices did not necessarily buy similar performance. The two RTX PRO 5000 GPUs generated tokens about 1.6 times as quickly as the two L40S GPUs, despite a relatively small difference in hourly price.

Response time also tells a different story from generation speed. In this test, two A100 40GB GPUs generated tokens faster than one A100 80GB GPU, but took considerably longer to begin responding. For frequent short interactions, choosing solely by tokens per second can give a misleading picture of the experience.

Next, we tested Qwen3.8-Flash-Next in its IQ3_S quantized format, using Strata 0.1.41.

This is a much larger mixture-of-experts, or MoE, model. An MoE contains many experts but activates only a subset for each token. Its total parameter count can therefore be large without requiring every parameter to participate in every computation. Official model description

Strata takes advantage of this structure. It keeps frequently used experts on the GPU, holds other experts in system RAM, and uses the CPU to compute experts that are not cached on the GPU. A large lookup table can remain on SSD. This spreads the model across the machine’s available resources. How Strata works

Combined with quantization, this makes models accessible that would otherwise exceed a consumer GPU’s memory capacity.

*The RTX 5070 Ti result comes from an earlier run using the same input length, output length, and concurrency.

The consumer GPUs produced impressive results: approximately 64 tok/s on an RTX 5070 Ti and 130 tok/s on an RTX 5090. A model of this scale can deliver substantial generation speed without placing all its weights in expensive GPU memory.

These are whole-machine results, however. Strata uses CPU offloading, so the CPU and system memory matter alongside the GPU. Our RTX 5090 machine, for example, used an AMD EPYC 7742 processor. A desktop with the same graphics card may perform differently.

The price comparison is revealing, too. The RTX 5090 generated tokens about 28% faster than the A40, while costing approximately 2.36 times as much per hour. It also began responding sooner. Which option makes sense depends on how much you value those improvements.

What about the quality lost through quantization?

For the IQ3_S model we used, its publisher, ISTA-DASLab, provides a direct comparison with BF16. Both scored 100.00 on AIME25. On LiveCodeBench v6, BF16 scored 87.43 and IQ3_S scored 86.86—a difference of 0.57 points. Within these published evaluations, the quantized model remained close to the original. Quantization evaluation results

The publisher lists a reduction in model size from 354GB to 83.6GB. IQ3_S uses different precision assignments across weight tensors; it does not simply convert every weight to three bits. Formats and file sizes

Those quality results come from the model’s publisher. Our tests measured speed, and quality on any particular workload still needs to be checked. Nevertheless, retaining similar benchmark scores while substantially reducing memory requirements makes this approach compelling for local use.

Our measurements are a small comparison, with five requests per condition. They include configuration differences, software fixes, and retries, so they should not be treated as a ranking of each GPU’s maximum performance.

They do show why testing the intended model matters. Specifications and hourly prices tell only part of the story. Before buying a GPU, renting a few candidates and testing them with representative inputs can reveal which configuration actually meets your needs. We plan to keep building that body of comparison data through Dorbit.