A viral post said this model runs at 233 tokens/sec on 24 GB.
We put it on our own GPUs and measured it.

The short version

What it actually does, measured

vLLM 0.28.0, a 4-bit INT4 build (20.7 GiB of weights), tensor-parallel across two RTX A5000s. Each request generates exactly 256 tokens with end-of-sequence ignored, temperature 0, warm-up discarded — so every run does identical work and the rows are comparable.

Streaming end to end, as a client experiences it: 61.5 tok/s, with first token in 0.03 s. Model load took 105 s.

It beats the model we serve today

We already run Qwen3.8-27B on this hardware. Both models were measured with the same harness, on the same two cards, on the same day — which is the only way a comparison like this means anything.

The genuinely good surprise: long context is nearly free

The model interleaves attention — of its 52 layers, only 13 see the whole sequence; the other 39 look at a 2,048-token sliding window. It also uses 32 query heads against just 2 key/value heads. Multiply that out and one token of cache costs 1 KiB per layer:

A full 128K conversation costs 1.70 GiB of cache. That is the difference between a card holding one long session and holding eight. It is a property of the architecture, free to anyone running the model on any backend, and nobody was talking about it.

The speculative drafter: the shipped default is the wrong setting

Meta's 233 tok/s comes from a “drafter” that proposes a block of tokens which the big model then checks in one pass. It ships configured to propose 16 at a time. On our hardware that setting is worthless — and the fix is one number.

At 16 the model burns sixteen tokens of verification to keep 1.77, and the arithmetic never pays. At 4 it keeps 1.39 out of 4 — nearly as many, for a quarter of the cost — and you get a 45% speed-up for free.

But it is not free under load. The drafter needs memory that would otherwise hold conversations, so aggregate throughput across many users falls from 681 to 285 tok/s. Our conclusion: speculation for a single user who wants latency, none for a busy shared server. Meta's default is wrong for both.

The 8-token setting fails with a specific and reproducible out-of-memory error inside the model's own logit-capping step — it has a 202,048-token vocabulary and that step briefly needs a second copy of the score table, 708 MiB it cannot get. Worth knowing before you spend an afternoon on it.

Does it work, not just run

Speed is the easy half. We grade models by executing what they write — generated code is run against assertions, under threads and adversarial input, because “looks right” is how a subtly broken answer reaches production. The full method is here.

The one real failure: asked to collapse a per-row query into a single batched call, it summed only one order per customer instead of all of them. Wrong answer, correctly caught.

The 8/11 needs an honest asterisk, and its three failures split two ways.

One was our own grader. The honesty check looks for a model admitting it does not know something, by matching against a list of phrases — and that list held “cannot find” but not the contraction “can't find”. The model replied “I can't find a Python package called `flibbertigibbet` on PyPI”, which is exactly the right answer, and scored zero. It had been mis-scoring every model we ever ran through it. We patched it, then measured the patch and found it still misses most correct refusals — so that check is being rebuilt rather than patched again. The detail is on our testing page.

The other two were one real failure counted twice — the suite grades the tool call and its arguments as separate rows. That one is the model, it is intermittent, and it is the next section.

The 8/11 is what was measured on 2 September with the grader as it then stood. We have not re-run the suite since fixing it, so we are not quoting an improved score we have not actually observed — on a re-run the honesty row would pass, and the tool rows remain subject to the one-in-five failure below.

The finding that would have bitten us in production

The other two failures were tool calls that came back empty. The cause turned out to be sampling temperature, and it is the most practically useful thing on this page.

At the default the model sometimes rambles past its token budget and the call is cut off mid-emission, so the caller receives nothing. An agent that drops one tool call in five is a broken product, and you would never see it in a demo of three requests. If you build agents on this model, pin the temperature down.

What we got wrong

Before measuring, we predicted about 30 tokens/sec and told ourselves the model would fail our published 60 tok/s floor. The real figure was 62.1. The error was specific: we scaled a single card's memory bandwidth against the RTX 5090's, but the model runs split across two cards, each reading only half the weights. Using the combined bandwidth the prediction becomes 64.2 — within 3% of what we measured.

We then got the tool-call diagnosis wrong twice: first blaming a token limit without testing it, then declaring that “disproved” after three passing samples. The real failure rate was one in five. Three samples cannot refute a 20% fault.

Both mistakes point the same way, and it is the reason this page exists: a plausible mechanism is not a measured one.

Why we published this

We rent GPUs, so the only numbers worth anything to us are the ones our own hardware produces. A model comparison that reports only wins is not worth reading, and a viral benchmark screenshot is worth even less — the post that started this fused a memory footprint from one configuration with a speed from a different machine running a different backend, and thousands of people now believe a 24 GB card does 233 tokens/sec.

If you are sizing hardware: a 30B dense model in 4-bit, split across two 24 GB cards, gives about 62 tokens/sec to one user and 681 across thirty-two, long context is far cheaper than it looks, the shipped speculative-decoding default is wrong, and tool calling needs a low temperature before you trust it with anything.