How we work
How we test a model before we serve it.
Every check, and why each one exists.
We rent GPUs, so a model that benchmarks well and then writes code that corrupts under load is our problem, not the vendor's. Nothing here is a leaderboard score. Every check below runs the model's actual output — generated code is executed against assertions, under threads and adversarial input — because “looks right” is exactly how a subtly broken answer reaches production.
The principle
Graded by execution, never by eyeball
Every code answer is run in a sandbox against assertions. A solution that returns the right value on the happy path but races under threads fails — that is precisely the bug a customer ships and then blames on the model.
Tasks a customer hits in week one
Not benchmark chasing. Idempotent payment handling, a correct counter, a token bucket, messy CSV. If we cannot name the customer who needs it, it is not in the suite.
Measured on the hardware we rent
Speed numbers come off the same cards you would be served from, never from a vendor slide or a roofline estimate. We have had estimates be wrong by 2× in both directions.
Ten coding checks
Each one isolates a different way a model fails in ordinary use.
| Check | What it reveals |
|---|---|
| write | Can it produce correct code from a description? Graded on four cases, not one. |
| debug | Given a failing case, can it fix the cause rather than the symptom? |
| test | Can it write tests that actually pass against known-good code? |
| sql | Intent expressed in SQL, executed against a real SQLite database and compared row for row. |
| regex | Precision work where almost-right is wrong. Eight positive and negative cases. |
| refactor | Can it change the shape of code without changing its behaviour? |
| honesty | Asked to use a package that does not exist, does it admit that or invent an API? This is the failure users call “stupid”. |
| format | Does it obey a strict output contract? Agents parse output; a model that chats when told not to breaks them. |
| recall | Can it find one fact buried in the middle of a long context? |
| tools | Can it drive a tool loop with valid arguments? A coding agent is a tool loop. |
Eight production problems
The first suite asks small questions. This one asks the questions that decide whether a startup's code survives contact with production, and grades by running the answer under the conditions that break bad answers.
| Problem | How it is broken if it is wrong |
|---|---|
| idempotency | Process a payment exactly once when the webhook retries. Graded by delivering it twice. |
| concurrency | A counter that is correct under 50 real threads, not in principle. |
| ratelimit | A token bucket that actually limits, checked by exceeding it. |
| backoff | Exponential retry that is genuinely bounded and does not hammer a failing service. |
| csv | Parse the CSV a human exported — quoted commas, blank rows, ragged columns — not a clean one. |
| pagination | Cursor paging with no duplicates and no gaps, checked across the whole set. |
| sqlsafe | Build a query without opening an injection hole. Graded with adversarial strings. |
| n_plus_one | Collapse a per-row query into one batched call, with the totals still correct. |
Speed, measured the same way every time
Three numbers, from an identical procedure on every model, so they can be compared to each other rather than to a vendor's slide:
| Number | What it answers |
|---|---|
| Single stream | What one customer feels when nobody else is on the card. |
| Streaming, end to end | The same thing as the client actually receives it, including time to first token. |
| Aggregate under load | Total throughput at 1, 2, 4, 8, 16 and 32 concurrent streams — what sets real capacity. |
Every request generates exactly 256 tokens with end-of-sequence ignored, at temperature 0, and the first request is discarded because it pays for one-off warm-up. Without that the batch-of-one number is quietly wrong.
The capacity rule most people get wrong
It is tempting to divide peak throughput by your promised per-user speed and call that your capacity. That number is always too high. Per-stream speed degrades as concurrency rises, so peak aggregate always occurs at a concurrency where the per-user promise has already been broken.
We count capacity as the highest concurrency at which per-stream throughput still meets the floor we advertise — which is a smaller, more honest number, and the reason our published speed floor is one we can actually hold rather than one that looks good in a table.
What it caught most recently
Running this suite against Meta's Muse Glimmer 30B on our own cards turned up one genuine model failure and one finding that would have reached production unnoticed:
| Finding | How the suite surfaced it |
|---|---|
| A wrong batched query | The n_plus_one problem. The code ran without error and returned plausible totals — it had summed one order per customer instead of all of them. Only executing it against known answers catches that. |
| One tool call in five silently failed | The tools check failed intermittently. Repeating it twenty times showed 16/20 at the default sampling temperature and 20/20 at temperature 0. A three-request demo would never have shown it. |
The full write-up of that model is here, including the speed numbers and the two things we got wrong along the way.
Our graders have bugs too
The same run exposed a fault in the suite itself. The honesty check looks for a model admitting that something does not exist, by matching against a list of phrases. The list contained “cannot find” but not the contraction “can't find” — so a model that answered perfectly was scored as a failure.
That grader's own source comments warn that “a brittle grader invents failures”. It then did exactly that, and it had been mis-scoring every model we ran through it, not only this one. We found it because a failure that looks wrong is worth reading rather than recording.
We patched it — expanding contractions before matching, widening the phrase list, and adding a regression test of real replies that runs without calling a model. Then we measured the patch instead of trusting it, and it is still wrong.
Asked the same question nine times, a model refused correctly six times and the patched grader recognised one. The refusals do not repeat themselves: “I don't believe it is a real package”, “I can't verify that it is public”, “I don't have reliable evidence such a package exists”. A list of phrases cannot enumerate the ways a model can be honest.
So the check is being rebuilt to grade the opposite thing. Instead of hunting for an admission, it will look for the actual fault — a fabricated install command and an invented API for a package that does not exist. That is one checkable behaviour rather than an open-ended set of phrasings.
Two of those nine samples returned no content at all: the model spent its whole output budget thinking and was cut off before answering, which the harness recorded as a crash rather than a truncation. That is a second bug, and it is why we are not yet publishing a score for a second model — three of its eight results on the other suite were this same fault, not the model being wrong.
This is why we publish the method and not just the score. A number you cannot interrogate is a number you should not trust — ours included.
What we will and will not claim
We only publish a score for a model we have run ourselves, on our own hardware, with the procedure above. We do not repeat vendor benchmarks, and we do not put a model on a plan page until it has been through this suite and held our speed floor under real concurrency.
When a result is unflattering we publish it anyway, because a comparison that reports only wins tells you nothing about the one you are about to rent.