Same Score, 1/36th the Bill

The bill

The receipts

Two engineers at Quesma made a batch of frontier models play Baba Is You (the puzzle game you win by shoving the rules around the board until “rock is push” becomes “rock is you”) and published the whole table on July 31, 2026. Read the bottom half slowly.

Be careful with the pass rates: the suite is eight puzzles, so 75% against 67% is a margin of well under one puzzle and nowhere near a clean win. Piotr Migdał and Piotr Grabowski chose their words precisely. It “reaches the level of Sonnet 5, GPT-5.6 Luna and Grok 4.5 at roughly 1/40 of their cost!” Then one flourish: “A true cost-effective marvel, a Pareto frontier hero.”

The scores are a tie. The bill is not: one thirty-sixth of Sonnet 5's, one fifty-first of Luna's.

The same weekend, one engineer posted thirty days of production usage: “Cost: $4.55USD, API requests: 3,467, Tokens: 323,183,886.”

Someone replied with a bigger meter. “Those are rookie numbers.” Twelve days, 7,877 requests, 2,116,598,952 tokens, total spend $19.27.

Two billion tokens for less than a pizza, near $9 per billion against the $343 per billion Artificial Analysis paid running its own index. Cache hits and input-heavy workloads explain the gap. The poster immediately undercut himself: “Ok this is a bit of lie, a lot of my tasks are very experimental loops whose 99% output is like rubbish.”

What changed

The body didn't change

DeepSeek did not build a new model. That is the whole event.

On August 1 a Hugging Face user posting as GaelicThunder needed to know whether a quantization recipe built for the old checkpoint would transfer, and posted the diff:

That reuse is safe, and I checked rather than assumed: diffing the two model.safetensors.index.json, the body is identical — 67,612 non-MTP tensors, name for name, zero differences. Only the MTP/DSpark module changed (1 -> 3 layers), and it is not in this file.

The transformer body — the part that does the thinking — is untouched. What grew is the MTP/DSpark drafter, a speculative-decoding helper that guesses ahead to make generation faster: one layer became three. The V4 technical report says 284B, Hugging Face's metadata says 304B, and the difference is the bigger drafter, not a bigger model.

A safetensors index maps tensor names to shards and shapes, so identical indexes mean identical architecture, not identical values. What the diff verified is the structural half of DeepSeek's claim: no one re-pretrained anything, and the model doing the reasoning is the same one that shipped in April.

They re-post-trained a model they already had, and on DeepSeek's own numbers its DeepSWE score went from 7.3 to 54.4, a 7.45x jump and the 645% figure that flew around all weekend. It is the one headline number that is at least internally consistent. Nobody outside DeepSeek has reproduced it.

Independent

The number that reproduced

Toolathlon is maintained independently and flags the rows it ran itself with a verified badge. As of August 3, 2026, its leaderboard carries DeepSeek V4 Flash 0731 (max), dated July 31, badge attached: Pass@1 of 70.7 ±0.9.

DeepSeek claimed 70.3. The independent harness measured 70.7, and the error bar swallows the difference whole.

Cost on the same suite

$1.16

against $42.21 for Claude Sonnet 5, on eight puzzles and the same 22 turns

Independent Toolathlon pass@1

70.7

±0.9, against a vendor claim of 70.3 — a reproduction, not a win

Same model body

67,612

tensors matched name for name — the reasoning core did not change

Nobody screenshots a tie, which is exactly why the numbers that travel are the unchecked ones.

DeepSeek sits sixth at 70.7, Claude Sonnet 5 fifth at 71.6. That gap is 0.9, the same margin we just called a tie. On a single attempt, the cheap open-weights model and Sonnet 5 are indistinguishable. Above them, Kimi K3 at 76.5, Opus 4.8 at 76.2, Muse Spark 1.1 at 75.6, and GPT-5.5 at 73.5 are ahead.

Pass^3 measures solving the task all three times out of three. DeepSeek scores 58.3. Sonnet 5 scores 53.7. The board publishes no error bar on that column, so treat it as suggestive rather than settled. The suggestion is that the cheaper model is steadier.

The bug

Ask for high, get low

The finding that reframes launch week is a missing file.

DeepSeek shipped the open weights without a Jinja chat template, providing a Python reference encoder instead. Every server using the standard apply_chat_template() path, which is most of them, was left without a correct way to format prompts. On August 1 someone opened a pull request to add one, noting that vLLM's native path produced identical output for low and high reasoning effort.

A second user, cayennecolloid, verified it independently on 2× DGX Spark hardware. Through the reference encoder, the three effort levels produce prefixes of 50, 526, and 576 characters. Through a standard endpoint:

low   → 5 tokens, empty
high  → 5 tokens, empty (prefix dropped)
max   → 84 tokens, ends "...no assumption is left unchecked."
        ↑ that is the high text

Ask for max, get high. As they put it:

So high is unreachable and max silently delivers high. Anyone benchmarking effort levels through a standard OpenAI-compatible endpoint has been measuring high while believing they measured max.

An unknown share of launch-week testing was measuring a weaker configuration than it asked for. Scope it honestly: this affects the open weights served through generic endpoints, not necessarily DeepSeek's first-party API, which uses its own encoder. Which of last week's evaluations hit which path is not publicly documented, so no one can yet say how much of the spread it explains.

The picture is uneven. Simon Willison, posting just after midnight on August 1: “I got a disappointing pelican on default reasoning mode... But a REALLY good pelican on reasoning mode high (via OpenRouter).” His blog verdict the same day: this “may currently be the best value-per-intelligence model out there.”

The plumbing that selects the effort level differs between serving stacks, and almost no one reports which stack they used. That is the state of model evaluation in August 2026.

The corrections

Where the amazement outran the evidence

Start with a number everyone repeated wrong, including us. The comparison that travelled furthest was “Terminal Bench 2.1: 56.9 to 82.7,” a 25.8-point leap. Pull the cards and it falls apart: the 0731 card lists Terminal Bench 2.1 and gives the previous Flash a baseline of 61.8 in the adjacent column, while 56.9 comes from the Preview card, sitting in a table headed Terminal Bench 2.0. Two benchmark versions, subtracted from each other.

The real like-for-like jump is 20.9 points. Still huge. Just not the one that got screenshotted.

The circulating Toolathlon comparison is wrong in a stranger way. Its “51.8” baseline belongs to V4-Pro-Max on the original Toolathlon, a different model on a different benchmark, when the card's own like-for-like baseline is 49.7. That version understates the gain: not inflated, just assembled from the wrong cells.

DeepSeek's card published the correct baselines in the correct columns on day one. The vendor didn't cook this one. We did.

DeepSeek self-reports 82.7 on Terminal Bench 2.1. Artificial Analysis measured 79% on the Terminus 2 harness. Vals AI, running its own suite on August 1, reports 67.04% ±1.35. Its published run metadata leaves the reasoning effort unset, while DeepSeek's 82.7 was produced explicitly at max, so those two are not like-for-like. Three numbers for one benchmark, and the configuration behind each is either different or undocumented.

• ReproducedToolathlon-Verified, 70.7 ±0.9 on a harness DeepSeek does not control, against a claimed 70.3
• ContestedTerminal Bench 2.1: 82.7 self-reported, 79 from Artificial Analysis, 67.04 from Vals AI
• UnverifiableDSBench-FullStack and DSBench-Hard are DeepSeek-internal test sets, by the company's own footnote

As of August 3, 2026, the official Terminal-Bench 2.1 leaderboard does not list DeepSeek at all. Nobody can settle it, because the card credits those scores to “the minimal mode of DeepSeek Harness (to be released)” and, three days later, the harness is still a parenthesis. Two of the nine benchmarks are DeepSeek-internal test sets, unverifiable by construction. The much-quoted DeepSWE 54.4 has no independent listing either, though in fairness Datacurve's board reads “updated July 25, 2026,” six days before the model shipped, which makes that one unverified rather than refuted.

DeepSeek's own table shows one more thing: Opus 4.8 still leads V4-Flash-0731 on every benchmark listed. The company published a chart of itself losing and the internet read it as a victory lap.

Local

It runs on hardware you can buy

The weights are MIT-licensed, and within hours people had them running on desks. One user reported two DGX Sparks hitting 60 tokens/second in a single session. GaelicThunder built a 2-bit quantization that lands at 86.7GB, small enough for a single 128GB box, and measured 28–33 tok/s decode.

Then they came back and marked their own homework down. A hunch about draft acceptance was retracted outright. “I have no evidence for it.” And the widely quoted 59 tok/s figure from an earlier thread had been measured differently: “An aggregate over 12 concurrent requests. Single-stream chat decode is ~28 tok/s.”

At the other end of the scale, one team runs it self-hosted on B300s for voice-transcription QA at “2-5% of the cost of running on Equiv Frontier” — then the sentence every self-hosting pitch leaves out: “It took us about a month to get the inference configured to achieve these numbers.”

The catch

What the celebration skipped

The verbosity is real, at least on some harnesses. Artificial Analysis measured 210M output tokens to complete its Intelligence Index against a median of 100M, calling it “very verbose,” at $72.02 for the run. On Hacker News, WhitneyLand measured “about 3.6x more tokens to finish the same work as Gemini Flash 3.6.” Yet on Quesma's harness the same model took 22 turns where GPT-5.6 Luna took 118. Every verbosity number you have seen is a number about somebody's stack.

Every invoice in this article was priced before that policy takes effect. The second window covers a European team's entire morning.

The stakes

Remember the date

The part of this model that does the thinking never grew. It is the same size and shape it shipped as in April, and it matched Claude Sonnet 5 for $1.16 against $42.21.

Keep the table honest, though: Claude Opus 5 solved that suite outright, 100% in seven turns for $17.15. The frontier still wins the hard version. What changed is what the second-best answer now costs.

As one commenter put it just after midnight on August 1: “DeepSeek keeps showing how much of a performance improvement can come from post-training alone while the architecture stayed the same. It's a strong reminder that we're probably underestimating how much optimization is still left after pretraining.”

If the last stretch of frontier capability comes from post-training rather than scale, it comes from the cheap end of the pipeline — and the moat that several hundred billion dollars of capex was supposed to buy is narrower than the invoices implied.

Verify the agentic numbers before you bet on them. The harness is still unreleased, and the spread is still wide enough to drive a roadmap into. But put a reminder on the replication, because if it holds, July 31, 2026 is the day the price of frontier-adjacent intelligence fell through the floor and the industry spent the weekend arguing about the wrong number.