On 5 February a paper introduced DFlash — a way to speed up generation on any language model without touching the model itself — and NVIDIA followed with benchmarks headlined "up to 15× higher throughput". That is the kind of number worth dropping everything to verify. We took the DFlash drafter for Gemma 4 26B — it ships in the same repository as the model itself — ran it on our own bench and got 2.31× — but only while a single request sits on the card. Past four concurrent requests DFlash starts losing to the plain model on total throughput; by the eighth it loses by half.
Here is how it works, where the 15× comes from, and why our chart diverges from the vendor's without anyone having lied.
How DFlash speeds up generation
A normal model decodes autoregressively: one pass over all the weights yields one token. On a batch of one that is monstrously inefficient. To compute a single token the GPU has to pull every weight through memory, and what it hits is memory bandwidth, not compute. The arithmetic units sit mostly idle.
Speculative decoding sells that idle time. A small "draft" model quickly proposes several next tokens, and the large model verifies the whole block in one pass — the very pass it was going to make anyway for one token. A matching prefix is accepted whole; the first mismatch truncates the rest. The scheme is mathematically honest: the output distribution does not change, so the speedup costs nothing in quality.
The bottleneck is the draft itself. Classic methods like EAGLE-3 generate it autoregressively too, i.e. sequentially: five draft tokens means five small passes. The idea behind DFlash is to replace the draft model with a block diffusion one: it emits the entire block in a single parallel pass rather than token by token. It is also fed internal features from the large model so the draft lands closer to the context. The authors report a sixfold lossless speedup and up to 2.5× the gain of EAGLE-3.
In our case the draft is a separate 451 MB diffusion module bolted onto a 26B-parameter MoE model with 4B active. On our bench it hit the right continuation about a quarter of the time, with a mean accepted block of 2.06 tokens: one pass of the large model yields two tokens instead of one.
Where the 15× comes from
The number is real, but it has to be read together with its conditions. NVIDIA got 15× on gpt-oss-120b, eight DGX B300 boxes and TensorRT-LLM, and not "in general" but at one specific point on the Pareto curve — where each user is served 500–600 tokens per second. That is a regime of extreme interactivity in which ordinary autoregressive decoding is barely viable: to push that many tokens to one user you have to keep the batch nearly empty and burn the card for nothing. DFlash does tear the baseline apart at that point, because the comparison is against a deliberately inefficient way of running.
Move to a normal operating regime and the multiplier settles. The same article gives a table at matched concurrency: 2.3× on average for gpt-oss-120b and 2.8× for Llama 3.1 8B. At concurrency one on a single card — 5.8× on Math500 and 4.4× on MBPP for Gemma-4 31B — a sibling of the model we took. So the vendor's own data spans 1.8× to 15× depending on where you measure, and 15× is the far edge.
What we got
We measured total generation throughput on our own bench while raising the number of concurrent requests. The baseline is Gemma 4 26B without speculation; the second column is the same model with DFlash.
The bench: one NVIDIA RTX 6000 PRO, driver 610.57.04, CUDA UMD 13.3, inference in Docker 29.7.2 — llama.cpp server-cuda, build 10711. Model gemma-4-26B-A4B-it-Q4_0, KV cache q4_0, 512 tokens per request, draft length --spec-draft-n-max 4.
| Concurrent requests | Baseline, tok/s | DFlash, tok/s | Ratio |
|---|---|---|---|
| 1 | 180.8 | 418.0 | 2.31× |
| 2 | 325.8 | 519.3 | 1.59× |
| 3 | 440.8 | 480.3 | 1.09× |
| 4 | 517.3 | 522.3 | 1.01× |
| 6 | 674.7 | 581.5 | 0.86× |
| 8 | 779.4 | 538.2 | 0.69× |
The interesting part is not the ratio but the shape of the two curves. The baseline grows almost linearly: 181 → 326 → 441 → 517 — the card fills up gradually and each added request brings nearly as many tokens as the first. DFlash hits a ceiling right after the first request and then wanders in a 480–580 band no matter where we push concurrency. The crossover sits around 4.1 requests. And even the left edge of our chart is less than half the vendor's 5.8× on a sibling model: NVIDIA measured Math500 on its own bench, we measured a mixed load on one card.
One detail matters here: what loses is the server as a whole, not the individual user. Per-stream speed under DFlash never dropped below the baseline at any point of the sweep — at eight concurrent requests it came out level, 98.8 against 97.5 tok/s per stream. What sags is total throughput: the card spends cycles on drafts that are thrown away moments later.
Why concurrency eats the speedup
Both techniques live off the same resource — idle GPU arithmetic units — so they do not add up, they compete.
Speculation spends that idle time on draft tokens, some of which are guaranteed to be thrown away: the block is almost never truncated at its last position, and everything past the cut was computed for nothing. With a single request this costs nothing — the arithmetic units were idle anyway, so the discarded work is free.
Batching spends the same idle time on other people's real tokens. Eight parallel requests pull the weights through memory once for all of them and compute eight useful tokens per pass, none discarded. As soon as the batch fills the card, the bottleneck moves from memory to compute — and that is when discarded draft tokens stop being free and start stealing cycles from real work. Hence the ceiling on our chart and the drop below the baseline.
This is not a defect in DFlash, it is its envelope — and the industry knows about it. Production stacks can disable speculation on their own as concurrency rises; the threshold is often set around 32 concurrent sequences.
An honest caveat: our crossover arrives far earlier than the thirty-plus requests typical for the industry. That does not refute the vendor's numbers — it is a different point on the same curve: different hardware — one card rather than eight Blackwell Ultra B300s, and a different stack — llama.cpp rather than TensorRT-LLM. On eight B300s with TensorRT-LLM the ceiling will sit noticeably further right. The point of the measurement is not that NVIDIA inflated anything, but that a multiplier cannot be carried from the vendor's bench to your own cluster.
Why the gain shows up only on maths and code
The second thing the measurements show is that the speedup depends heavily on what you ask. We got a clear gain on maths problems and on code generation. On free-form prose it does not merely disappear, it goes negative.
Here is a separate run on the same bench, broken down by task type: one request on the card, 1024 tokens, drafts of up to 15 tokens — markedly greedier than in the table above. The third row is the mtp mode, a second speculation scheme we ran alongside.
| Mode | Maths | Code | Prose | Mean | vs baseline |
|---|---|---|---|---|---|
| Baseline | 188.0 | 189.6 | 189.4 | 189.0 | — |
| DFlash | 303.9 | 254.5 | 179.1 | 245.8 | 1.30× |
| MTP | 211.7 | 175.0 | 143.7 | 176.8 | 0.94× |
Maths 1.62×, code 1.34×, prose 0.95×: on free-form text DFlash stops helping and starts hurting, and the 1.30× mean rests entirely on the first two columns. For mtp, maths managed only 1.13× while code and prose went negative — on average it loses to the baseline, and we dropped it from there.
This table also explains why the figure here is 1.30× while the sweep above gives 2.31×: it comes down to draft length. We took fifteen tokens from the documentation examples, and it turned out to be the worst option we tried — the same 1024 tokens at n-max=4 give 330.6 tok/s against 245.8 at n-max=15. The longer the draft, the more computation burns on rejection, so the advertised block size is an upper bound, not a recommendation.
The mechanism is the same — the share of accepted drafts. The draft model guesses continuations better the less entropy they carry. In a derivation or in code the next tokens are pinned down by syntax and boilerplate: a closing bracket, a variable name, an equals sign, a standard function signature. In live prose there are dozens of near-equally likely continuations at every step, the draft misses at position two or three, the block gets truncated — and we pay for the whole computed block to gain one or two accepted tokens. Morphologically rich languages make it worse: the same word costs more tokens, which means more chances to miss inside a block.
That is exactly what the vendor's benchmark sets show: DFlash is tested on Math500, GSM8K, HumanEval and MBPP — the most predictable text there is. In NVIDIA's own table the multiplier falls to 1.8× on writing tasks against 2.6× on coding. There is an internal limit too: draft accuracy decays toward the end of the block — per the authors of the method's second version, from 99.5% at the first position to 87.8% at the seventh.
A side finding: a quantised KV cache costs 39% of the speed
While tuning we ran into something unrelated to DFlash but far cheaper to fix. We were keeping the KV cache quantised (--cache-type-k/v q4_0) for the sake of a long context. Switching it to f16 lifted the baseline from 189.0 to 263.1 tok/s — 39%, with no speculation at all; at eight concurrent requests, 917 against 784.
The gain is not free: an f16 cache takes roughly four times the memory, and on a smaller card it simply will not fit alongside the model and a long context — you end up choosing between context and speed. But the order of magnitude is telling: before bolting on speculation, check whether you already gave away those 39% on a quantised cache.
And it illustrates the article's main point. On the q4_0 cache the best DFlash variant gave 1.75× over the baseline; on f16 it gives 1.35×. The more carefully the base inference is squeezed, the less speculation has left to squeeze.
What this changes
DFlash is good engineering with a narrow envelope, not free acceleration for everyone. It pays off where one user's speed matters more than the hardware's total throughput: a local run on a single card, an IDE assistant, code completion, an agent's tool calls. In all of those the card really is underloaded, and speculation claims idle time that would otherwise go to waste. And the slower the card, the better speculation pays: the more a step of the target model costs, the more the draft earns back.
Where a queue of users lives on the card — that is, in any public service — plain batching wins, and speculation actively gets in the way. The practical, boring takeaway: measure your own configuration yourself, on your tasks, your languages and your real concurrency. A single multiplier from a press release does not transfer anywhere.
Where to see speed on our side
We applied that conclusion to ourselves. GPTunneL runs Grom 1.5 — our own model on our own hardware, and there we chased exactly what a user feels: the first token in roughly 500 ms and up to 120 tokens per second in the stream. That comes not from speculation but from controlling the whole stack: a balancer routes requests into pools by task weight, the dialogue prefix stays in GPU memory, and the model size is matched to the hardware it actually runs on.