Where a token actually spends its time

Sep 18, 2026

I have written the same paragraph in three different design docs now. Prefill is compute-bound, decode is memory-bound, plan capacity accordingly. Everyone nods. Nobody’s intuition changes, mine included, because the sentence is a conclusion with the arithmetic taken out.

So I built the arithmetic instead. The GPU token explainer follows a single token through sixteen stages, from an HTTPS request hitting a load balancer to text streaming back out, and recomputes every cluster figure as you change the configuration.

Decode re-reads everything

On every step, decode reads every weight and the whole KV cache.

A forward pass over one new token does very little math, but to do it the GPU has to pull the entire model out of HBM. Tiny work, enormous traffic. The arithmetic intensity lands far below the hardware’s ridge point, the memory system sets the pace, and the tensor cores idle.

Prefill has the opposite shape. It processes the whole prompt at once, so that same weight read is amortised across hundreds of tokens. What grows instead is attention:

2 × layers × heads × headDim × seq²

Causal masking computes only half the score matrix, which cancels one factor of two. Even so, quadratic is quadratic. Long prompts get expensive out of proportion to their length while the weight matmuls stay linear.

The explainer models both as a roofline, taking time as max(compute, memory) rather than branching on one or the other. The two resources overlap, so whichever is slower sets the floor. The effective ridge point, peak FLOP/s × MFU over bandwidth × MBU, keeps the invariant honest: below the ridge you are memory-bound, above it you are not.

Try the batch size slider

Set the batch to 1 and watch decode. You are paying for a whole GPU’s worth of bandwidth to produce one token. Raise the batch and throughput climbs almost for free, because that one weight read now serves thirty-two sequences instead of one. Keep going and the KV cache grows until it dominates the read, and the curve bends.

That bend is the capacity planning problem. It moves when you change precision, sharding or context length, and it is much easier to believe when you can drag it.

Collectives cost less than the textbook says

Tensor parallelism needs two all-reduces per layer, one after attention and one after the FFN. A ring all-reduce moves 2(N-1)/N × payload through each GPU’s links, which approaches twice the payload and never drops below it however many GPUs you add. That is a floor, not a scaling curve.

For the tiny payloads decode produces, the payload term barely matters and latency dominates. I model that as log2(N) rather than the textbook ring’s 2(N-1), because NCCL switches to tree and NVLS algorithms for small messages. The ring term overstates small-message collectives by roughly 3× at TP8, which is enough to send you looking for decode time in the wrong place.

What is modelled and what is not

The transformer stages are illustrative. They show the shape of the computation, nothing more. The cluster figures are real arithmetic over the configuration you set: parameter counts, memory footprint, KV cache size, roofline placement, collective cost.

MFU and MBU are knobs rather than measurements. They are where everything ugly about production, kernel efficiency and scheduling gaps and thermal behaviour, gets parked behind two numbers. Read the output as the shape of the problem, not as a forecast for your cluster.

It runs entirely in your browser, with no API and no account, and the link carries the whole configuration in its query string.

There is a companion piece, the LLM token explainer, which zooms the other way: into the math of the one token this app spends sixteen stages delivering.