Memory bandwidth

Memory bandwidth is how fast a GPU can read data out of its memory, measured in gigabytes per second (GB/s). It’s different from memory size (how much the memory holds, in GB).

Beast’s 5090 has about half an H100’s bandwidth and under a quarter of a B200’s.

Why it decides how fast a model writes

When an LLM writes an answer, it produces one token (roughly three quarters of a word) at a time. To produce each token, the GPU has to read all of the model’s weights from memory. Then, for the next token, it reads them all again.

So the time to write one token is mostly the time it takes to read the weights once:

$\text{max tokens per second} \approx \frac{\text{memory bandwidth}}{\text{size of the model's weights}}$

That’s why beast writes faster than a Spark even though the Spark has four times the memory: beast reads its memory about 6.5 times faster. The Spark’s extra size only lets it hold bigger models.

Where it comes from on a spec sheet

Bandwidth is calculated from two other lines on the spec sheet:

• Memory interface (bus) width: how many bits travel between the GPU and its memory at once. The 5090 has 512 bits.
• Memory speed (data rate): how many bits each of those lines carries per second. The 5090’s GDDR7 runs at 28 Gbps.

$512 \text{ bits} \times 28 \text{ Gbps} \div 8 = 1{,}792 \text{ GB/s}$

(Divide by 8 to turn bits into bytes.)

How this plays out in the cloud

The same rule applies on cloud GPUs; it’s just hidden from you as a user. One person alone on a B200 would get very fast writing (8,000 ÷ 17 ≈ 470 tokens/s ceiling for the example model above).

But providers don’t give you a whole GPU. They batch: one read of the weights can produce the next token for every request in the batch at once. That’s why serving lots of users is cheap.

• More users per GPU means a lower cost per token, but slower speed per user, because each step does more work.
• Each provider picks a point on that trade-off, and most lean towards cost.
• That’s why a home box like beast or a Spark can feel as fast as a cloud service despite slower hardware: at home, the whole GPU is yours.

When it becomes your problem

If you deploy a model yourself (Hugging Face Inference Endpoints, AWS, Kubernetes), you choose the GPU, and three questions decide the choice:

1. Does it fit? Memory size has to hold the weights plus room for the conversations (kv-cache, a later lesson).
2. How fast is each answer? Memory bandwidth sets the ceiling for one user.
3. How many users per GPU? Batching sets how far you can stretch it, and that drives cost.

This sizing work is a core part of AI platform and inference engineering.

What it does not control

Reading your prompt (called prefill) works differently: the whole prompt is processed in one go, so it’s limited by the GPU’s raw compute power, not mainly by bandwidth. That’s a later lesson.

Check yourself

1. What’s the difference between memory size and memory bandwidth?
2. Why does the model have to read its weights again for every token?
3. If you squeeze a model from 17 GB down to 13 GB, what should happen to its top writing speed on beast, and roughly by how much?
4. Why can a Spark run models that beast can’t, even though beast is faster?
5. A cloud B200 is far faster than beast, so why might your answer from a cloud provider arrive slower than from beast?

Try it on beast

Pick a model you already serve and note its size on disk and its measured generation speed (tokens/s) from your lab. Work out the ceiling with the formula above and see how close your real number gets. Write what you find in drafts/.