How fast do small models run on a 4-core CPU? 65 model files measured
I ran 65 GGUF model files from five model makers on a small CPU-only cloud server and measured how fast each one reads a prompt (prefill) and writes an answer (decode). The short version:
- Decode speed is memory bandwidth divided by file size. For the ordinary ("dense") models, decode tokens per second × file size came out at a median of 18 GB/s (53 of 58 files between 14 and 24). On this machine that is about 18 GB/s ÷ file size: a 0.8 GB file writes ~20 tokens/s, a 2.5 GB file ~7.5.
- Two kinds of model beat that rule: a mixture-of-experts model (LFM2.5-8B-A1B, a 5.2 GB file) writes 17.6 tokens/s, faster than a dense 2.5 GB 4B model. Google's Gemma 4 "E" models are about twice as fast as their file size predicts.
- Reading the prompt is the slow part for agents. A 3–4B model reads 15–30 tokens per second, so a 1,000-token prompt costs it 30–60 seconds before it writes anything.
- Some quantization types read prompts at half speed: Q5_0, Q5_1 and Q4_1 were 2–3× slower at prefill than the K-quants and Q4_0 of the same model.
The machine and the method
- Hardware: a cloud VM with 4 vCPUs (Intel Broadwell: AVX2, no AVX-512), 8 GB of RAM and no GPU. A single core reads memory at about 10 GB/s.
- Software: llama.cpp v0.7.0 built from source with every CPU variant (it picked the AVX2 "haswell" build),
llama-server, 4 threads, one request at a time, temperature 0. - Workload: the same 11 short tasks per file: prompts of 30–80 tokens on average, answers of about 10–20 tokens (two LiquidAI models always think first and wrote 70–110 tokens per task). Speeds are the server's own timings, summed over the 11 tasks.
- Cross-check: for two files I also ran
llama-bench(512-token prompt, 128 generated tokens). Both methods agree within the noise:
| File | Method | Prefill tok/s | Decode tok/s |
|---|---|---|---|
| Qwen3-4B Q4_K_M | llama-bench | 20.0 | 5.8 |
| Qwen3-4B Q4_K_M | server, short tasks | 19.9–21.4 | 7.6 |
| Qwen3.5-0.8B Q8_0 | llama-bench | 76–82 | 16.5–18.9 |
| Qwen3.5-0.8B Q8_0 | server, short tasks | 74.6 | 20.0 |
- Noise: the VM shares its CPUs with other customers. I recorded the stolen CPU time with every sample; in an earlier speed test it correlated with speed at −0.77. Treat single numbers as ±15%.
Decode: bandwidth ÷ file size
To write one token, a dense model has to read all of its weights from memory once. On a CPU, that read is the bottleneck, so the speed is set by how many bytes the file has, not by how the bytes are packed. Across the 58 dense files, decode × size had a median of 18 GB/s, and 53 of them fell between 14 and 24 GB/s. The rest were three Granite files (8.6–13.7) and two at 24–25; with answers of only ~10 tokens and noisy neighbours, single files scatter.
That has a simple consequence for choosing a file: **a quantization only makes decode faster by making the file smaller.** BF16 and F16 files are twice the size of Q8_0 and write at half the speed (Qwen3.5-0.8B: 11.9 vs 20.0 tokens/s).
Mixture-of-experts breaks the rule. LFM2.5-8B-A1B has about 8B parameters but uses only about 1B for each token, so it reads only a slice of its 5.2 GB file per token. It wrote 17.6 tokens/s; the dense rule predicts about 3.5. The catch for this particular model: it always thinks before answering, even with thinking switched off. It used 66 tokens to say "OK", so per answer it is slower than its speed suggests.
Gemma 4 E2B and E4B are also faster than their size (31–45 GB/s "effective"). The "E" stands for effective parameters: a large part of the file is per-layer embedding tables, of which only a few rows are needed per token. That is my reading of the architecture, not something I measured directly.
Prefill: compute, not bandwidth
Reading the prompt works on many tokens at once, so it is limited by arithmetic, not memory. Here the model's size in parameters matters much more than the file:
| Model (file) | File GB | Prefill tok/s | Decode tok/s |
|---|---|---|---|
| LFM2.5-230M Q6_K | 0.19 | 351 | 99.8 |
| LFM2.5-350M Q4_K_M | 0.23 | 274 | 79.1 |
| Qwen3.5-0.8B Q8_0 | 0.83 | 75 | 20.0 |
| LFM2.5-1.2B-Instruct Q4_K_M | 0.73 | 72 | 23.1 |
| LFM2.5-2.6B Q4_K_M | 1.67 | 31 | 10.6 |
| Gemma 4 E2B (Google QAT q4_0) | 3.35 | 32 | 12.5 |
| Ministral 3 3B Q4_K_M | 2.15 | 23 | 8.5 |
| Granite 4.2 3B Q4_K_S | 2.14 | 26 | 9.1 |
| Qwen3-4B Q4_K_M | 2.50 | 20 | 7.6 |
| Gemma 4 E4B (Google QAT q4_0) | 5.16 | 16 | 7.0 |
| LFM2.5-8B-A1B Q4_K_M (MoE) | 5.16 | 38 | 17.6 |
For an agent on a CPU, prefill is usually the bigger cost: a tool list, a few instructions and the conversation so far easily reach 1,000 tokens. On a 4B model that is close to a minute per turn before the first word. Two things help: short prompts, and a prompt whose beginning never changes, because llama-server can reuse the part it has already read. (One more open question: the LiquidAI models reused none of it in my runs; the Qwen models did.)
The slow quantization types
For two families I had many quantization types of the same model. The "legacy" types Q5_0, Q5_1 and Q4_1 read prompts far slower than the rest:
| File | Prefill tok/s |
|---|---|
| Granite 4.2 3B Q4_K_S | 25.6 |
| Granite 4.2 3B Q4_0 | 22.3 |
| Granite 4.2 3B Q5_0 | 12.0 |
| Granite 4.2 3B Q4_1 | 10.3 |
| Granite 4.2 3B Q5_1 | 7.5 |
| Qwen3-4B Q4_K_M | 19.9 |
| Qwen3-4B Q5_0 | 9.6 |
Qwen3-4B Q5_0 was slow in all 8 samples of two separate benchmarks, so this is not noise. Q4_0 is not affected. My guess is that llama.cpp has a re-packed fast path for Q4_0 and the K-quants on AVX2 but not for these types; I have not checked the source. Either way: on a CPU like this one, pick Q4_K_M (or Q4_0) and avoid Q5_0, Q5_1 and Q4_1.
More things worth knowing
- Threads: going from 1 to 2 to 4 threads sped up the 4B's prefill 1.95× and then only 1.37×.
- Long answers get slower: while thinking for thousands of tokens, Qwen3-4B fell from 3.9 to 2.8 tokens/s, because attention reads a growing context for every new token.
- A real workload: on 172 small agent tasks (prompts of 350 tokens on average, up to 1,270), Qwen3.5-0.8B averaged 86 tokens/s prefill and 19.7 tokens/s decode, 15.9 minutes in all. The 230M and 350M models did the same tasks in 3.3 and 4.9 minutes.
The data
All 65 rows (file, size, prefill, decode, decode × size, tokens generated, stolen CPU time) are in 2026-10-28-cpu-speed.csv. One machine, one llama.cpp version: other CPUs, especially ones with more memory channels or AVX-512, will give different numbers, but the bandwidth rule should carry over.