New tiny models on a CPU, part 1: a 2-bit file that isn't, a fast tiny MoE, and a thinker that wins

Preliminary. This is a smoke test: 22 easy questions per file, one pass each. It finds broken files and setup problems; it can't rank models. The real worker tests (pulling data out of text, tool calls, fixing their own mistakes, resisting planted instructions) come in part 2.

Three new families of tiny models arrived this week, so I ran them through my easy suite on the same small CPU-only server as before, next to the tiny models I had already tested:

• Qwen2.5-0.5B-Instruct: 8 of Qwen's own GGUF files, from "q2_k" to fp16.
• Qwen3-0.6B: Qwen's Q8_0 file.
• SmallThinker-4B-A0.6B-Instruct by Tiiny AI and Shanghai Jiao Tong University's IPADS lab: a mixture-of-experts model built for local devices. 4B parameters in all, but only about 0.6B are used per token. 10 files.
• LFM3-1.7B by LiquidAI, released days ago with a new architecture: 7 files, as the "final boss".
• For reference: LiquidAI's LFM2.5-230M and 350M and Qwen3.5-0.8B from my earlier tests.

LFM3 needed a newer llama.cpp than the one I had used so far, so this whole run is on build b11946. The three reference models gave exactly the same answers on the new build as on the old one, so the numbers below are comparable with my earlier posts.

The test

The 11 easy questions check the basics an agent needs: answer in a given format, follow a system prompt, stop after a list, echo accents and Japanese characters, use the conversation history, call a tool when asked, not call one for small talk, use a tool's result, and produce JSON. Each has a twin with different wording that I keep sealed: I only look at its total. Greedy decoding, thinking off (unless noted), 4 threads, one request at a time.

Memory is with a 32,768-token context; that matters a lot, as you'll see. All rows are in the CSV linked at the end.

Finding 1: the "2-bit" file that isn't

Last time, a 2-bit file (IBM's Granite 4.2 3B at Q2_K) couldn't stop talking: it had lost its end-of-turn token. So I expected Qwen2.5-0.5B's "q2_k" to break too. Instead it passed all 11 easy questions and 9 of the 11 sealed ones, as well as its own 8-bit file.

The file explains it: it contains no 2-bit tensors at all.

The K-quant types (Q2_K, Q3_K, Q4_K, …) work on blocks of 256 numbers. This model is 896 wide, which is 3.5 blocks, so for most tensors the quantizer quietly falls back to another type. The "q2_k" file ends up mostly 4-bit and is nearly the same size as "q4_0". The "q4_k_m" file is mostly Q5_0, a type that reads prompts slowly on this CPU, and it did: 112 tokens/s against 174 for "q4_0".

Lesson: for a small model, read the tensor types (any GGUF viewer shows them) before trusting the file name. This says nothing about real 2-bit weights. Granite's Q2_K showed what those can do.

Finding 2: the tiny MoE is fast, chatty with tools, and memory-hungry

SmallThinker's makers say that at Q4_0 it runs at "over 20 tokens/s on ordinary consumer CPUs". On my 4-vCPU cloud server, Q4_0 wrote 24.6 tokens/s. Claim confirmed, for short answers at least.

The chart shows why. For a dense model, decode speed is memory bandwidth divided by file size. SmallThinker's files range from 1.9 to 4.6 GB, yet nine of the ten decode at 20–27 tokens/s (the tenth, Q4_K, at 15). Only the 0.6B active parameters are read per token, so the file size hardly matters. Its Q8_0 file writes faster than the dense rule predicts for a file a quarter of its size.

Two things to watch for part 2:

• It calls tools it doesn't need. Given the weather a tool had just returned and asked about the weather, it called the weather tool again instead of answering, on 8 of its 10 files. Told "Hi! Just say hello back, no need to look anything up", it called a tool anyway on 3 of 10. Of the 16 other new files, only one did either.
• It needs far more memory than its "1 GB" claim. That claim is for the makers' own inference engine, which keeps experts on disk. With llama.cpp the whole file is mapped, and for some types a second copy is made (finding 4). Peak memory was 3.0–5.5 GB.

Its scores were middling, 8–11 of 11. Part 2 will show whether the speed comes with usable answers on real tasks.

Finding 3: LFM3 is the final boss, but only when it may think

With thinking off, LFM3 scored 9–10 of 11, no better than the tiny models. Its mistakes all looked like this:

• asked for the number only: 7 + 5 = 12 then 12;
• asked for a list: 1, 2, 3, 4, 5</think>1, 2, 3, 4, 5.

I first suspected my setup, so I looked at the prompt the server builds. It was correct: with thinking off, the model's turn starts with an empty <think></think>. LFM3 does its working in the visible answer anyway. It is built to think.

With thinking on, nothing else changed, **all 7 files answered all 11 easy questions, and 6 of 7 also got all 11 sealed twins.** No other model came close. The price: about 120 tokens of thinking per answer, so the Q4_K_M file took 3.6 minutes for the 22 questions instead of 0.9. In part 2 we'll see whether that ratio holds on harder tasks, where thinking might pay off more.

Finding 4: memory you don't see in the file size

Two effects made some models need several times their file size:

• The context window. llama-server reserves the memory for the whole context up front. Qwen3-0.6B stores 115 KB per token of context (28 layers × 8 key/value heads × 128 values × keys and values × 2 bytes). At 32,768 tokens that is 3.8 GB. Its 0.6 GB file needed 4.25 GB of RAM. Qwen2.5-0.5B, with only 2 key/value heads, needs 12 KB per token (0.4 GB). For a model like Qwen3-0.6B, set the context to what the task needs.
• Re-packing. On a CPU like this one (AVX2), llama.cpp re-packs Q4_0, Q4_K and IQ4_NL weights into a faster layout, and the re-packed copy sits next to the memory-mapped file. I checked this in the source this time; I had only guessed at it in my speed post. SmallThinker shows it plainly: its Q3, Q5, Q6 and Q8 files needed about file + context, its Q4 files about 2 GB more.

Coming in part 2

The easy suite only finds broken files. Part 2 runs the tests that decide what a tiny model can be used for:

• pulling fields out of messy text, tool calls with up to 15 tools offered;
• fixing its own work after a failed check, recovering from tool errors;
• and not obeying instructions planted in an e-mail. Last time, a 0.8B model called delete_files every time it was told to.

I'll run them on the best file of each new model, with LFM3 thinking, against the same tasks my earlier tiny models did. I'll also check whether SmallThinker's speed holds once prompts are a few hundred tokens long. My guess, to be checked: LFM3 wins on quality but costs several times the time, and SmallThinker is the most interesting for speed.

The data

Every file's row (size, passes, prefill and decode speed, peak memory, time, tokens generated, thinking tokens, CPU steal, llama.cpp build) is in 2026-10-28-tiny-models-part1.csv. Speeds come from 22 short tasks with answers of about 10 tokens, so treat them as ±15%.