A grammar for tiny models: every answer valid, many still wrong

Follow-up, 2026-11-03: the language with a third variable, "not" in rules, and an unexpressible token.

In Master's design, a Ministry of Reason would define a language that every fact has to be written in. It doesn't exist yet, but I wrote a tool that turns such a language (an example schema I made for testing) into a grammar llama.cpp enforces while a model writes, and gave six small models 150 sentences to translate. With the grammar, 900 of 900 answers were valid and every model got more right (Qwen3-4B 96 → 115 of 150, Qwen3.5-0.8B 48 → 72). But the grammar only removes mistakes of form: the small models then wrote perfectly valid statements of the wrong kind, and on statements about relations even the 4B stayed at 40%. Two side findings change how I run tests here: slots barely help on this CPU, and the LFM models never reused llama-server's prompt cache.

The test

The language (the example schema from the Ministry post): 14 classes (person, food, city, …), 18 relations with argument types (eats(agent, food), capital_of(city, country), …) and 26 entities. Six kinds of statement:

The last three are what logicians call second-order in spirit: they say something about relations, not about things. (General quantification over relations, "for every relation R…", is not in the language; that stays as fixed logic in code.)

The grammar generator reads the schema and writes a GBNF grammar (227 lines, 34 KB) that admits exactly the well-typed statements: eats only with an agent and a food, symmetric only for relations whose two arguments have the same type, inverse only for pairs with swapped types, rules only with atoms that fit the declared variable types (93 rule variants, one per pair of classes that has any fitting atom). A checker in code parses any statement, type-checks it and reduces it to one canonical form (so disjoint(dislikes, likes) equals disjoint(likes, dislikes), and a one-line rule over the full types equals a subrelation). To test the grammar I compiled a small program against llama.cpp's own grammar engine and compared it with the checker: 59 checks, all pass, including 10,000 random statements (2,649 of them well typed) with no disagreement, 10,000 random well-typed statements of every kind all accepted, and no dead end in any prefix of a valid statement.

The sentences: 150, generated with their correct statements, 25 per kind, in 2–3 wordings each ("Rex the dog is fond of the car.", "It never happens that A loves B and A likes B at the same time."). The content is arbitrary on purpose ("every city is a kind of vegetable" is a valid item): the test is faithful translation, not world knowledge. The prompt (about 800 tokens) lists the classes, relations, entities, statement forms and six examples, none reused as items.

Setup: a 4-vCPU cloud server, no GPU, llama.cpp b11946, llama-server with 4 threads and 4 slots (requests in parallel), temperature 0, thinking off, one model at a time. Each model ran every sentence twice: with the grammar and free (graded by the same checker, which forgives fences, quotes and a final full stop). Right = the canonical form equals the correct one. CPU steal (time the cloud host gives to other machines) was recorded with every set.

Results

Out of 150 each. Every grammar answer was also in exact grammar form. Right with the grammar, by kind (out of 25):

1. The grammar fixes form, not meaning

Where the grammar helps most, the free answer was nearly right. Qwen3-4B's rules go from 10 to 25 of 25, and 12 of its free failures were the right statement with brackets around the conditions (. (likes(X, Y) & eats(X, Y)) -> …), which the language doesn't allow. LFM2.5-230M, free, copied the English sentence back in 116 of 150 answers; with the grammar it had to write a statement.

What's left is wrong in meaning. I sorted every wrong grammar answer by what went wrong: the right kind of statement with the wrong content (relation or arguments), a statement of another kind, or a copy of one of the prompt's examples:

The biggest error for every model below 4B is the wrong kind of statement: a rule where the sentence described a property, a fact where it described a class. A grammar can't help there, because both are valid. The obvious next step is to decide the kind first (by code, or as a six-way choice) and then use a grammar for that kind only. I haven't tested that yet.

2. Statements about relations: priors, gaps and copying

On properties and relation pairs the 4B stayed at 10 of 25 each, with or without the grammar. Reading its wrong answers (not counted, just read), three things happen:

• What it knows beats what it reads. "Whenever A is the child of B, it is also true that B is the child of A" became inverse(child_of, parent_of), three times for three relations. The sentence is false in the real world, and the model translated what it believes instead of what it was told.
• The language has gaps. For "if A loves B and A loves C, then B and C are the same", the 4B, free, wrote forall X, Y, Z. loves(X, Y) & loves(X, Z) -> same(Y, Z): the definition of functional, but the language has no third variable and no equality. For "if A dislikes B, then it is not the case that A knows B" (disjoint) no rule can say "not" in its conclusion, so under the grammar it wrote rules like dislikes(X, Y) -> likes(X, Y).
• Copying the example. The prompt's one relation-pair example was an inverse, and inverse is what it wrote for sentences that were symmetric, transitive or disjoint.

The first one would matter for a Ministry of Truth: a translator that corrects the text toward its own beliefs would quietly turn reported claims into the model's opinions.

3. A grammar must allow what the model wants to write first

My first grammar started directly with the statement. In a 12-item trial, LFM2.5-230M wrote "S" first (it wanted to write "Statement:", because the prompt's examples carry that label), and the only statement that starts with an "s" is a sibling_of(...) fact: it wrote one for 11 of 12 sentences and got none right. With an optional Statement: allowed in front, it got 2 of 12. A grammar that forbids the model's first token doesn't just block it, it steers it into the nearest legal word.

4. What the grammar costs: about nothing per item

llama.cpp checks each sampled token against the grammar, and only when the token is invalid does it mask the whole vocabulary and sample again. I expected the cost to grow with vocabulary size (Qwen3.5 has 248,320 tokens, LFM2.5 65,536). It didn't: with one request at a time the grammar added 11.5 ms per output token for both LFM2.5-230M (15.3 → 26.8 ms) and Qwen3.5-0.8B (56.6 → 68.2 ms).

Per item that cost disappears, because answers under the grammar are shorter (10.3 instead of 13.3 tokens for the 0.8B). Three rounds with the order alternated, all slots warmed up first:

The spread is mostly steal: across the six sets, items/min fell by about 1.4 per percent of steal (correlation −0.91). My first two comparisons said the grammar cost 50% more time per item and 25% fewer items per minute; both were noise. One compared sets with 3% and 6% steal, and in both the grammar ran first on a fresh server and paid the prompt for every empty slot. Lesson for the bench: warm up every slot, alternate the order, compare under the same steal.

On a very fast model the fixed cost per token weighs more: LFM2.5-230M made 26% fewer items per minute with the grammar (81.2 against 109.3, one slot).

5. Parallel slots barely help on this CPU

llama-server can run several requests at once in "slots", reading the weights once for all of them. In a raw test (llama-batched-bench, 256 prompt + 64 output tokens per sequence) decoding gained this much over one sequence:

(a) A short compile ran during these two; one run each, so the dips at 8 are unconfirmed. I had predicted 2.5–3.5x at 4 for the 0.8B.

On the real task (about 30 new prompt tokens and 12 output tokens per sentence) the gain was smaller still: at best 1.2x (Qwen3.5-0.8B and LFM2.5-230M without the grammar), with the grammar 1.0–1.2x, Qwen3-4B 1.08x at 8 slots. The 4B shows why. With one slot it processed 8.5 tokens per second (prompt and output together); with eight, 10.4, which is exactly the raw test's ceiling at 8. Four CPU cores are simply busy: there's no idle capacity for a second request to use. Per sentence, reading the ~25 new prompt tokens took about as long as writing the 12 output tokens (2.0 and 2.2 seconds for the 4B with one slot), so on this machine the lever is fewer tokens per request, not more requests at once. A machine with more cores or a GPU may look very different; I haven't measured one.

6. LFM models never reused the prompt cache

The LFM models were oddly slow: LFM2.5-1.2B took 7 seconds per sentence where Qwen3-0.6B took 1.1. The server timings showed why: every LFM request re-read about 515 of its ~800 prompt tokens, while the Qwen models re-read 24–50.

LFM2.5 is a hybrid model with a recurrent state that can't be rewound to an arbitrary position; llama-server instead saves snapshots ("checkpoints") and rewinds to the nearest one. It normally saves one where the last user message starts, so the shared system prompt is kept. But llama.cpp handles LFM2.5's chat template with special code that doesn't mark where messages start, so the only snapshots sit 516 and 4 tokens before the end of the prompt (4 + the batch size, 512). With an 800-token prompt the usable one is at token 277. I confirmed this in the server's verbose log and in its source.

The workaround is a smaller batch, -ub 64, which moves the snapshot to 68 tokens before the end. On the full set:

Right answers stayed within 2 (1.2B: 41 → 40 and 11 → 13). Prompts shorter than 516 tokens get no usable snapshot at all, which is why the LFM models reused nothing in my earlier tests either, as I had noticed but not explained in the speed post. I've added correction notes to the posts whose times it affected; the largest effect is in part 1 of the tiny-models series.

7. Temperature 0 isn't the same answer with more slots

With temperature 0 I expected identical answers whatever the number of slots. The number right barely moved (Qwen3.5-0.8B with the grammar: 73, 73, 73, 72 at 1, 2, 4, 8 slots), but single answers did. Compared with one slot, Qwen3.5-0.8B changed 3–7 of 150 answers with the grammar and 10–13 without it; LFM2.5-230M 16–19 with and 24–26 without. Batching changes the order of the arithmetic slightly, and a tiny model near a tie flips. The grammar damps this, since it removes most of the choices.

My predictions

Written before the runs:

What I take from it

• A grammar is worth using for every structured output of a small model. For the 0.8B it cost about nothing per item (a very fast model like the 230M pays up to a quarter of its speed), and it turned 0 right answers into 12 for the smallest model and 96 into 115 for the 4B.
• It doesn't make a tiny model a translator. Below 4B, most answers were valid and wrong. The kind of statement has to be decided another way, and a Ministry of Truth would have to assume that a valid statement from a tiny model may still be wrong: that is what its scores are meant for.
• The language needs two additions that the models asked for by themselves: a third variable with equality, and rules that conclude "not".
• On a small CPU, measure your prompt cache before your slots. Fixing one llama-server flag made an LFM model up to 5.9x faster; four slots made the same machine at most 1.2x faster.

The data

• Every answer (5,400 rows: run, model, slots, condition, sentence, correct statement, model output, verdict, right, server timings, steal). The "main" run is the results table, "slots" the slot and -ub tests, "interleaved" the three rounds. In "slots", the 0.8B and 230M sets ran without warm-up and the grammar ran first, so their grammar timings include the cold start.
• Summary per set, the chart's data, the raw slot test.
• How far to trust it: one run per condition except the three interleaved rounds; 150 generated sentences from one schema, so a different schema or real text may behave differently; timings on a shared cloud CPU move with steal by ±10% or more.