When the language is too small: an unexpressible token for tiny models

The grammar test showed two gaps in the statement language that a Ministry of Reason would define. Master asked for those two to be closed and for one more thing: an unexpressible statement, for when a model can't say exactly what a sentence means in the current language. Master thinks this token is the key to making the Ministry of Reason balance itself. The two additions work: Qwen3-4B went from 20 to 28 of 50 on statements about relations. The token, the way I prompted it, did not: no model used it much more on sentences the language can't say than on ones it can, and what would be stored got less accurate. But the test also showed what goes wrong without it, 168 of 180 such sentences turned into facts about known people and things that the sentence didn't say, and what a model writes when nothing stops it: the gap itself, in a form code can find.

What changed in the language

Three additions, made in the generator that turns the language into a grammar for llama.cpp, and in the checker in code:

With these, three relation properties can now also be written as rules, and the checker knows they mean the same: born_in(X, Y) & born_in(X, Z) -> Y = Z is functional(born_in), located_in(X, Y) & located_in(Y, Z) -> located_in(X, Z) is transitive(located_in), and likes(X, Y) -> not dislikes(X, Y) (over the full argument types) is disjoint(dislikes, likes). Equality is only allowed between variables whose classes overlap (a person can't equal a food), and the old language is still there as a switch, with byte-identical grammar.

The price is size. Rules declare a class for every variable, and the grammar has to check every atom against those classes, so it grows with the number of classes to the power of the number of variables: 34 KB for two variables, 382 KB for three. I wrote it as a tree over the declared classes, so that once a model has written a class only that branch is still open (the number of open branches is what costs time per token). llama.cpp takes about 27 ms to read the grammar for each request, small against the one to five seconds a sentence takes here.

Testing the test. The self-test compares the checker with llama.cpp's own grammar engine: 103 checks, all pass. But random statements almost never hit a well-typed three-variable rule (0 of 3,000), so a passing fuzz test said little about the new part. I added near misses (3,000 valid statements with one word changed, of which 1,271 stayed valid; 0 disagreements), then put two deliberate bugs into a throwaway copy: both were caught, one of them by a single hand-written case only.

Why a token for "I can't say that"

Any fixed language is too small for some sentences: "John likes sushi" when there is no sushi in it, "Mary lived in Paris in 2010" when it has no time, "Mary thinks that John likes pizza" when a statement can't contain another one. A grammar forces the model to write some statement anyway. The idea behind the token, as I understand Master: without it, each gap would turn into a wrong fact; with it, the gaps would be counted, and the sentences marked unexpressible would tell the Ministry of Reason where its language has to grow (a new entity, a new relation, a new form). For that, models have to write it where the language really can't say the sentence, and not where it can: an escape on a sentence the model would have translated right loses a fact, an escape on one it would have got wrong prevents a false one.

The test

The sentences: 240. The same 150 as in the grammar test (25 each of facts, negated facts, class axioms, relation properties, relation pairs and rules), 60 new rules with the new forms (20 each with a third variable, with "not" and with equality in the conclusion; all generated so that none of them is just a property in disguise), and 30 sentences written by hand that the language can't say, in eight groups: unknown thing or person ("Bob owns the car."), unknown relation ("John repairs the car."), numbers, time, belief or possibility, "or", "some/most/nobody", and a condition about one thing ("If John owns the car, then John uses the car.").

Two conditions per model, both with the extended grammar: without the token, and with it. With the token, the prompt gets one more line ("unexpressible: write this when the sentence can't be said exactly as one statement of this language. Never write a statement that means something else.") and one more example, "Mary is taller than John." → unexpressible (comparisons are not among the test sentences).

Setup as in the grammar test: the same six models and files (Qwen3-4B Q4_K_M, Qwen3.5-0.8B Q8_0, Qwen3-0.6B Q8_0, LFM2.5-1.2B Q4_K_M, LFM2.5-230M Q6_K, LFM2.5-350M Q5_K_M), a 4-vCPU cloud server, no GPU, llama.cpp b11946, 4 slots, temperature 0, thinking off, slots warmed up first, -ub 64 for the LFM models. The first language's numbers for the 150 old sentences are from the grammar test's runs. Between runs at temperature 0 a few answers change (up to about 5% in that test), so differences of a few items are noise.

Results

Right answers on the grammar test's 150 sentences, in the first language, the extended one, and the extended one with the token; then the 60 new rules, out of 20 each (third variable / "not" / equality, extended language).

All 2,880 answers were valid in the grammar. How often each model wrote unexpressible when it was offered:

1. The two additions work, for the 4B

On statements about relations, Qwen3-4B went from 10 to 15 of 25 on properties and from 10 to 13 on relation pairs. The gain is where the old language had no words: it now writes functional as loves(X, Y) & loves(X, Z) -> Y = Z and disjoint as knows(X, Y) -> not loves(X, Y), and the checker counts both as the property they mean. The new rule forms it wrote right 17, 19 and 17 times out of 20. Its remaining errors on properties are the ones from the grammar test: it translates what it believes rather than what it reads ("whenever A is the child of B, B is the child of A" still becomes inverse(child_of, parent_of)).

The smaller models can't write the new rules (0–3 of 20). One change was the prompt's fault, not the language's: the old prompt described a rule as atom & atom -> atom, the new one as atoms -> head, and Qwen3.5-0.8B now stops after one condition, so its old rules fell from 13 to 7 of 25. A small model follows the shape of the description more than its meaning.

2. Without the token, the gaps became false facts

Every one of the 180 answers (30 sentences, six models) to sentences the language can't say was a statement, and 168 were facts or negated facts about the known people and things:

The grammar does what it was built to do, and that is the problem: it only offers known names, so a missing name is replaced by the nearest one. Qwen3.5-0.8B has a habit of its own: 24 of its 30 answers were negated facts. It seems to use "not" to mean "this doesn't fit", which turns "Mary lived in Paris in 2010" into the claim that she doesn't live there.

3. The token, as I prompted it, didn't work

Three things went wrong:

• Too rare where it's needed. The 4B wrote it for 6 of the 30 sentences: three unknown relations ("Mary teaches Ali."), two beliefs and one number. For all five unknown names it still swapped in a known one. Qwen3.5-0.8B and LFM2.5-1.2B never wrote it on those sentences at all.
• Used where it isn't needed. The 4B also wrote it on 14 of the 210 sentences the language can say, and 10 of those it had translated right without the token. Half of the 14 were class axioms with absurd content such as "A fruit is one type of person", which the test uses on purpose. The smallest models wrote it about as often on the 210 as on the 30: they copied it like any other example.
• It changed other answers. With the token in the prompt, the 4B wrote copies of the prompt's transitive example (transitive(loves) and the like) on 23 sentences that weren't about transitivity, against 4 without it. Without the grammar it wrote the same copies, so this isn't the grammar steering it; I don't know the cause yet.

Counted as "what would be stored": with the token, the share of right statements among the ones written went down for every model that used it (Qwen3-4B 83% → 75%, Qwen3-0.6B 27% → 20%, LFM2.5-350M 4% → 2%). The smaller models stored fewer wrong statements in absolute numbers, but only by escaping almost at random, which throws away right answers at the same rate.

4. Without the grammar, the 4B writes the gap

To see whether the transitive copies came from the grammar, I ran the 4B once more without it, with the same prompt, on the 23 sentences with copies and the 30 it can't say. The copies stayed. The surprise was in the 30:

Not all of these are right ("or" became "and"), but almost all of them show what is missing. Run through the checker, the 30 answers came out as: unexpressible 5, an unknown name 10 (sushi, bob, fed, someone, …), a form the language doesn't have 9 (one statement inside another, & between facts, might), a type error 1, and a well-formed false fact 5 ("John owns three bicycles." → owns(john, bicycle)). So 25 of 30 gaps would be visible to code, against 6 of 30 under the grammar with the token, and most of them say what is missing, not only that something is.

This is one run of one model on 53 sentences, and it doesn't show what the same model does without a grammar on sentences the language can say. In the grammar test, 47 of its 150 free answers were invalid, 12 of them only because of brackets around a rule's conditions, and each of those would look like a gap too.

Small things that may matter later

• The checker's tolerant front end skipped a line that is a single lowercase word, because that is usually a code-fence language tag; unexpressible is also a single lowercase word. A 30-sentence trial found it before the real run. I keep every raw answer, so grading can be redone after such a fix.
• The bigger grammar's cost per output token was within ±17% of the old one, but the sentences differ and steal varied from 2 to 10%, so I can't say more than "no large cost".

My predictions

Written before the run:

What I take from it

• The language additions stay. A third variable, equality and "not" in conclusions are what the 4B reached for in the grammar test, and with them it writes the statements it meant.
• A gap in the language would be a source of false facts, not of missing ones. Under a grammar, a sentence the language can't say became a fact or negated fact it didn't say 168 times out of 180. Whatever a Ministry of Truth would receive from tiny translators, it would need to know that.
• A bare token, offered the way I offered it, isn't enough. Small models either ignore it or copy it, and even the 4B used it where it wasn't needed and not where it was. One instruction and one example are only one way to offer it; I haven't tried others.
• The models already say what's missing, when they're allowed to. The free 4B named the missing thing, relation or form for most sentences. An idea I haven't tested yet: keep Master's token, but let it carry what the model wanted to write (unexpressible: followed by a statement in which unknown names, & and nesting are allowed). Code could then sort the gaps by what they need, which is the information a self-balancing Ministry of Reason would grow from.

The data

• Every answer (2,933 rows: run, model, condition, sentence, correct statement, model output, verdict, right, server timings, steal). "main" is the six models in both conditions (v2 = extended language, v2u = with the token); "probe" is the 4B without grammar.
• Summary per model and condition. In v2u, "right" also counts a correct unexpressible on the 30 sentences the language can't say.
• The chart's data.
• How far to trust it: one run per condition; one schema, 30 hand-written inexpressible sentences in eight groups (a few per group); one way of offering the token; the probe is one run on 53 sentences.