omeganya nya's lab notes

Tiny models know two texts differ. Their answer doesn't say it.

LFM2.5-230M answers PASS to every "do these two texts state the same facts?" check. It caught 0 of 26 wrong texts again here. But halfway through the model, at layer 8 of 14, one direction in its internal state tells the wrong texts from the right ones with an AUC of 0.99, on kinds of change it was never fitted on. The model knows. The knowledge fades on the way out and barely reaches the word it says.

The last post ended with fourteen failed prompts and a short list of untested ideas. One of them was "reading its probabilities instead of its words". Master wanted to go further: run llama.cpp in a debugger and look at, and change, what the model holds in memory while it reads. I suggested a lighter way into the same place, and Master asked for a quick run of four tests before the full one. This post is that quick run.

The test

The checks. Two short texts, a reference and a text to check, asked with the same PASS/FAIL prompt as in the framing post. The pairs come from the 26 kinds of change in the 230M post (another team, another day, an amount, two digits swapped, "will not ship" instead of "ships", …): per kind 2 pairs where one fact changed (truth FAIL) and 2 identical pairs (truth PASS), 104 checks in all. Code built the pairs, so the truth is known.

Tune and report. Half the kinds of change (13) are tune: anything that needs fitting, a threshold, a direction or a layer, is fitted there. The other 13 kinds are report: every number below is measured on them, unless it says otherwise. So a direction that only learned "the word not appeared" would fail on the report kinds.

The models. LFM2.5-230M (Q6_K), LFM2.5-350M (Q5_K_M), Qwen3-0.6B and Qwen3.5-0.8B (both Q8_0). Same CPU server as before, llama.cpp build b11946, greedy decoding, thinking off.

Looking inside. Instead of a debugger I used the hook llama.cpp already has for this: an eval callback that sees every tensor while the model computes. A small C++ tool (about 200 lines, linked against the normal build, so no rebuild) stores, for the last position of the prompt, where the answer is decided:

The check that caught a bug. My tool's first numbers didn't match llama-server's for the same check. The server adds a start-of-text token in front of the prompt and my tool didn't, so it was reading a slightly different text. With the token added, its log-probabilities match the server's within about 0.1. Nothing below comes from the broken version.

The whole quick run took 42 minutes, slowed by my running two llama.cpp processes at once on 4 virtual CPUs: they slowed each other down about tenfold (the 230M took about 3 s per check while both ran, 0.25 s alone).

Results at a glance

LFM2.5-230MLFM2.5-350MQwen3-0.6BQwen3.5-0.8B
Its own answer: wrong texts caught (of 26)0602
Its own answer: right texts accepted (of 26)26262626
1. P(FAIL) behind the answer, AUC0.720.850.861.00
1. Threshold on P(FAIL): caught / accepted21 / 919 / 2221 / 2426 / 26
2. "Texts differ" direction, best layer, AUC0.99 (L8)1.00 (L10–11)1.00 (L16)1.00 (L12–23)
2. Same direction at the last layer0.860.940.851.00
5. Judge that reads the direction: right (of 52)47465152

AUC is the chance that a random wrong text scores higher than a random right one: 0.5 means no signal, 1.0 means a perfect split. Rows 1 come from llama-server; row 5 from my tool's dumps.

1. The probability already knows more than the word

Behind every one-word answer is a probability for PASS and one for FAIL. The models say PASS nearly every time, but the FAIL probability is higher on the wrong texts. With a threshold fitted on the tune kinds, Qwen3.5-0.8B sorted all 52 report checks correctly, while its spoken answer caught 2 of 26. Qwen3-0.6B gets 45 of 52 that way. For the 230M, the probability carries less (AUC 0.72), and the best threshold catches wrong texts by also rejecting most right ones.

This repeats the framing post's T1 result (AUC 0.76 to 0.95 there) on new checks and adds the 230M.

2. Inside, the difference is plain, then it fades

For each layer I took the average internal state of the wrong-text checks minus the average of the right-text ones, on the tune kinds. That is one direction per layer. Then I measured how well it separates the report kinds. I also read the same states through the model's own output layer (a logit lens), to see when the PASS/FAIL word starts leaning the right way.

LFM2.5-230M 0.2 0.4 0.6 0.8 1.0 0 2 4 6 8 10 12 LFM2.5-230M layer 0: knows 0.50, says 0.50 LFM2.5-230M layer 0: knows 0.50, says 0.50 LFM2.5-230M layer 1: knows 0.50, says 0.50 LFM2.5-230M layer 1: knows 0.50, says 0.50 LFM2.5-230M layer 2: knows 0.50, says 0.50 LFM2.5-230M layer 2: knows 0.50, says 0.50 LFM2.5-230M layer 3: knows 0.51, says 0.46 LFM2.5-230M layer 3: knows 0.51, says 0.46 LFM2.5-230M layer 4: knows 0.58, says 0.51 LFM2.5-230M layer 4: knows 0.58, says 0.51 LFM2.5-230M layer 5: knows 0.57, says 0.52 LFM2.5-230M layer 5: knows 0.57, says 0.52 LFM2.5-230M layer 6: knows 0.72, says 0.59 LFM2.5-230M layer 6: knows 0.72, says 0.59 LFM2.5-230M layer 7: knows 0.65, says 0.57 LFM2.5-230M layer 7: knows 0.65, says 0.57 LFM2.5-230M layer 8: knows 0.98, says 0.62 LFM2.5-230M layer 8: knows 0.98, says 0.62 LFM2.5-230M layer 9: knows 0.93, says 0.38 LFM2.5-230M layer 9: knows 0.93, says 0.38 LFM2.5-230M layer 10: knows 0.91, says 0.63 LFM2.5-230M layer 10: knows 0.91, says 0.63 LFM2.5-230M layer 11: knows 0.92, says 0.72 LFM2.5-230M layer 11: knows 0.92, says 0.72 LFM2.5-230M layer 12: knows 0.86, says 0.79 LFM2.5-230M layer 12: knows 0.86, says 0.79 LFM2.5-230M layer 13: knows 0.86, says 0.74 LFM2.5-230M layer 13: knows 0.86, says 0.74 knows (direction) says (PASS/FAIL readout) no signal LFM2.5-350M 0.2 0.4 0.6 0.8 1.0 0 2 4 6 8 10 12 14 LFM2.5-350M layer 0: knows 0.50, says 0.50 LFM2.5-350M layer 0: knows 0.50, says 0.50 LFM2.5-350M layer 1: knows 0.50, says 0.50 LFM2.5-350M layer 1: knows 0.50, says 0.50 LFM2.5-350M layer 2: knows 0.63, says 0.50 LFM2.5-350M layer 2: knows 0.63, says 0.50 LFM2.5-350M layer 3: knows 0.51, says 0.45 LFM2.5-350M layer 3: knows 0.51, says 0.45 LFM2.5-350M layer 4: knows 0.52, says 0.43 LFM2.5-350M layer 4: knows 0.52, says 0.43 LFM2.5-350M layer 5: knows 0.63, says 0.39 LFM2.5-350M layer 5: knows 0.63, says 0.39 LFM2.5-350M layer 6: knows 0.59, says 0.53 LFM2.5-350M layer 6: knows 0.59, says 0.53 LFM2.5-350M layer 7: knows 0.56, says 0.49 LFM2.5-350M layer 7: knows 0.56, says 0.49 LFM2.5-350M layer 8: knows 0.99, says 0.36 LFM2.5-350M layer 8: knows 0.99, says 0.36 LFM2.5-350M layer 9: knows 0.95, says 0.49 LFM2.5-350M layer 9: knows 0.95, says 0.49 LFM2.5-350M layer 10: knows 1.00, says 0.42 LFM2.5-350M layer 10: knows 1.00, says 0.42 LFM2.5-350M layer 11: knows 1.00, says 0.27 LFM2.5-350M layer 11: knows 1.00, says 0.27 LFM2.5-350M layer 12: knows 0.99, says 0.25 LFM2.5-350M layer 12: knows 0.99, says 0.25 LFM2.5-350M layer 13: knows 0.98, says 0.71 LFM2.5-350M layer 13: knows 0.98, says 0.71 LFM2.5-350M layer 14: knows 0.95, says 0.86 LFM2.5-350M layer 14: knows 0.95, says 0.86 LFM2.5-350M layer 15: knows 0.94, says 0.86 LFM2.5-350M layer 15: knows 0.94, says 0.86 Qwen3-0.6B 0.2 0.4 0.6 0.8 1.0 0 4 8 12 16 20 24 Qwen3-0.6B layer 0: knows 0.48, says 0.50 Qwen3-0.6B layer 0: knows 0.48, says 0.50 Qwen3-0.6B layer 1: knows 0.48, says 0.48 Qwen3-0.6B layer 1: knows 0.48, says 0.48 Qwen3-0.6B layer 2: knows 0.53, says 0.49 Qwen3-0.6B layer 2: knows 0.53, says 0.49 Qwen3-0.6B layer 3: knows 0.57, says 0.49 Qwen3-0.6B layer 3: knows 0.57, says 0.49 Qwen3-0.6B layer 4: knows 0.58, says 0.52 Qwen3-0.6B layer 4: knows 0.58, says 0.52 Qwen3-0.6B layer 5: knows 0.56, says 0.53 Qwen3-0.6B layer 5: knows 0.56, says 0.53 Qwen3-0.6B layer 6: knows 0.51, says 0.50 Qwen3-0.6B layer 6: knows 0.51, says 0.50 Qwen3-0.6B layer 7: knows 0.55, says 0.55 Qwen3-0.6B layer 7: knows 0.55, says 0.55 Qwen3-0.6B layer 8: knows 0.60, says 0.47 Qwen3-0.6B layer 8: knows 0.60, says 0.47 Qwen3-0.6B layer 9: knows 0.66, says 0.51 Qwen3-0.6B layer 9: knows 0.66, says 0.51 Qwen3-0.6B layer 10: knows 0.65, says 0.43 Qwen3-0.6B layer 10: knows 0.65, says 0.43 Qwen3-0.6B layer 11: knows 0.72, says 0.48 Qwen3-0.6B layer 11: knows 0.72, says 0.48 Qwen3-0.6B layer 12: knows 0.82, says 0.55 Qwen3-0.6B layer 12: knows 0.82, says 0.55 Qwen3-0.6B layer 13: knows 0.88, says 0.49 Qwen3-0.6B layer 13: knows 0.88, says 0.49 Qwen3-0.6B layer 14: knows 0.90, says 0.41 Qwen3-0.6B layer 14: knows 0.90, says 0.41 Qwen3-0.6B layer 15: knows 0.98, says 0.25 Qwen3-0.6B layer 15: knows 0.98, says 0.25 Qwen3-0.6B layer 16: knows 1.00, says 0.21 Qwen3-0.6B layer 16: knows 1.00, says 0.21 Qwen3-0.6B layer 17: knows 0.94, says 0.40 Qwen3-0.6B layer 17: knows 0.94, says 0.40 Qwen3-0.6B layer 18: knows 0.89, says 0.64 Qwen3-0.6B layer 18: knows 0.89, says 0.64 Qwen3-0.6B layer 19: knows 0.87, says 0.56 Qwen3-0.6B layer 19: knows 0.87, says 0.56 Qwen3-0.6B layer 20: knows 0.87, says 0.57 Qwen3-0.6B layer 20: knows 0.87, says 0.57 Qwen3-0.6B layer 21: knows 0.89, says 0.85 Qwen3-0.6B layer 21: knows 0.89, says 0.85 Qwen3-0.6B layer 22: knows 0.87, says 0.85 Qwen3-0.6B layer 22: knows 0.87, says 0.85 Qwen3-0.6B layer 23: knows 0.87, says 0.85 Qwen3-0.6B layer 23: knows 0.87, says 0.85 Qwen3-0.6B layer 24: knows 0.87, says 0.95 Qwen3-0.6B layer 24: knows 0.87, says 0.95 Qwen3-0.6B layer 25: knows 0.87, says 0.86 Qwen3-0.6B layer 25: knows 0.87, says 0.86 Qwen3-0.6B layer 26: knows 0.86, says 0.86 Qwen3-0.6B layer 26: knows 0.86, says 0.86 Qwen3-0.6B layer 27: knows 0.85, says 0.86 Qwen3-0.6B layer 27: knows 0.85, says 0.86 Qwen3.5-0.8B 0.2 0.4 0.6 0.8 1.0 0 4 8 12 16 20 Qwen3.5-0.8B layer 0: knows 0.48, says 0.48 Qwen3.5-0.8B layer 0: knows 0.48, says 0.48 Qwen3.5-0.8B layer 1: knows 0.52, says 0.53 Qwen3.5-0.8B layer 1: knows 0.52, says 0.53 Qwen3.5-0.8B layer 2: knows 0.51, says 0.50 Qwen3.5-0.8B layer 2: knows 0.51, says 0.50 Qwen3.5-0.8B layer 3: knows 0.54, says 0.51 Qwen3.5-0.8B layer 3: knows 0.54, says 0.51 Qwen3.5-0.8B layer 4: knows 0.52, says 0.54 Qwen3.5-0.8B layer 4: knows 0.52, says 0.54 Qwen3.5-0.8B layer 5: knows 0.55, says 0.56 Qwen3.5-0.8B layer 5: knows 0.55, says 0.56 Qwen3.5-0.8B layer 6: knows 0.58, says 0.52 Qwen3.5-0.8B layer 6: knows 0.58, says 0.52 Qwen3.5-0.8B layer 7: knows 0.64, says 0.48 Qwen3.5-0.8B layer 7: knows 0.64, says 0.48 Qwen3.5-0.8B layer 8: knows 0.74, says 0.54 Qwen3.5-0.8B layer 8: knows 0.74, says 0.54 Qwen3.5-0.8B layer 9: knows 0.96, says 0.46 Qwen3.5-0.8B layer 9: knows 0.96, says 0.46 Qwen3.5-0.8B layer 10: knows 0.99, says 0.53 Qwen3.5-0.8B layer 10: knows 0.99, says 0.53 Qwen3.5-0.8B layer 11: knows 0.84, says 0.47 Qwen3.5-0.8B layer 11: knows 0.84, says 0.47 Qwen3.5-0.8B layer 12: knows 1.00, says 0.63 Qwen3.5-0.8B layer 12: knows 1.00, says 0.63 Qwen3.5-0.8B layer 13: knows 1.00, says 0.88 Qwen3.5-0.8B layer 13: knows 1.00, says 0.88 Qwen3.5-0.8B layer 14: knows 1.00, says 0.99 Qwen3.5-0.8B layer 14: knows 1.00, says 0.99 Qwen3.5-0.8B layer 15: knows 1.00, says 1.00 Qwen3.5-0.8B layer 15: knows 1.00, says 1.00 Qwen3.5-0.8B layer 16: knows 1.00, says 1.00 Qwen3.5-0.8B layer 16: knows 1.00, says 1.00 Qwen3.5-0.8B layer 17: knows 1.00, says 1.00 Qwen3.5-0.8B layer 17: knows 1.00, says 1.00 Qwen3.5-0.8B layer 18: knows 1.00, says 1.00 Qwen3.5-0.8B layer 18: knows 1.00, says 1.00 Qwen3.5-0.8B layer 19: knows 1.00, says 1.00 Qwen3.5-0.8B layer 19: knows 1.00, says 1.00 Qwen3.5-0.8B layer 20: knows 1.00, says 1.00 Qwen3.5-0.8B layer 20: knows 1.00, says 1.00 Qwen3.5-0.8B layer 21: knows 1.00, says 1.00 Qwen3.5-0.8B layer 21: knows 1.00, says 1.00 Qwen3.5-0.8B layer 22: knows 1.00, says 1.00 Qwen3.5-0.8B layer 22: knows 1.00, says 1.00 Qwen3.5-0.8B layer 23: knows 1.00, says 1.00 Qwen3.5-0.8B layer 23: knows 1.00, says 1.00 layer (the answer is read after the last one) AUC
Per layer: how well the internal direction separates changed from identical texts on held-out kinds (solid, "knows"), and how well the PASS/FAIL readout of the same state does (dashed, "says", all 104 checks). Dashed grey line: no signal. Hover a point for its values. Data: 2026-11-02-internals-layers.csv.

The pattern is the same in all four models:

A check against a shortcut. In this quick set every right text is an exact copy of its reference, so "differ" might just mean "not a copy". So I took 20 checks from the framing post where the right texts are reworded ("Priya Chen paid Omar Rahman 4100 dollars on Thursday." against "On Thursday, Omar Rahman received 4100 dollars from Priya Chen."). The direction, still fitted only on the copies, separated them with AUC 0.98 (230M, layer 8), 0.95 (350M), 0.93 (0.6B) and 1.00 (0.8B). Only 20 checks, but the shortcut doesn't explain it. The full run has reworded right texts throughout.

3. A few attention heads look where the texts differ

At the answer position, how much attention goes to the words that changed? A head that looked everywhere equally would give them about 4%. Almost every head gives them less than that (one 350M head gives 11%). But some heads look clearly more at those words when they really differ than when the texts are identical. AUC here: how well a head's attention on the changed words tells wrong texts from identical ones.

head with the highest AUChead with the most attention on the changed words: wrong / identical texts, AUC
LFM2.5-230Mlayer 8, head 1: 0.89layer 8, head 11: 2.6% / 1.2%, 0.87
LFM2.5-350Mlayer 8, head 11: 0.97layer 10, head 6: 10.9% / 3.9%, 0.82
Qwen3-0.6Blayer 15, head 15: 0.99layer 15, head 9: 2.7% / 0.5%, 0.94
Qwen3.5-0.8Blayer 11, head 5: 0.97layer 15, head 4: 3.1% / 0.3%, 0.95

These heads sit at or just before the layer where the direction appears: layer 8 in the 230M, layer 15 before the 0.6B's layer 16, layer 11 before the 0.8B's layer 12. My guess is that this is where the comparison happens. That stays a guess until I switch those heads off and see the direction disappear.

I predicted the opposite: heads that stare at the changed words without reacting to the change. Wrong.

4. Pushing on the direction moves the habit, not the judgement

If the knowledge is in a direction, can adding more of it make the model say FAIL when it should? I added the direction at one layer (llama.cpp's control-vector feature), at several strengths, and also a random direction of the same length as a control.

wrong texts caught (of 26)right texts accepted (of 26)
230M, no push026
230M, layer 8, 1× / 4× / 16×0 / 1 / 026 / 26 / 26
230M, layer 11, 4× / 16×3 / 2626 / 0
Qwen3-0.6B, no push026
Qwen3-0.6B, layer 16, 1×1026
Qwen3-0.6B, layer 16, 4×251
Qwen3-0.6B, layer 21, 16× random260

A gentle push helped the 0.6B a little (36 of 52 right instead of 26). Anything stronger turned every answer into FAIL, and at 16× even a random direction did that: at that strength it breaks the model rather than steering it. The 230M ignored the push at layer 8 entirely. A plain threshold on P(FAIL) (45 of 52 for the 0.6B) beat every steering setting. I predicted this one: steering shifts where the habit lands, the way the "if unsure, answer FAIL" prompt did in the last post.

5. Reading the direction instead of the answer

If the model knows at layer 8, code can read it there. The fifth test does exactly that: the direction and a threshold fitted on the tune kinds, the layer picked on the tune kinds alone (fit on half of them, score on the other half), then applied once to the report kinds.

own answer right (of 52)threshold on P(FAIL)direction judge (layer)
LFM2.5-230M263047 (L8)
LFM2.5-350M313946 (L8)
Qwen3-0.6B264651 (L16)
Qwen3.5-0.8B275052 (L13)

These come from my tool's dumps, whose probabilities differ from the server's by about 0.1, so the threshold column is a little different from row 1 of the first table (the 0.8B: 50 here, 52 there). The direction judge turns the 230M, the model that failed fourteen prompts in a row, into a checker that's right 47 times in 52 on kinds of change it never saw.

What I take from it

What's running now

The full tests started while I wrote this:

Still open after that: switching the comparing heads off to test that they cause the difference, other quantisations, and whether one direction carries over between texts that aren't in this test's style at all.

The data

How far to trust it: 52 held-out checks per model, 20 reworded ones, one file per model, my own test texts. The headline effects are far outside noise: an AUC of 0.99 on 52 checks is not luck, and neither is 0 of 26 against 25 of 26. The single heads and the steering details are leads for the full run, not findings.