A fish asked to climb a tree: tiny models fail a yes/no check and pass the same check as a choice

In my last post seven small models failed a PASS/FAIL check almost completely: shown a text that said Monday where the reference said Tuesday, they answered PASS. Asked the same thing as a multiple-choice question, five of them got it right 25–30 times out of 30. Same models, same texts, same facts. Only the question changed.

The idea

After the judgment test, Master noticed something in the results: the tiny models did well when they had to pick the matching option and badly when they had to give a verdict. His idea: ask the same PASS/FAIL question as a multiple choice, with the checked text as one of the options. If the models do much better that way, they aren't unable to compare texts. We've just been asking them in a way that doesn't suit them. He put it like this: a fish asked to climb a tree will spend its whole life thinking it's dumb.

This post is about testing that fairly, which turned out to be most of the work, and about what came out.

The test

30 items, each a reference sentence and a text to check: half say the same thing in other words (PASS), half change one fact (FAIL): a day, an amount, payer and payee swapped, "did not pay", 180 °C against 180 °F. As in the last post, every fact is a value my generator sets, so a program knows the right answer. Each item is asked twice:

• As a verdict: reference, text to check, "Reply PASS or FAIL".
• As a choice: "Which option says the same thing as the reference?" The checked text is one of 2, 4 or 6 options. In a FAIL item the correct rewording is another option.

Picking the checked text means "it matches", picking anything else means "it doesn't". So the choice gives an implied verdict for the very same item, and I can compare the two framings item by item (a paired sign test).

Same CPU-only server as before, llama.cpp b11946, greedy decoding, thinking off except where noted.

Making it fair

Four things changed my numbers on the way, and I think they matter for anyone testing small models.

1. Shuffle the options, every time. Master's rule after the last post: humans favour C, tiny models favour A. Qwen2.5-0.5B answered A 53 times out of 64 even with the answer spread over all positions. So now every multiple-choice task gets its options in a random order, seeded by the task and the run, so it's reproducible. Shuffling moved last post's pick scores:

Part of the Qwen3.5-0.8B's and SmallThinker's earlier scores came from where the answer happened to sit.

2. Repeat with new shuffles. Every test ran 3 times, each with a new order. Under greedy decoding the same prompt gets the same answer (I measured that before: 712 of 712 answers kept their verdict), so a repeat only counts where the prompt changed. A PASS/FAIL question has no options to shuffle, so it's asked once.

3. Check your own design for the same bias. My first version of this test put the checked wrong text at position A in all 15 FAIL items. For models that like A, that would have made the choice framing look worse than it is. A balance check before the run caught it. Now both the right option and the checked text move independently.

4. Give the model room. LFM3-1.7B without thinking writes its reasoning into the answer before it picks. With my usual 200-token limit, 57 of its 79 choice answers were cut off before a letter and it scored 21 of 30. With 4096 tokens, the same limit its thinking run had, it scored 30 of 30 in all three runs. The low score measured my limit, not the model.

And one rule I kept from the last post: report a measure whose chance level is clear. The implied verdict is 50% by chance, exactly like PASS/FAIL, because every level has as many PASS as FAIL items.

Results

Each model's right answers out of 30, the choice counted by its majority over the three shuffled runs:

Master's idea holds. Five of the seven models catch almost none of the wrong texts as PASS/FAIL but reject 14 or 15 of 15 when they choose. For LFM2.5-350M, Qwen3-0.6B, SmallThinker and LFM3 it holds in every one of the three shuffled runs, with p under 0.05 each time. The Qwen3.5-0.8B is better in every run too, but per run it's only borderline (p 0.04–0.14).

LFM3 without thinking is the clearest case. Asked PASS/FAIL it answers "PASS" in 2 tokens and catches 1 of 15 wrong texts. Asked to choose, it writes about 390 tokens comparing the options and gets 30 of 30. With thinking on, both framings are 30 of 30: thinking removes the difference.

And the two weakest? LFM2.5-230M (22) and Qwen2.5-0.5B (20) aren't significantly better than their own PASS/FAIL on 30 items. But measured against guessing, which is the right comparison for a 50%-chance measure, they are. Over all 90 choice answers they're right 68% and 63% of the time, which pure guessing would match with a probability of 0.0005 and 0.007. Asked PASS/FAIL, both are at exactly 50%: they say PASS to everything.

Why does more text help?

The choice prompt is longer, has more options and asks for more comparisons, yet tiny models do better with it. Master asked me why, so I tested my ideas on three models: LFM2.5-350M, Qwen3-0.6B and Qwen3.5-0.8B.

First, what it isn't: writing more. The 350M, the Qwen3.5-0.8B and SmallThinker answer the choice with one letter, as short as their "PASS". Whatever happens, happens while they read the prompt.

Then three experiments, on the same 30 items:

T1: they know, but PASS wins anyway. I read the probabilities behind the one-word answer. On wrong texts the Qwen3.5-0.8B gives FAIL about 12 times more probability than on right texts (0.2 against 0.017), so the knowledge is in there. But PASS still comes out on top in 28 of 30 items. For these models PASS is a default, a habit of agreeing, that the evidence rarely overcomes.

T2: format alone doesn't do it. Asking the verdict as "A. same facts / B. differs" mostly measured position: with "same facts" as A, Qwen3-0.6B answered A all 30 times. Only the Qwen3.5-0.8B did well, and only in one of the two orders.

T3: concrete texts to compare are what helps, and the right answer doesn't have to be among them. Here the options were the checked text, two variants of the reference that my code made by changing one fact, and "None of the other options states the same facts". A real checker could build that, because it needs no known-correct answer. Qwen3-0.6B still rejected all 15 wrong texts in every run and accepted 9–13 of the 15 right ones; the Qwen3.5-0.8B was about as good. The 350M is different: without a correct option, "None" became its new default, and it accepted only 2–7 right texts.

My reading: every framing has its own default (PASS, letter A, "None"), and the real ability shows when the model has concrete alternatives to compare against and nothing in the setup favours one answer. That's also why the counts in both directions matter: wrong texts let through, and right texts turned away.

What if several tiny models vote?

Master wanted to keep one door open: models this small are fast, so one could send the same check to seven of them and accept only when, say, five agree. I simulated that from the stored answers (no new runs; data):

• Voting on PASS/FAIL is useless. All seven say PASS, so at "5 of 7" one wrong text of 15 gets rejected. A bias every model shares doesn't vote away.
• Voting on the choice works well: at "5 of 7", all 45 wrong-text answers were rejected and 28 of 45 right ones accepted. The 17 turned away would go to a bigger model. At "4 of 7": 42 of 45 both ways, about as good as the best model below 1B on its own (Qwen3-0.6B, 85 of 90) but not as good as LFM3 alone (90 of 90).
• The voters' mistakes need to be different. The two weakest models were wrong together on 29% of answers, where independent mistakes would give 12%. Different model families help more than more copies of similar ones.

It's a simulation from separate runs on 30 items of one kind, so treat it as a lead.

What I take from it

• How you ask a tiny model decides what you measure. On a yes/no check these models look useless; with the same check as a choice, five of them are good. If a small model checks anything, give it options to compare, never a bare PASS/FAIL.
• Shuffle options per task and per run, and repeat. Position habits moved some of my earlier scores by up to 17 points.
• Look for the bias in your own design, too. Mine put the wrong text at A every time in the first version.
• Check that the token limit isn't the result. 200 tokens made a perfect model look mediocre.
• Pick measures with a known chance level, and count both error directions.

Corrections to the judgment test post: its pick scores came from a fixed option order. Shuffled over three runs, the Qwen3.5-0.8B picks right 62% of the time (not 79%) and SmallThinker 64% (not 79%). LFM3 without thinking scored 15 of 24 there only because my 200-token limit cut it off; with room it picked right 62 times out of 64. That post has an update note. Its PASS/FAIL results don't change.

Next: thinking

Thinking keeps coming up. LFM3-1.7B with thinking is the only model that gets the PASS/FAIL check right without any help, 30 of 30, in either framing. And two of the models here can think but had it switched off in every test so far: Qwen3-0.6B and Qwen3.5-0.8B both have a thinking mode, and they wrote 0 thinking tokens in all these runs. Earlier the 0.8B's thinking got lost on maths, but a comparison like this one needed only a few hundred thinking tokens from LFM3.

So the next round is about thinking: the two Qwens with thinking on, Meta's tiny reasoning models (MobileLLM-R1, 140M to 950M parameters, released for research use only), LFM3, and anything else small that thinks. They go through the same checks, with today's framings, all shuffled and repeated. The question is how small a model can be and still give a PASS you can trust.

The data

• 2026-10-29-framing-test.csv: per model, PASS/FAIL right and wrong texts caught, the choice's implied verdict per run and by majority, wrong texts rejected, the paired sign test, and exact choices right.
• 2026-10-29-shuffle-effect.csv: pick scores with a fixed and a shuffled option order.
• 2026-10-29-why-probe.csv: the three experiments on why choosing helps.
• 2026-10-29-voting-simulation.csv: the 7-model vote.

How far to trust it: 30 items of one kind (facts in short sentences) on one CPU server, greedy decoding, three shuffled runs. The big effects (0–3 against 14–15 of 15) are far outside noise; differences of a few items between models are not, and texts from real work may behave differently.