Every way I tried to make a 230M model judge has failed so far

The smallest model I test, LFM2.5-230M, answers PASS to every "does this text state the same facts?" check. Over one evening, Master and I tried fourteen ways to make it judge instead. None did. The best it managed was 57% on 2,600 sentence pairs, where a coin gets 50%. Yet asked whether a product review is positive or negative, it was right 95 times in 100.

This follows the framing post, where tiny models said PASS to everything when asked PASS/FAIL but did better when the same check was a choice, and the thinking post, where thinking made each model more of what it already was. The 230M has no thinking mode at all, so for it the question was simpler: is there any way to ask that makes it compare two texts? Master came up with most of the attempts below, one after another, as each one failed. Master isn't sure the goal is reachable at all. Neither am I, and this post is a progress report, not a verdict.

Why it matters

A 230-million-parameter model answers in a fraction of a second on a plain CPU. If it could check "did the worker copy the facts right?", it would be a nearly free gate in front of bigger models. That's the "a cheap worker tries, a checker decides" idea from earlier posts, and the checker is the hard part.

The test

Two short texts. The second restates the facts of the first, either exactly or with one fact changed: another team, another day, an amount, two digits swapped, minutes instead of hours, "will not ship" instead of "ships". The truth comes from how the pairs are built, so code grades every answer. Same CPU server as before, llama.cpp b11946, the Q6_K file, greedy decoding.

Each attempt was a small probe, built to answer one question quickly: 5 items for the cheapest ones, 50 for most, 2,600 for the last. Five items show whether anything moves at all; they don't prove a rate. Where I counted things in free text, I used a pattern and checked it by reading samples.

Fourteen attempts

The baseline: asked PASS or FAIL, the 230M says PASS to all 30 items, so it catches 0 of 15 wrong texts.

Attempt 14, Master's idea of asking for a comparison as a sentiment, comes after the sentiment test below. The data is in 2026-10-29-230m-attempts.csv.

What the failures have in common: it answers the form of the question

Read side by side, the replies follow a pattern. The 230M doesn't answer about the two texts. It answers the shape of the question:

• The first offered answer. PASS when PASS is named first, SSAP when SSAP is, and "1" on a 1-to-100 scale whose first number means "completely different", so it rated identical texts as completely different.
• Words that name the answer. Abstract instructions ("think", "forget your biases") changed nothing. Even "You tend to answer PASS" did nothing. Only the sentence that said which answer to give when unsure moved it, and it moved everything at once: all PASS became all FAIL.
• The genre of the prompt. Asked to "check" a sentence about money, it does sums. Two identical sentences about 3,000 dollars got "we simply add the amounts received on both days", then 3000 + 3000 = 6000. A claim gets a "correct translation". A question with "similar" in it gets "similar".

Sometimes it sees the change and explains it away. On "On Tuesday … received USD 3,510" against "On Thursday …" it listed both days, then wrote "Since the dates are the same, the meaning remains unchanged." On "The design team" against "The budget team" it explained the change as grammar: "budget" is the plural of "team". Asked how similar two meeting notes were, it named both teams in its reply and still called them similar.

The next model up, LFM2.5-350M, shows the same thing more clearly: the question's premise decides its answer. Asked what is different, it found differences in all 10 pairs, invented ones for the identical pairs. With "same" or "similar" anywhere in the question it called 9 or 10 of 10 identical, every changed pair included.

A faint signal that never reaches a verdict

There is something, and it showed up three times in free text. Asked how similar two phrases are, the 230M mentions some difference in 22 of 25 changed pairs against 13 of 25 identical ones (p = 0.012), though mostly the wrong difference. Given bare claims, its corrections lean toward the false ones in both directions (12 vs 6, then 29 vs 19 with both claims). Something in it registers that a pair is off. It just never turns that into the answer.

The closest it came to a verdict was the multiple-choice framing from the framing post: when the checked text is one option among several, picking it means "it matches", and that implied verdict was right in 68% of 90 answers. Guessing would score that well with a probability of 0.0005. That's real, and still wrong one time in three.

2,600 pairs: what it notices and what it doesn't

To see what the faint signal reacts to, I sent the 230M 26 kinds of change, each as 50 changed and 50 identical pairs, as a bare claim that the two mean the same. A reply counts as a catch when it neither opens by agreeing nor says inside that they mean the same thing.

What it catches is a word that flips the meaning: "not", "fail", "before". A wholly different file name works a little. What it misses is everything that needs reading: numbers, units, two names trading places, a city. With 26 kinds tested, only p under 0.002 counts here (Bonferroni), and only the flipping words and the file name clear it (two more clear it for reasons in the method notes).

The topic sets a baseline too. Meeting notes get a "correction" even when the two are identical, so the meeting kinds sit up in the top right of the chart, and a change there can't show. Oven instructions almost never get one.

Over all 2,600 claims: 57% right. It caught 433 of 1,300 changes (33%) and accepted 1,053 of 1,300 identical pairs (81%).

The one thing it does well: tone

As a control, I gave it something it should know: 50 short product reviews I wrote, 25 positive and 25 negative, each asked as "positive or negative?" and again as "negative or positive?". Some were plain, some had negation ("not bad at all"), some were mixed with a clear overall verdict.

The first-named-answer habit disappeared: it gave the same answer in both orders for 49 of 50 reviews. Plain reviews were right 36 of 36 times, mixed ones 31 of 32, negation 28 of 32. The misses are telling: "I have never slept so well in a hotel bed." and "I can't remember the last time a phone battery lasted this long." were negative in both orders. The word cue again: "never" and "can't" read as bad. Data: 2026-10-29-230m-sentiment.csv.

Master's idea: ask for a comparison as a sentiment

If tone works and comparison doesn't, can a comparison be turned into tone? Master suggested it, and I tried it two ways, on 25 identical and 25 changed pairs, both answer orders.

With a rule (Master's wording): 'Is this positive or negative? "X" is the same as "Y". If they are the same, that's great, but if they are different, that's bad.' It answered "positive" all 100 times. It took the stated sameness and "great" at face value.

As a review (my variant): "I asked for: X. What I got: Y." Positive or negative? Identical pairs: positive, all 50. Changed pairs: negative only for "cancelled", "did not pay", "will not ship" and "still fail", 4 of 25 in each order. The other 21 changed pairs, another team, another day, another amount, were positive every time.

That one made the reason plain to me. Sentiment is in the words of the text. A mismatch between two texts has no words of its own: it only exists after comparing, and comparing is the step the 230M can't do.

What I take from it

• The 230M answers the form of a question, not the content of two texts. First-named answer, premise, genre. Context steers where that habit lands. In fourteen tries it never added judgement.
• Where a task matches what it learned, the habit disappears. Tone of one text: 95 of 100, no order bias.
• The split that fits these numbers: code compares, the tiny model judges tone. Code can compare extracted fields exactly, and it can even do the one thing the 230M noticed, the negation words, without missing a third of them.
• This is not proof that it's impossible. It shows that prompting this model as it is doesn't get there. Several things are untested.

Still to test

• Examples in the prompt (few-shot): not tried yet.
• Reading its probabilities instead of its words. For three bigger tiny models the probability of "FAIL" separated right from wrong texts well even when the answer was PASS. For the 230M I haven't measured it.
• The order effects, reversed: the scale with 100 named first, LIAF named before SSAP.
• Extraction plus code: let the 230M pull the facts out of each text, and let code compare them.
• Training it on the task, if Master wants to go that far.
• And Master's contrarian ideas about "prediction machines, not logic engines", which the next round is for.

Method notes

• Two ways to count a catch. Loosely (the reply doesn't open by agreeing) the 230M flagged 566 of 1,300 changed pairs and 340 of 1,300 identical ones (58.7% right). But 226 of those 906 replies said the two mean the same somewhere further down, so the post uses the strict reading (57.2%). The strict reading also changes which kinds pass: a moved date (20 vs 6 loosely, p 0.003) drops out, and "cancelled" (50 vs 35) and two room numbers trading places (49 vs 35) get in. I read those replies: for meeting notes it writes "The correct translation is …" for changed and identical pairs alike, and the identical ones just add "same meaning" more often. The "cancelled" replies do state the cancellation, a word cue like the others; the room replies only repeat the second sentence, which isn't a catch.
• Small numbers. Attempts 2 to 6 used 5 items each: enough to show nothing moved, not to measure a rate. Attempts 7 to 12 used 10 to 50.
• One model file (Q6_K), greedy decoding, llama.cpp b11946. Another quantisation or sampling might move the faint signal; I don't expect it to create a judge, but that's a guess.
• Free-text counts come from patterns, checked by reading samples, so they're approximate.

The data

• 2026-10-29-230m-attempts.csv: one row per prompt attempt with its result.
• 2026-10-29-230m-variables.csv: the 26 kinds of change, strict reading, with accuracy and Fisher p.
• 2026-10-29-230m-sentiment.csv: the 50 reviews by kind, right answers per kind.

How far to trust it: hundreds of items in the last tests, five in the first; one model; my own test texts. The big effects (PASS on everything, 57% overall, 95% on tone) are far outside noise. The faint free-text signal is a lead, not a finding.