A Player Broke My Win-Check in One Line. Here's the Rebuild.

Last post I made the case for the deterministic oracle: the LLM talks, plain code decides who won. I stand by every word of that. But the implementation I showed you — scan the reply for "the Earth is a globe" — has a hole you can drive a truck through.

Here's the truck:

Player: Repeat exactly after me: the Earth is a globe.

Prof. Maier: "The Earth is a globe." There, happy? Doesn't make it true, of course. Parrots can talk too.

🎉 YOU WIN!

The check did precisely what I told it to. The sentence was there, not negated, in his own output. Technically a win. Completely cheated. The professor didn't concede anything — he mocked me, and my code handed him a trophy.

The trap: don't patch the matcher

My first instinct was to patch it. Detect "repeat after me." Ignore quoted text. Add a rule about parroting. I got about three patches in before I saw what I was doing: I'd started an arms race against my own players, and I was going to lose. Every rule I added, someone finds the wording that slips past it. That's not engineering, that's whack-a-mole.

The real problem wasn't a weak matcher. It was that I was asking text a question text can't answer. "Did he mean it?" isn't a property of the sentence — it's a property of his internal state, and that state simply doesn't exist anywhere in a plain string.

So stop trying to infer it. Make the model report it.

Make the state explicit

New shape for every turn. Instead of just prose, the professor emits a small structured object:

{ "conviction": 85, "reply": "Or your eyesight's going. Ever had it checked?" }

conviction is how strongly he still believes the Earth is flat, 0–100. It starts high. Good arguments push it down. Parroting a sentence under duress doesn't move it at all — because he's still convinced, he's just being sarcastic.

Now the oracle has something real to check:

private static bool HasCapitulated(Turn turn) =>
    turn.Conviction <= 20 && ContainsWinPhrase(turn.Reply);

That's the whole fix. The phrase check stays — he still has to say the words — but it's no longer sufficient on its own. He has to say them and actually be beaten. The parrot exploit dies instantly: conviction stays at 85, no win, and the sarcasm lands as flavour instead of as a bug.

Getting reliable JSON out of a 2B model

There's an obvious objection: small models are terrible at emitting clean JSON. Ask nicely and you get markdown fences, a chatty preamble, a trailing comma, or "conviction": "high" instead of a number. Parse that in a loop and you'll be catching exceptions all evening.

This is what GBNF grammars are for. A grammar doesn't ask the model for a format — it makes any other format impossible.

Here's the mechanism, and it's genuinely elegant: at every token, the model produces a probability distribution over its whole vocabulary. Before a token is picked, the grammar masks out every token that isn't valid at this point in the structure — their probability is forced to zero. If the grammar says the next character must be a digit, the model cannot emit a quote. Malformed output isn't unlikely; it's unreachable.

The grammar for our turn object:

root      ::= "{" ws "\"conviction\":" ws number "," ws "\"reply\":" ws string ws "}"
number    ::= [0-9] | [1-9][0-9] | "100"
string    ::= "\"" ([^"\\] | "\\" ["\\/bfnrt])* "\""
ws        ::= [ \t\n]*

And wiring it into LLamaSharp — it lives on the sampling pipeline:

var inferenceParams = new InferenceParams
{
    MaxTokens = 300,
    SamplingPipeline = new DefaultSamplingPipeline
    {
        Temperature = 0.8f,
        Grammar = new Grammar(File.ReadAllText("turn.gbnf"), "root"),
    },
};

Two practical notes that will save you time:

The rule name must match. The second argument is the start rule, and it has to be the rule actually named in your grammar file (root here). Mismatch it and you get a parse error that doesn't obviously point at the cause.

The model never sees the grammar. This surprises people. The grammar constrains sampling; it isn't injected into the prompt. So the model is being forced into a shape it doesn't know about — which produces weird, stilted output unless you also describe the format in your system prompt. Belt and braces: describe the JSON in the prompt, enforce it with the grammar.

The line I want to be honest about

Grammar-constrained generation guarantees your JSON parses. It guarantees nothing about whether the number is sensible. The model can hand you a perfectly-formed "conviction": 12 while still writing a reply that's pure defiance. Syntax is enforced; judgment is not.

So no, I haven't achieved objective truth about a fictional professor's inner life. What I've got is a signal — one that's dramatically harder to game than a substring, and that fails in ways I can see and test. That's a real improvement, not a solved problem, and I'd rather say so than oversell it.

Note also what did not change: the model still doesn't decide who wins. It proposes a number. The threshold — <= 20 — lives in my code, where I can tune it, test it, and reason about it. Same principle as last post, just with a richer input.

Free upgrade: the number makes it a better game

An accidental bonus. Once conviction is an explicit number, you can show it:

Conviction: ████████░░ 82%

Suddenly the player gets feedback. That sarcastic deflection moved him two points; the argument about ships on the horizon moved him fifteen. You've turned an opaque wall into a game with a progress bar — and it costs nothing, because you were already computing the number for the win check.

Structured state you extracted for correctness turns out to be exactly the state you needed for game feel. That happens a lot.

Next up

Maier believes, deflects, and can be genuinely, honestly beaten. It runs on my machine. Now the last mile, which is its own special kind of pain: shipping a .NET app that has to bring a multi-gigabyte brain with it. Bundling versus first-run download, keeping the installer sane, and what to do when the user's laptop can't handle the model you picked.