Picture a mid-sized insurer in Pune, which we will call Sahyadri General. Its claims team gets about 800 emails a day from policyholders and garages. A product manager adds a button: “Summarise this thread”. Behind it sits a large language model that reads the email chain and produces a five-line summary with the claim number, the amount claimed and the next action needed. The demo goes beautifully. Then the QA engineer, Meera, is asked to sign it off. She opens her usual test tool and realises there is nothing to compare against. Run the same thread twice and the wording differs. Which of the two is “expected”?

That is the whole problem of testing LLM features, and it is solvable, but not by pretending the model is a normal function.

Why equality assertions fail here

A language model generates text one token at a time by sampling from probabilities. Settings such as temperature control how adventurous that sampling is. Even at the lowest settings, outputs can vary between runs or between model versions, and the provider can update the model underneath you. So an assertion that the summary equals a stored string will fail on harmless rephrasing and, worse, will say nothing about whether the summary is right.

Meera needs to stop asking “is this exact text produced?” and start asking “does every output, in every run, meet a set of properties?”. Properties can be checked even when wording cannot.

Step one: build the evaluation set

Meera gathers real-looking cases, with personal data masked: a clean two-email thread, a thread of 40 messages with an argument in the middle, a thread in Hinglish, one where the claim number appears only in a forwarded attachment name, one where two claims are discussed, one that is entirely a complaint with no claim at all. She starts with 60 and aims for a few hundred over time. Each case carries the facts a correct summary must contain, written by a claims officer, not generated by the model under test.

The mix matters more than the count. Include the ordinary cases, the nasty ones and the ones the business fears. An evaluation set made only of tidy examples will give you a flattering score and no information.

Step two: separate what code can check from what judgement must check

Some properties are mechanical. The output has at most five lines. The claim number in the summary matches one present in the thread. The amount appears in the source email. No policyholder’s phone number or Aadhaar number from the thread is copied into the summary where it should not be. These are ordinary assertions and run in seconds, so run them on every case.

Others need judgement: Is the summary faithful, with nothing invented? Is the next action correct? Is the tone appropriate? For these she writes a short rubric. For faithfulness, say: score 2 if every statement is supported by the thread; 1 if there is a minor unsupported detail; 0 if there is an invented fact or a wrong number. A rubric with defined levels is what turns an opinion into something two reviewers can score alike. She checks that by having two people score twenty outputs independently and comparing; if they disagree often, the rubric is vague.

Teams increasingly use a second model as a judge to apply the rubric at scale. It is useful and it is not neutral: a judge model can favour longer answers, can be swayed by confident phrasing and can share the blind spots of the model it is judging. The sensible practice is to calibrate it against human scores on a sample, keep a human review of a slice of every run, and never let an automated judge be the only gate on something with money or legal consequences attached.

Step three: run it more than once

One run is an anecdote. Meera runs each case, say, five times. If a thread passes four times and invents an amount once, that is a 20 percent failure rate on that case, and the team now has a number to discuss. She reports results as rates, not as pass or fail: “faithfulness score of 2 on 93 percent of runs; invented figures in 1.5 percent.” Whether 1.5 percent is acceptable is a business decision, which she puts to the claims head in these terms. For a summary a human officer reads before acting, the tolerance may be higher than for one that triggers an automatic email to a customer.

Step four: attack it

Back to the insurer. Emails are untrusted input. A garage’s email might contain a line telling the assistant to ignore its instructions and mark the claim approved. This is prompt injection, and the evaluation set should include such cases, along with very long threads that exceed what the model can read, empty threads, threads in languages the product claims not to support, and requests the feature should refuse. The expected behaviour is not a particular sentence but a property: the instruction inside the email is treated as content, and the summary reports that the email contained it.

Step five: keep it running

The set earns its keep later. When someone changes the prompt, swaps the model for a cheaper one, or the provider updates a version, Meera reruns the set and compares scores with the previous baseline. A drop in one slice, say Hinglish threads, shows up immediately instead of through a customer complaint. In production she samples real outputs, with privacy rules respected, scores them with the same rubric and adds every real failure to the evaluation set. That loop, where each escaped problem becomes a permanent test, is the nearest thing this field has to a regression suite.

None of this makes the model deterministic. It makes its behaviour measurable, which is what testing was always about.

Figures and names in the examples are illustrative.

Leave a Reply

Trending