How Do You Actually Test an AI Feature? Evals for Teams Shipping LLM Products

We wrote earlier about the gap between an AI chatbot that demos well and one a business actually trusts. This is the other half of that story, the part nobody demos, because it does not produce a screenshot. If you are putting an LLM in front of real users, the question that decides whether it is safe to ship is not "does it work?" It is "how would we know if it stopped working?" And your existing test suite cannot answer it.

That is not a criticism of your tests. It is a property of the thing you are testing. A traditional test asserts that a known input produces a known output: expect(add(2, 2)).toBe(4). The whole discipline rests on determinism, same input, same output, every time. LLMs break that assumption deliberately. Ask the same model the same question twice and you can get two different answers, both correct, worded differently. Change the temperature, the system prompt, the model version, or the retrieved context and the output shifts again. There is no single string to assert against. expect(response).toBe("...") is not just brittle here; it is asking the wrong question.

What you need instead is an eval: a test whose oracle is a property, not an exact string. This article is a practitioner's walkthrough of how to build one, how to score it, where LLM-as-judge earns its place and where it quietly lies to you, how to wire the whole thing into CI, and how to decide when a number is good enough to ship behind.

Why unit tests fall apart on LLM output

Three things break at once when you point a conventional test at a language model.

The output is non-deterministic. Even at temperature zero you cannot rely on byte-identical responses. Thinking Machines Lab traced the main cause to server-side batching: the batch size a request lands in changes the order of floating-point reductions inside the GPU kernels, so the same prompt can produce different text depending on how requests happen to be grouped, and that is before model updates and provider-side routing move the target again. A snapshot test will go red on a change that is completely correct, and you will learn to ignore it, which is worse than not having it.

Correctness is semantic, not literal. "Your order ships on Tuesday" and "We'll have that out to you on Tuesday" are the same answer. A string comparison says they differ; a customer says they are identical. The thing you actually care about (did it convey the right fact?) lives at the level of meaning, and meaning is exactly what === cannot see.

The failure modes are new. Traditional bugs throw, return the wrong type, or crash. LLM failures are confident and well-formed: a hallucinated policy, a subtly wrong number, a refusal to answer something it should have, a leaked chunk of another user's context, a jailbreak that makes it ignore its instructions. None of these raise an exception. All of them pass a "did we get a 200 and a non-empty string?" check. Your test harness has to look for problems it was never designed to name.

So the goal shifts. You are no longer proving a function is correct. You are measuring the behaviour of a system across many cases and watching that measurement over time, closer to how you monitor a fraud model's precision than to how you test a date formatter.

What an eval actually is

Strip away the tooling and an eval has three parts:

1. A dataset, a collection of inputs, ideally with some notion of what a good response looks like for each.
2. A task, running each input through the exact pipeline you ship: prompt, retrieval, tools, model, post-processing. Not the model in isolation. The system.
3. A scorer, a function that turns each output into a judgement: a pass/fail, a score, or a labelled failure category.

Run all three and you get a number: pass rate, average score, or the percentage of answers grounded in the source. That number is meaningless in absolute terms and invaluable as a trend. Nobody can tell you whether 82% is good. But everybody can tell you that 82% down from 91% after last night's prompt change is a regression, and 82% up from 74% is progress. Evals convert "the AI feels worse today", the single most useless bug report there is, into something you can put on a graph and block a deploy on.

Building the eval set

This is where most teams stall, because they imagine they need thousands of labelled examples before they can start. You do not. A focused set of 30 to 50 cases will catch more real regressions than a vague sense that things seem fine, and you can build it in an afternoon.

Pull cases from four places:

• Production traffic. The single best source. Real questions your users actually ask, in their actual phrasing, messy, ambiguous, and nothing like the tidy examples you would invent. Log your inputs from day one specifically so you can mine them later.
• Failures you have already seen. Every time someone reports "it did the wrong thing here," that is a permanent eval case. Add it, fix it, and it never silently comes back. This is regression testing in its purest form.
• Adversarial and edge cases. Prompt injection ("ignore your instructions and…"), off-topic questions, requests for things the product must refuse, empty input, someone else's data. You are writing down the ways it must not behave.
• The boring golden path. The five questions 80% of users ask. If the system ever gets these wrong, nothing else matters.

For each case, capture what "good" means at whatever fidelity you can afford. Sometimes that is a reference answer. More often it is a set of properties: must mention the refund window, must not invent a policy, must cite a source, must stay under 150 words. Properties scale better than golden answers, because there are many right responses and you rarely want to pin the model to one exact wording.

Keep the set in version control next to the code. It is a test asset, it deserves review in pull requests, and it will grow with the product.

Scoring: three families, in order of preference

Not every eval needs a language model to grade it. In fact the best scorer is usually the dumbest one that works, because dumb scorers are cheap, fast, and deterministic. Reach for them in this order.

1. Programmatic checks. If correctness can be expressed in code, express it in code. Did the output parse as valid JSON against the schema? Does it contain the order number that was in the context? Is it under the length limit? Does it avoid a banned phrase? Did the extracted date match the expected date exactly? These are ordinary assertions, they cost nothing, and they never disagree with themselves. A surprising amount of "AI testing" is really just checking structure and grounding, and you should never pay a model to do what a regex can.

2. Reference-based scoring. When you do have a known-good answer, compare against it, but semantically, not literally. Embedding similarity catches "same meaning, different words." For extraction and classification tasks you can compute real precision and recall against labelled data. These give you a stable number without an LLM in the scoring loop.

3. LLM-as-judge. When correctness is genuinely subjective (is this answer helpful, is it grounded in the source, is the tone right, did it actually resolve the user's problem?) you use another model to grade it. This is powerful and it is the part people reach for first and trust too much. It deserves its own section.

LLM-as-judge, honestly

For anything genuinely subjective (is this answer helpful? is it grounded in the source? is the tone right?) you end up using another model to grade the output. It works, and it is a common approach. But a judge model is not an oracle. It has its own biases, and the first three below were documented and measured in Zheng et al.'s MT-Bench study of LLM judges:

• Verbosity bias: longer answers get rated higher, regardless of quality.
• Position bias: in a head-to-head, the order the answers are shown in swings the verdict.
• Self-enhancement bias: a model tends to favour output that looks like its own.
• Sycophancy: asked "is this good?", a judge drifts toward agreement, a behaviour Sharma et al. found to be general across assistants.

The fix for all of them is the same instinct: give the judge a narrow, checkable question instead of a vague one. Three habits get you most of the way there.

Give it a rubric. "Rate this 1 to 5" invites noise. Spell out what a passing answer must contain, and grade against that.

Ask yes or no, not "how good". "Is this answer grounded in the provided context?" is far more reliable than an abstract quality score.

Calibrate against people. Hand-label 50 cases, run the judge over the same 50, and check how often it agrees. High agreement means you can trust it at scale; low agreement means you are measuring the judge's taste, not your system's quality. This is the step that makes the technique defensible: once its biases were controlled for, Zheng et al. found their strongest judge matched human preferences at over 80% agreement, roughly the rate at which humans agree with each other.

In practice, a grounding judge is just that narrow question wrapped in a prompt:

const GROUNDING_RUBRIC = `
You are grading whether an ANSWER is fully supported by the SOURCE.
Reply with JSON: { "grounded": boolean, "reason": string }.
- grounded=true only if every factual claim in the ANSWER appears in the SOURCE.
- grounded=false if the ANSWER adds any fact, number, or policy not in the SOURCE,
  even if that fact is plausibly true.
Explain your reasoning in "reason" before deciding.
`

async function judgeGrounding(source: string, answer: string) {
  const res = await model.generate({
    system: GROUNDING_RUBRIC,
    input: `SOURCE:\n${source}\n\nANSWER:\n${answer}`,
    responseFormat: 'json',
  })
  return JSON.parse(res.text) as { grounded: boolean; reason: string }
}

Notice what it is not doing: it is not asking "is this a good answer?" It is asking one narrow, checkable question with a rubric attached. Narrow questions are where judges are trustworthy.

Regression testing and CI

An eval you run by hand once a month is a comfort blanket, not a safety net. The value compounds only when it runs automatically, the way your unit tests do, and the shape is a little different.

Run the eval suite on every pull request that touches a prompt, the retrieval pipeline, the model version, or the tools. Those are the four things that change AI behaviour, and all four are invisible to a conventional diff review: a one-word prompt tweak can move your grounded-ness rate five points, and no reviewer can eyeball that.

The pass condition is a threshold, not a green tick. Because scores are noisy, you do not fail a build on a single wrong case; you fail it when the aggregate drops below a bar you set, or regresses meaningfully against the main branch. "Grounded-ness must stay above 90%" and "no more than 2% refusals on the golden set" are the kinds of gates that actually protect you. Split them by severity, too: a guardrail eval (did it leak data, did it get jailbroken, did it give unsafe advice) should hard-fail the build at the first failure, while a quality eval (was the tone warm, was the answer concise) can be a soft threshold you monitor and tune.

Two practicalities bite here. Cost and latency: a 500-case suite with an LLM judge is real API spend and real minutes on every PR, so run a fast core set on each push and the full suite nightly or pre-release, and sample rather than running everything every time. Flakiness: even good evals wobble a point or two run to run, so set thresholds with headroom, and when something fails, re-run before you trust it. Track the number over time on a dashboard, not just as a per-run pass/fail; the trend line is where regressions announce themselves before a user does.

What "good enough to ship" actually means

Here is the part teams find hardest to accept: you will never hit 100%, and chasing it is a category error. There is no eval score at which an LLM feature becomes provably correct, because the space of inputs is infinite and the notion of "correct" is partly subjective. That is not a reason to despair. It is a reason to think like risk management rather than like a compiler.

"Good enough to ship" is three things, none of them a single magic number:

A bar you set deliberately, per risk. A feature that drafts an internal summary a human will read and edit can ship at a much lower quality bar than one that quotes a customer their statutory rights. Decide the bar based on what a wrong answer costs, and write it down before you look at the score, so you are not tempted to move the goalposts to whatever number you happened to get.

A trend that is flat or improving. Absolute score matters less than direction. A feature holding at 88% grounded-ness across the last ten releases is in far better shape than one that hit 94% once and has drifted down unmeasured since.

Guardrails that never regress. The safety-critical evals (data leakage, injection resistance, refusing what must be refused) are not on a sliding scale. Those are pass/fail, and a single new failure blocks the ship regardless of how good everything else looks.

Ship when the risk-appropriate quality bar is met, the trend is healthy, the guardrails are green, and (this is the part that makes it real) you have a monitor in production sampling live traffic through the same evals, so the number keeps updating after launch. An eval suite that only runs before release tells you how the feature behaved on the day you tested it. Production is where the inputs you never imagined actually arrive.

The honest summary

Testing an AI feature is not testing at all in the sense you are used to. It is measurement under uncertainty, and the discipline that makes it work is unglamorous: a small, growing set of real cases; the cheapest scorer that answers each question; an LLM judge used narrowly and kept honest against human labels; thresholds wired into CI on the four things that change behaviour; and a clear-eyed definition of "good enough" that is a risk decision, not a green checkmark.

None of it produces a demo. All of it is the difference between an AI feature you hope is working and one you can prove is not getting worse. That gap is exactly where the trust lives, and trust, not the demo, is what businesses are actually buying.

Sources

• Lianmin Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (NeurIPS 2023). Position, verbosity, and self-enhancement biases, and judge-versus-human agreement.
• Mrinank Sharma et al., "Towards Understanding Sycophancy in Language Models" (Anthropic, ICLR 2024). Sycophancy as a general behaviour of AI assistants.
• Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference" (2025). Batch-size variance in GPU reduction kernels as the primary cause of non-determinism at temperature zero.

The percentage scores used above (an 82% grounded-ness rate, and so on) are illustrative examples, not measured figures.