If you can’t measure it, don’t ship it: evaluating LLM systems
A practical framework for testing generative AI applications before and after launch: golden datasets, graded rubrics and the metrics that matter.
ATAiko TanakaLead ML Engineer
Every team building with large language models eventually hits the same wall. The prototype looks impressive in a demo, but nobody can say with confidence whether this week’s version is better or worse than last week’s. Prompts change, models are upgraded, retrieval settings are tuned, and quality drifts in ways that only show up when a customer complains.
The fix is not glamorous, but it is reliable: treat evaluation as a first-class engineering artifact, built alongside the application from day one.
Start with a golden dataset
A golden dataset is a curated set of realistic inputs paired with what a good answer looks like. It does not need to be large to be useful. Fifty to two hundred carefully chosen examples will catch most regressions.
The important word is realistic. Pull examples from actual user logs, support tickets or subject-matter experts, not from what the development team imagines users will ask. Include the awkward cases deliberately:
- Questions the system should decline to answer
- Questions whose answer is not in the knowledge base
- Ambiguous requests that need a clarifying question
- Inputs that attempt to override the system’s instructions
Grade against a rubric, not a vibe
“Does this look right?” is not a metric. Define a short rubric with specific, observable criteria. For a knowledge assistant we typically score:
- Correctness: is the answer factually right given the source material?
- Groundedness: is every claim supported by a retrieved document?
- Completeness: does it address the whole question?
- Appropriate refusal: does it say “I don’t know” when it should?
- Tone and format: does it match the brand and the channel?
Each criterion gets a simple scale, and each example gets scored. Human graders establish the baseline; a carefully validated model-based grader can then score most changes automatically, with humans auditing a sample.
Run it on every change
Evaluation only prevents regressions if it runs before changes reach users. We wire the evaluation suite into continuous integration so that any change to prompts, models, retrieval parameters or tools produces a scorecard. A drop below agreed thresholds blocks the release, exactly as a failing unit test would.
The goal is not a perfect score. The goal is knowing, with evidence, whether you just made things better or worse.
Keep measuring after launch
Offline evaluation is necessary but not sufficient. In production we track:
- User feedback rates: thumbs up and down, and the comments attached
- Escalation rates: how often a conversation is handed to a person
- Groundedness sampling: a weekly sample of live answers graded against the rubric
- Cost and latency: per request, per user and per feature
New failure modes discovered in production go straight back into the golden dataset, so the system steadily gets harder to break.
Where to begin
If you have an LLM application in pilot today, start small: collect fifty real questions, write down what good answers look like, and score your current system honestly. That single afternoon of work usually reveals more about readiness for launch than weeks of informal testing.
If you would like help designing an evaluation framework for your own system, get in touch.