Evaluating output quality
Defining what makes AI output “good” and testing against that standard is core design work. Just as you would evaluate whether a UI pattern serves someone’s needs, evaluating AI output means asking whether the model’s response is useful, trustworthy, and appropriate in context. That judgment requires UX expertise. Without it, AI systems are optimized for what’s easiest to measure, not what’s most valuable to people.
What is an eval?
At its simplest, an eval is a test. You give an AI system an input, then apply defined criteria to judge whether the output meets expectations for a specific job and scenario.
Evals are often used to:
- Clarify what “good” AI output means in a given context
- Compare approaches or iterations
- Detect changes in quality as systems evolve
You might not run evals directly. Many UX practitioners contribute by helping to define scenarios, quality criteria, examples, or failure patterns that others later formalize or automate.
Why evals matter for UX
An AI output can be technically correct and still be a poor user experience. It may be confusing, low trust, misaligned with intent, or risky to act on.
UX input is often valuable in evals because UX focuses on whether behavior is:
- Useful in a real context
- Understandable without extra interpretation
- Trustworthy enough to act on
Without a human-centered lens, systems tend to optimize for what is easiest to measure rather than what matters most to people. Evals make human judgment more explicit and more repeatable.
Core components of an eval
Most evals include some version of the same core components.
Prompt set: A collection of prompts representing user goals or intents, written in natural language that reflects how people actually describe those goals.
Context or data set: Grounding material the system may reference, such as files, emails, chats, or records.
Rubric: The criteria used to judge output quality or behavior.
Golden set: A small collection of examples where the input and the expected correct answer (“ground truth”) have been reviewed and agreed on. It’s used to compare the consistency of outputs.
These elements can be simple or more structured depending on context. Or, start with one or two elements and add the rest over time.
Key decisions in human-centered evals
Consider these seven decision points to get from user intent to measurable quality.
1. Define the audience and scenario
The eval process often begins by clarifying who the output is for and what the user is trying to accomplish. This helps keep evals grounded in real customer scenarios.
Consider these questions:
- Who is the system working for?
- What purpose does the output serve?
- Which tasks do users need the AI system to do?
- What does success look like from the customer’s perspective?
Example
AI system: Meeting Prep Agent
User: Information worker at Contoso.
Agent description: Assists users in preparing for upcoming meetings by quickly clarifying the context, stakeholders, expectations, materials, and desired outcomes.
Scenario: “Help me understand the upcoming meeting and those who will attend.”
Outcomes: The user can:
- Quickly understand what the meeting is about
- Who is attending and what they care about
- Understand how this meeting connects to their broader goals.
2. Define the data boundary
Clarify which information the AI system is allowed to access and rely on, and what it should ignore. These decisions are typically defined by humans (for example, through retrieval setup, system design, or MCP configuration), and help reduce confident but unsupported outputs.
Considerations may include:
- Which sources are in scope
- Which sources are out of scope
- What up-to-date means (time window)
- Which source is authoritative when sources conflict
Example
The Meeting Prep Agent may use the last 30 days of meeting notes, the current calendar invite, recent messages, and the attendee list. It should ignore unrelated project docs and content the user can’t access.
3. Build a prompt set
Prompt sets reflect how people actually ask for help, using real user language, not internal feature terms. Also include ambiguity to test clarifying questions.
Prompt sets may include:
-
Common, straightforward prompts:
- “Summarize where we are for tomorrow’s meeting.”
- “What decisions do we need to make?”
- Short prompts with missing context: “Fastest prep.”
- Ambiguous or edge case requests: “Summarize where we are. Only include confirmed decisions and cite where each decision came from.”
4. Define “good” behavior
Define what “good” output looks like in this scenario, using examples and criteria that describe what information is in those examples and how it should be presented. Then, use the learnings to guide decisions about model behavior, prompt design, and the end-to-end UX.
Example
For the Meeting Prep Agent, criteria for “good” output might mean:
- Accurate summary with source links
- Clear agenda and decisions listed only when they’re backed by notes
- Action items only when an owner is named, and clear callouts when something is unknown (no guessing).
It must not imply certainty when context is missing. If required information is unavailable, the response must not present guesses as facts.
5. Turn quality criteria into assertions
Translate “good” into clear, testable statements that can be checked consistently across outputs. Some assertions are designed for automated (LLM) evaluation, while others require human judgment. Most evals use both, so it’s important to be clear which type you’re writing and how they work together.
Example
Meeting Prep Agent assertions:
- Agenda with timeboxes (0/1)
- Top 3 risks, each grounded in the provided context (0-2)
- No invented facts (must pass)
6. Test your eval
Run the prompt set and score each output against your assertions using a mix of automated checks and human review. Automated checks verify specific, observable requirements, while human review determines whether the output is actually useful and ready to act on. Record scores and failure reasons.
Example
For each prompt in your prompt set, run an automated check that every “decision” bullet links back to a real note or email. Then have a human reviewer answer: “If I had 10 minutes before the meeting, would this prep actually help me?“
7. Refresh over time
Evals aren’t “set it and forget it.” As the product changes and people start using it in new ways, your evals can get out of date. If they don’t evolve, they stop reflecting real usage and quality can drift without being caught.
Over time, revisit:
- Prompts: add the kinds of requests you’re seeing in real usage
- Examples: update them when your preferred format or workflow changes
- Criteria: adjust when expectations or risks change (for example, stricter rules for high-stakes scenarios)
Example (meeting prep):
After a few weeks, you notice people often ask, “What do I need to do before this meeting?” Add that prompt to the set, update your “good” example to include a short “Your prep” section, and add an automated check that action items are only listed when they’re explicitly stated in the notes.