Understand eval results
Two outputs can both fail, but one may be unsafe to act on while the other is simply less helpful. Read eval results in terms of what could go wrong if someone acts on the output, not only whether each check passed or failed.
Separate unsafe output from lower-quality output
| Type of result | What it means | What to do |
|---|---|---|
| Reliability failure | Output introduces details not grounded in the prompt, task, or source, such as a decision date that never appeared in the source. | Fix before doing anything else. |
| Quality issue | Output is correct but less clear or useful, such as listing risks without explaining why they matter or what action to take. | Improve after reliability passes. |
Identify what failed and why
Do not just record that a result failed. Record the failure pattern.
| Failure pattern | What it looks like | Why it matters |
|---|---|---|
| Invented details | Adds owners, dates, or decisions not in source data. | Leads to incorrect actions based on information the source does not support. |
| Incorrect decision labeling | Treats discussion as a decision. | Creates false certainty. |
| Missing information ignored | Continues instead of asking. | Hides gaps that must be resolved before someone can act. |
| Scope drift | Adds information outside the request or data boundary. | Changes the requested task or conclusion. |
Look for patterns across prompts
One failure does not tell you much. Repeated failure points to a gap in the product, prompt, data, or eval.
Check whether:
- the same mistake shows up in multiple outputs, such as invented dates in five of ten prompts
- failures happen in specific situations, such as missing key information or conflicting data
- similar prompts produce different results, such as one response refusing missing data while another guesses
When the same pattern fails across several prompts, treat it as a gap to fix at the source, not a one-off mistake.
Separate product issues from evaluation issues
Do not assume every failure means the model needs to change. The cause can be unclear requirements, weak prompts, missing data, or inconsistent assertions.
| Issue type | What it looks like | What to check |
|---|---|---|
| Product gap | Output lacks required behavior across prompts. | Is the requirement clear and complete? |
| Prompt gap | Failures appear only in certain wording or edge cases. | Does the prompt set reflect real usage? |
| Data gap | Output sounds confident but is wrong. | Is required data available and in scope? |
| Eval gap | Reviewers disagree on results. | Are assertions observable and enforceable? |
Fix the right problem. Do not change model behavior when the eval itself is unclear.
Decide what to fix first
Fix failures that could lead to incorrect actions before improving usefulness or presentation quality.
Prioritize:
- reliability failures that could trigger incorrect action
- failures that repeat across prompts
- failures in high-impact scenarios
Defer isolated edge cases and low-impact quality issues.
Use results to guide next steps
| If you see this | Do this |
|---|---|
| Repeated reliability failure | Make the requirement explicit so the system cannot guess. |
| Failures tied to missing information | Improve stop-and-ask behavior. |
| Inconsistent results across similar prompts | Expand or refine the prompt set. |
| Reviewer disagreement | Rewrite assertions to be more precise. |
If results do not change what you do next, the eval did not help you decide anything.
Evidence to use when reviewing results
Use:
- the outputs from this eval and their scores
- reviewer notes explaining why each output passed or failed
- the prompts used to generate the outputs
- source data and constraints given to the system
Do not use:
- assumptions about what the output probably meant
- knowledge from past evals or incidents not shown here
- information that cannot be traced back to the inputs, outputs, or review notes
Reviewer checklist
- Classify each failure as a reliability failure or quality issue and cite evidence.
- Prioritize reliability failures before quality issues.
- Record at least one failure pattern tied to the prompt, task, or source it broke against.
- Treat repeated failures across prompts as a pattern to fix.
- Trace root cause to product, prompt, data, or eval.
- Tie next steps to the observed failure type.
Key takeaway
Fix failures that could lead to incorrect actions before improving clarity, tone, or polish.