Turn eval results into the right fixes

Evaluation only helps when it changes the right part of the system. Each gap from the previous step, whether product, prompt, data, or eval, points to a different fix. Match the failure to the fix so each eval cycle changes what the output actually does, not just how it scores.

Match each failure to the right fix

If you see this	Do this
Output introduces unsupported details	Fix requirement clarity.
Output behaves inconsistently across prompts	Fix the prompt set.
Output continues without required information	Fix data and context.
Reviewers disagree on results	Fix evaluation checks.

Focus on the root cause so each eval cycle improves real behavior.

Fixes by area

Change one area, rerun the same evals, then move to the next. That is the only way to know which change fixed the failure.

Fix requirement clarity

Tighten what the output must do.

Clarify what is allowed and not allowed.
Make failure conditions explicit.
Specify the exact evidence needed for decision, commitment, and required-input checks.

Focus on requirements when failures show unclear rules, reviewers interpret the same output differently, or outputs pass in one scenario and fail in another because the requirement is vague.

Fix the prompt set

Change how behavior is tested.

Add prompts that reflect real usage.
Include edge cases that expose failures.
Remove prompts that do not represent real scenarios.

Focus on prompts when failures appear only in specific phrasing, behavior breaks under ambiguity, or the prompt set is too narrow to reveal recurring patterns.

Fix data and context

Change what the output can access.

Ensure required inputs are available.
Clarify which sources are in scope.
Define what the output should do when required information is missing.
Resolve conflicting, stale, or missing data.

Focus on data and context when output appears confident but is wrong, failures come from missing or inconsistent information, or correct behavior is impossible with current inputs.

Fix evaluation checks

Change how results are measured.

Make checks observable and repeatable.
Remove subjective criteria.
Align scoring rules and thresholds.

Focus on evaluation checks when reviewers disagree, results are inconsistent across reviewers or runs, or checks are hard to apply the same way every time.

Confirm the fix worked

Apply the priority order from the previous step: reliability failures and repeated patterns first, isolated edge cases later. Then verify that the fix changed behavior.

Rerun the same prompts.
Check whether the failure still appears.
Verify that behavior holds across similar inputs.

If results do not change, the fix did not address the real problem.

Avoid cosmetic fixes

Changes that improve how the output sounds do not fix behavior. Avoid treating these as improvements:

rewriting for tone without fixing errors
adding detail that is not grounded in the source
making output longer without improving accuracy

If the behavior did not change, the problem is not solved.

Key takeaway

Fix the cause, not the symptom. If the same failure shows up again, the change did not fix behavior.