← all posts

Prompt Evaluation And Retry Patterns

November 18, 2025
Prompt Evaluation And Retry Patterns

Most prompt systems look good in demos because demos only measure the median case. Production pain comes from the tails.

A prompt is not ready because it worked once. It is ready when you know how it fails, how you will detect failure, and what the retry path is.

That makes evaluation less like copy review and more like quality engineering.

Define The Failure Modes

Common LLM failure modes include:

  1. Wrong facts.
  2. Missing required fields.
  3. Invalid JSON.
  4. Unsafe output.
  5. Tool misuse.
  6. Overconfident unsupported claims.
  7. Ignoring user constraints.

If you do not name the failure modes, you cannot build a serious gate. You can only say "this felt okay."

Use Deterministic Validators First

Before using another model as a judge, validate what can be validated mechanically:

  1. JSON schema.
  2. Required fields.
  3. Length limits.
  4. Allowed labels.
  5. Forbidden terms.
  6. Link and citation shape.

Deterministic checks should be your first line of defense because they are cheap, fast, and non-flattering.

They also catch the embarrassing failures humans are bad at checking repeatedly, like missing fields, malformed JSON, or forbidden labels.

Add Model-Based Judges Carefully

Model judges are useful for qualities such as helpfulness, relevance, tone, or reasoning quality. They are weak when you ask them to stand in for hard truth they cannot independently verify.

When using a judge prompt, include a rubric. "Is this good?" is not a rubric. "Score factual grounding, constraint compliance, and output usefulness from 1 to 5" is.

Retry With New Information

Retries should not be blind. A useful retry includes the validation error:

Your previous output failed because field `risk_level` was missing.
Return the same object with all required fields.

Blind retries are one of the easiest ways to hide a bad system under higher token spend. A good retry changes the information state.

Keep A Small Golden Set

Maintain examples that represent real edge cases:

  1. Short input.
  2. Long input.
  3. Ambiguous input.
  4. Malicious input.
  5. Missing data.
  6. Domain-specific terminology.

Run prompts against this set before changing templates, judge prompts, or models. If the "improved" version only helps friendly examples, it is not an improvement.

Gate Production Changes

Treat prompt changes like code changes. Review the diff, run evaluation, and record what changed. If an agent uses tools, evaluate tool selection and goal success, not just final text.

The Operating Principle

Evaluation is not there to prove the model is smart. It is there to catch predictable failure before users do.

Good prompt systems assume failure, design recovery deliberately, and treat retries as controlled remediation rather than wishful regeneration.

;