When to use
Use a judge when you want consistent, scalable evaluation of:- Hallucinations, safety/policy violations
- Response quality (helpfulness, tone, structure)
- Latency, cost, and error patterns tied to specific criteria
Judges
Introduction
Continuously evaluate your production traffic with judges that learn over time
Calibrated LLM judges are AI evaluators that watch your traces, sessions, or spans and score outputs according to criteria you define. They get better over time the more you refine and correct their evaluations.