LLM-as-Judge: Scalable AI Evaluation with Language Models › LLM-as-Judge in Production Systems

Moving from a research evaluation pipeline to a production system introduces additional engineering constraints that reshape how you design your judge. Latency, cost, consistency over time, and the ability to detect when the judge is failing all become first-class concerns.

Production systems often need to evaluate thousands or millions of model outputs, either offline (batch evaluation of a test set) or online (real-time quality monitoring of user-facing responses). These two modes have different latency requirements but share the same accuracy concerns.

For offline batch evaluation, the primary constraint is throughput and cost. A frontier model judge evaluating one million responses at 2,000 tokens per evaluation (prompt plus response plus judge output) costs roughly $10 per thousand API calls at current pricing, meaning a million evaluations costs around $10,000. At this scale, caching is needed: if you evaluate the same response multiple times (for example, re-running calibration after updating the rubric), cache the judge outputs rather than re-calling the API. Similarly, you can use cheaper models for initial triage (flagging obviously low-quality responses) and reserve the frontier judge for borderline cases and a random sample for calibration.

For online evaluation, the constraint is latency. Calling a frontier judge API in the necessary path of a user-facing request adds 500ms to 2000ms of latency, which is unacceptable for interactive applications. Two approaches are common: asynchronous evaluation (the judge runs after the response is returned to the user, with results logged for monitoring) and lightweight local judges (a smaller model that can run on the same infrastructure as the main model with negligible latency). For monitoring applications where you do not need real-time scores, asynchronous evaluation is usually the right choice.

Judge model updates are one of the most underappreciated challenges in production LLM evaluation. When the judge model changes, historically collected scores become incomparable to new scores, because you cannot tell whether a quality trend is a real change in the evaluated model's output or a change in the judge's behavior.

The mitigation is to maintain a frozen calibration set: a sample of responses with human-annotated quality scores that you never use for training. Whenever the judge model changes, re-evaluate the calibration set with the new judge and compare the new judge-human correlation to the previous judge's correlation on the same set. If the new judge achieves higher correlation, the upgrade has improved evaluation quality. If correlation drops or the score distribution shifts substantially, the new judge may be introducing new biases that require rubric updates or recalibration.

Running parallel evaluations (both old and new judge) for a period after a judge model update allows you to compute a normalization factor that makes scores from different judge versions comparable. This is analogous to equating in standardized testing, where parallel forms of a test are calibrated to a common scale.

Rather than relying on a single judge model, some production systems use ensembles of judges and aggregate their scores. This approach reduces dependence on any single model's biases and produces more reliable estimates when judge models disagree.

The simplest ensemble strategy is to average scores from multiple judge models. If judge A gives a response 4 and judge B gives it 3, the ensemble score is 3.5. This reduces the variance from any single judge's idiosyncrasies, including position bias, verbosity bias, and model-specific scoring conventions.

A more principled approach is to treat judge agreement as a confidence signal. When all judges in the ensemble agree, the score is high-confidence. When judges disagree substantially, the response is in a disputed quality region that benefits from human review. Multi-judge disagreement is often more informative than any individual judge's score: it identifies the examples where the evaluation task is most ambiguous and human judgment is most valuable.