Definition
Automated evaluation is a method that measures output quality mechanically using rules or other models to speed up iteration cycles. It scales well but must be checked against human judgment to avoid drift.
Having humans read hundreds of outputs every time a model is improved is simply not realistic. Evaluation cannot keep pace with the speed of development. Automated evaluation is an assessment method that programmatically scores AI output quality using rule-based or model-based approaches.
Rule-Based Automated Evaluation
The simplest form of automated evaluation uses clear rules for judgment. For example, code generation uses test case pass rates, translation uses BLEU scores (measuring agreement with reference translations), and summarization uses ROUGE scores (measuring overlap with reference summaries). These metrics have been used for years. They offer the advantage of being fast to compute and highly reproducible, but they have limitations for tasks where there is no single correct answer. The quality of natural conversation or creative writing cannot be measured by simple string comparison.
LLM-as-Judge
An approach that has rapidly gained popularity in recent years is "LLM-as-Judge," which uses an LLM itself as the evaluator. High-performance models such as GPT-4 or Claude 3.5 are given evaluation criteria (rubrics) and asked to score the outputs of other models. This method is used in well-known benchmarks such as MT-Bench and AlpacaEval.
The advantage of LLM-as-Judge is that it can make flexible judgments close to human evaluation while being able to evaluate thousands of outputs in minutes. Because rapid feedback can be obtained with every prompt change or model update, the iteration speed of development improves dramatically.
The Need for Calibration
To trust automated evaluation, calibration that verifies correlation with human evaluation is essential. LLM-as-Judge has known biases: it tends to rate longer answers higher (verbosity bias), prefer the answer presented first (position bias), and favor outputs from the same model family (self-bias).
To correct these biases, countermeasures include randomizing the order of evaluation targets, comparing results using multiple evaluation models, and periodically cross-checking with human evaluation to verify score alignment.
Practical Usage
The current best practice is a two-tier approach: using automated evaluation for day-to-day development cycles and supplementing with human evaluation for important release decisions. Automated evaluation excels at quickly detecting obvious quality degradation, while human evaluation excels at accurately judging subtle quality differences. Combining both enables achieving both development speed and evaluation accuracy.