Definition
A benchmark is a standardized set of tasks and evaluation procedures used to compare multiple models under the same conditions. It helps quantify performance but should be complemented with real-world tests.
Every time a new AI model is announced, claims like "surpasses GPT-4" and "highest accuracy in the industry" fly around, and the evidence behind these claims is benchmarks. A benchmark is an evaluation standard for quantitatively measuring and comparing AI model performance using standardized test sets.
Major Benchmarks
Several types of benchmarks are widely used for evaluating LLMs today. MMLU (Massive Multitask Language Understanding) is a multiple-choice test covering academic knowledge questions across 57 fields, measuring a model's breadth of knowledge. HumanEval presents programming problems to evaluate code correctness, serving as an indicator of code generation ability. MT-Bench evaluates LLM conversational ability through multi-turn dialogue, using a system where one LLM scores another LLM's responses.
Other benchmarks exist for various purposes, including GSM8K for mathematical reasoning, HellaSwag for commonsense reasoning, and RULER for long-text comprehension.
How Benchmarks Work
The basic process is simple. Models are given a pre-prepared set of problems to solve, and accuracy rates or scores are calculated. Multiple-choice tests yield accuracy rates, code generation uses test case pass rates, and dialogue evaluation uses scores from 1 to 10. The core value of benchmarks lies in enabling fair comparison between different models by using the same problem set.
Limitations and the Gaming Problem
However, benchmarks have serious limitations. First, test scores can diverge from actual user experience. It's not uncommon for a model that scores highly on benchmarks to fall short of expectations when creating real business documents or handling complex consultations.
An even more serious issue is gaming (strategic optimization). Since benchmark problems are publicly available, it's possible to achieve inflated scores by including similar problems in training data or making adjustments specifically tailored to benchmark formats. This is similar to only studying past exam papers -- it differs from genuine capability improvement.
How to Use Benchmarks Wisely
Benchmarks are not perfect, but they are undeniably valuable reference information for model selection. The key is to never judge based on a single benchmark score alone. Checking multiple benchmarks cross-sectionally and then testing on your actual use cases is the most reliable approach to model selection. Recently, benchmarks using non-public test sets as a contamination countermeasure and systems that dynamically generate problems have emerged, showing that evaluation methods themselves continue to evolve.