Definition
Context length refers to the token limit of a model's context window and is directly tied to its long-document handling capability. It also influences latency and cost as prompts grow.
You want the AI to read an entire 100-page PDF, but the content gets cut off partway through. Or the AI completely forgets conditions you specified at the beginning of a long conversation. At the root of these problems is the context length constraint. Context length is the maximum number of tokens an LLM can process in a single operation, equivalent to the model's "working memory" capacity.
The Gap Between Advertised and Effective Values
Model spec sheets list numbers like "128K tokens" or "200K tokens," but taking these numbers at face value is risky. First, most models have separate limits for input and output. For example, Claude 3.5 Sonnet has a 200K context length, but maximum output is limited to 8,192 tokens. GPT-4 Turbo also has 128K input but a standard output of 4,096 tokens. Even more important is the issue of "effective context length." Multiple benchmarks have confirmed that information retrieval accuracy drops significantly near the advertised limit.
The Needle in a Haystack Test and "Lost in the Middle"
A leading method for measuring effective context length is the NIAH (Needle in a Haystack) test. It embeds a specific piece of information (the needle) within a large volume of text (the haystack) and measures whether the model can accurately retrieve it at various positions and depths. A key phenomenon revealed by this test is the "Lost in the Middle" problem: information placed at the beginning and end of the input can be retrieved with high accuracy, but accuracy for information in the middle can drop by 20-30%. A 2023 Stanford University study systematically demonstrated this phenomenon and significantly influenced how long-context models are evaluated.
Practical Strategies for Leveraging Long Context
There are practical tips that account for these characteristics. The cardinal rule is to place the most important information at the beginning or end of the prompt. Rather than inputting an entire long document, extracting only the relevant portions using RAG (Retrieval-Augmented Generation) before inputting them is often superior in both accuracy and cost. Adding clear headings or markers to each section also makes it easier for the model to locate information.
The Cost Trade-off
Models with larger context lengths also incur higher processing costs. Since API pricing is proportional to token count, fully utilizing 128K of context can cost dozens of times more than a short prompt. It is important to determine whether a truly long context is necessary for the use case, and to choose a smaller model when shorter inputs suffice for better cost-performance. Context length is not a case of "bigger is better" -- it is a design decision that should be optimized for the balance of accuracy, cost, and speed.