Definition
An autoregressive model predicts the next token step by step from the preceding context and generates text sequentially. Knowing this helps explain why generation depends strongly on earlier tokens.
Have you ever watched ChatGPT's response appear character by character and wondered why it doesn't display all at once? That's not a UI effect -- the model is actually generating one token at a time in sequence. An autoregressive model generates text by repeatedly predicting the next token using the sequence of previously generated tokens as input. All of today's major LLMs -- GPT, Claude, Gemini -- use this approach.
A Chain of Next-Token Predictions
The operation of an autoregressive model is straightforward. Given the input "Today's weather is," it first predicts "sunny." Then, using "Today's weather is sunny" as the full input, it predicts "and," and from there predicts the next token. At each step, it calculates a probability distribution over the entire vocabulary and selects the next token from it. This mechanism runs on top of the Transformer architecture. By strictly following the left-to-right order, it achieves natural text generation that follows context.
The Speed Bottleneck
Because tokens are generated sequentially one at a time, generation time increases proportionally with output length. A 1,000-token response requires 1,000 inference steps, making this the biggest bottleneck in LLM response speed. This is an especially severe constraint for applications requiring low latency, such as real-time translation and voice conversation. No matter how much GPU computing power improves, the structural constraint of sequential processing remains.
Acceleration Through Speculative Decoding
A notable solution to this speed problem is speculative decoding. A small, fast draft model generates multiple token candidates in advance, and a larger main model verifies them in batch. Since verification can be parallelized, generation speed improves by 2-3x while maintaining output quality. This technique is being integrated into the inference engines of major models including Google's Gemini and Meta's Llama.
Comparison with Diffusion Models
Diffusion models, the mainstream approach for image generation, gradually restore images from noise and can process all pixels in parallel. For text generation, however, word order determines meaning, giving the autoregressive approach an overwhelming advantage. While recent research has explored applying diffusion models to text generation, they still fall short of autoregressive models in text quality. For the foreseeable future, the autoregressive approach will remain the standard for LLMs.