Definition
An attention mechanism is a method that assigns weights to parts of the input to focus on what matters and incorporate the needed information into the output. Understanding it makes model behavior easier to reason about.
When reading lengthy meeting minutes, it's natural to focus your attention on the key points rather than reading every statement equally. LLMs work the same way, learning where to focus within input text. The attention mechanism is a system that assigns importance weights to each part of the input, enabling focused processing on the most relevant information.
How Self-Attention Works
The most important type of attention mechanism is self-attention. Each token in the input sentence computes a relevance score with every other token in the same sentence. Technically, three vectors -- Query, Key, and Value -- are generated from each token, the dot product of Query and Key produces the relevance score, and those weights are used to compute a weighted average of Values. This calculation yields context-dependent semantic representations.
Why It Captures Long-Range Dependencies
In traditional RNNs, capturing relationships between distant words required information to propagate through many steps. The attention mechanism directly computes relevance between all token pairs regardless of distance, so relationships between the beginning and end of a sentence are captured in a single step. In a sentence like "I finally started reading the book I bought yesterday," the ability to accurately grasp the relationship between "book" and "started reading" is thanks to this mechanism.
Multi-Head Attention
In actual Transformers, rather than using a single attention mechanism, multiple "heads" run in parallel. This is called multi-head attention. Each head learns to capture different types of relationships. One head might focus on grammatical relationships (subject and predicate), while another focuses on semantic connections (synonyms, antonyms). By integrating information from multiple perspectives, richer contextual understanding becomes possible.
Applications and Computational Cost
Attention mechanisms are used not only in LLMs but across a wide range of fields including image recognition and speech processing. However, because computation is required between all token pairs, there is a challenge of computational cost that scales quadratically with input length. To mitigate this issue, active research is underway on efficiency improvements such as Flash Attention and Sparse Attention.