Definition

The attention mechanism is a way for a neural network, while processing one element, to score how relevant every other element is and then take a weighted sum of their information. Highly relevant elements contribute more, less relevant ones contribute less, much like a reader focusing on the key words in a sentence.

Attention was first used in neural machine translation to improve encoder-decoder models, letting the model “look back” at the most relevant parts of the source sentence as it translated each word. The Transformer architecture later made it the model’s central computation.

How It Works

The most common form today is scaled dot-product attention. Each token’s representation is transformed into three vectors:

  • Query (Q): “what information am I looking for?”
  • Key (K): “what kind of information can I offer?”
  • Value (V): “the information I actually carry.”

The computation has three steps:

  1. Take the dot product of the current token’s query with every token’s key to get relevance scores.
  2. Divide the scores by the square root of the vector dimension (to keep values from growing too large), then apply softmax to turn them into weights that sum to 1.
  3. Use those weights to take a weighted sum of all value vectors. The result is the token’s new representation.

For example, in “The kitten didn’t jump onto the table because it was too tired,” attention for “it” leans heavily toward “kitten,” which is how the model resolves the reference.

Common variants:

Self-attention. Q, K and V all come from the same sequence, letting elements within it relate to each other.

Cross-attention. Q comes from one sequence, K and V from another, such as the target language attending to the source language in translation, or an image attending to a text prompt in image generation.

Multi-head attention. Several independent attention computations run in parallel, each picking up a different kind of relationship (syntax, reference, meaning), and their outputs are concatenated.

Causal attention. In generative models, each position can only see what came before it, so the model can’t peek at the future while generating token by token.

Standard attention compares every pair of positions, so compute and memory grow roughly with the square of sequence length. That’s a major reason long contexts are expensive. Researchers have proposed sparse attention, linear attention and memory-efficient implementations such as FlashAttention to address it.

Typical Applications

  • Understanding context, resolving references and tracking long-range dependencies in LLMs
  • Aligning source and target languages in machine translation
  • Matching images to text prompts in image generation (cross-attention)
  • Focusing on key regions in vision models
  • Aligning audio segments with text in speech recognition
  • Transformer: the architecture built around attention.
  • Context window: attention’s cost is one factor limiting context length.
  • Embedding: attention operates on vector representations of tokens.
  • Multimodal models: often use cross-attention to fuse information across modalities.

Papers: Neural Machine Translation by Jointly Learning to Align and Translate, Attention Is All You Need