Definition

The Transformer is a deep learning architecture introduced by Google researchers in the 2017 paper “Attention Is All You Need.” It was first applied to machine translation. Its key innovation was modeling relationships between elements of a sequence entirely through attention, dropping the recurrent neural network (RNN) structure that dominated before.

Today nearly every mainstream large language model is built on the Transformer or a variant of it, and the architecture is also widely used for images, speech and protein structure prediction.

How It Works

Before Transformers, text was mostly handled by RNNs and their improved form, LSTMs. These process text one word at a time, each step waiting on the last, which makes them hard to parallelize. Information from distant words also tends to fade as it passes along.

The Transformer takes a different approach: every position in the sequence can look directly at every other position and weight them by relevance. That captures long-range dependencies and allows the whole sequence to be processed in parallel during training, making full use of GPUs.

A Transformer layer typically contains:

Self-attention. Each token gathers information from other tokens according to how relevant they are, and updates its own representation. In practice this is multi-head attention: several attention computations run side by side, each capturing a different kind of relationship.

Feed-forward network. A nonlinear transformation applied to each position independently, adding expressive power.

Residual connections and layer normalization. These stabilize training in deep networks and help information and gradients flow across many layers.

Positional encoding. Attention on its own ignores order, so position information is injected to tell the model where each token sits.

Stack dozens or more of these layers and you have a full model.

The original Transformer had an encoder and a decoder. Later models settled into three main shapes:

  • Encoder-only, like BERT: strong at understanding tasks such as classification and retrieval.
  • Decoder-only, like the GPT series: strong at generation, and the dominant form for today’s LLMs.
  • Encoder-decoder, like T5: common for input-to-output tasks such as translation and summarization.

Typical Applications

  • Large language models and chat assistants
  • Machine translation and summarization
  • Text classification and semantic search (via embeddings)
  • Image recognition (Vision Transformers) and denoising networks in image generation
  • Speech recognition and synthesis
  • Bioinformatics, such as protein sequence modeling
  • Attention mechanism: the Transformer’s core computation.
  • Large language model: large generative models built on Transformers.
  • Context window: sequence length is limited partly by the cost of attention.
  • Mixture of Experts (MoE): a Transformer variant that replaces the feed-forward network with multiple “experts.”
  • Embedding: the vector representation tokens are mapped to at the Transformer’s input.

Primary source: Attention Is All You Need