00 Foundations

The Transformer: where attention replaced recurrence

By Hiva Mohammadzadeh · Stanford MSCS, UC Berkeley EECS

The 2017 paper Attention Is All You Need replaced recurrent and convolutional sequence models with one built entirely on attention. Instead of reading tokens one after another, self-attention lets every token look at every other token at once. That removed the sequential bottleneck of RNNs, made long-range dependencies cheap to model, and let training run in parallel. It also turned out to be general: BERT, GPT, T5, and LLaMA are the same architecture at heart. This post covers the architecture, the three attention patterns, and the three model families it produced.

The problem with RNNs

Before the Transformer, sequence modeling ran on Recurrent Neural Networks. An RNN processes a sequence one token at a time: the encoder reads the input and a hidden state, produces the next hidden state, and passes it forward; the decoder turns that hidden state into output tokens one by one. RNNs handled machine translation, summarization, and language modeling well enough for years.

Three problems kept getting in the way:

  1. Sequential processing is slow. Each token depends on the hidden state from the previous one, so the work cannot be parallelized. On GPUs built for parallel math, that is a real bottleneck: training on a long sequence means finishing each step before the next can start.

  2. The processing order is fixed. The sequence has to be read left to right, so the model sees "The" before "cat" before "sat" and cannot attend selectively or skip ahead. That rigid order limits how it can reason about the input.

  3. Long-range dependencies are hard. The hidden state is a fixed-size vector that has to carry everything seen so far. By token 500, information about token 1 has been compressed through hundreds of transformations. In practice RNNs struggle to link tokens more than a few dozen apart, and vanishing gradients make it worse: gradients shrink exponentially as they flow back through time, so early tokens get almost no learning signal.

Attention, introduced by Bahdanau et al. in 2015, was the first real fix. It let the decoder look back at different parts of the source sentence at each step instead of relying on one compressed vector. It helped, but the model underneath was still recurrent, so the sequential bottleneck stayed.

The core limitation. An RNN compresses the whole input into a fixed-length hidden state and processes it in order. Bolting attention onto that does not remove the parallelism problem.
RNN: sequential
Token 1h₀
→
h₁wait
→
Token 2h₁
→
h₂wait
→
Token 3h₂
→
h₃O(n) steps
Transformer: parallel
Token 1attend all
Token 2attend all
Token 3attend all
Token nattend all
→
All outputsO(1) steps

An RNN processes tokens in order; each step waits for the one before it. A Transformer processes every token in parallel through self-attention, cutting sequential steps from O(n) to O(1).

The Transformer

In 2017 Vaswani et al. published Attention Is All You Need. It removed recurrence completely, with no RNN cells and no convolutions. The whole model is built from attention, feed-forward layers, and residual connections.

The architecture is an encoder and a decoder, each a stack of identical layers. An encoder layer has two sub-layers: multi-head self-attention and a position-wise feed-forward network. A decoder layer has three: masked multi-head self-attention, multi-head cross-attention over the encoder output, and a feed-forward network. Every sub-layer sits inside a residual connection followed by layer normalization.

The numbers held up on the benchmarks of the day. On WMT 2014 English-to-German it beat every prior model by more than 2.0 BLEU after 3.5 days of training. On English-to-French it matched the best BLEU at a quarter of the training cost. It set state-of-the-art perplexity on the One Billion Word Benchmark, and it did well on English constituency parsing, a task it was not designed for. Those gains, and the parallelism behind them, are what made the architecture spread.

Self-attention

Self-attention is the mechanism at the center of the Transformer. For each token in the sequence, it computes how much that token should attend to every other token, then builds a weighted combination of all token representations from those weights.

self-attention every token looks at every other token, at once The cat sat on mat query

Figure Self-attention lets the query token “cat” weigh every token in the sentence in parallel, with no left-to-right ordering. The weights come from softmax(QKT / √d).

Each token is projected into three vectors: a query (Q), a key (K), and a value (V). The attention score between two tokens is the dot product of the first token's query and the second token's key. The scores are divided by the square root of the key dimension, then passed through a softmax to get weights. The output for a token is the weighted sum of every value vector.

Inputembeddings
→
Q, K, Vprojections
→
QKᵀ / √dscores
→
Softmaxweights
→
weights · Vweighted sum
→
Outputcontext-aware
Attention(Q, K, V) = softmax(QKᵀ / √dk) V
The intuition. A query asks "what am I looking for?" A key answers "what do I contain?" A value carries "what information do I provide?" The dot product of query and key measures relevance, and the weighted sum of values produces a context-aware representation.

This addresses all three RNN problems. There is no sequential dependency, so every attention score can be computed at once. There is no fixed order, so any token attends to any other regardless of distance. And long-range links are cheap: a token at position 500 attends to position 1 at the same cost as to its neighbor, with no hundreds of compression steps in between.

Training uses cross-entropy loss, the same as any sequence-to-sequence model. The difference is speed, because the attention computation runs in parallel.

Multi-head attention

A single attention function computes one set of weights, but different parts of a sentence carry different relationships. The word "it" may need to attend to its antecedent for coreference and to a nearby verb for syntax at the same time.

Multi-head attention runs several attention functions in parallel. Each head has its own learned Q, K, and V projections, so each head can specialize: one on syntax, one on semantic similarity, one on positional proximity. The head outputs are concatenated and passed through a final linear layer.

This is where a Transformer gets much of its representational power. A single head would force every relationship into one attention pattern; multiple heads keep separate, specialized patterns and combine them only at the output.

The encoder-decoder structure

The original Transformer is an encoder-decoder, building on the seq2seq framework of Sutskever et al. (2014). That earlier model used two RNNs, one to encode the input into a fixed-length vector and one to decode it, and the fixed-length bottleneck was its main limit.

The Transformer keeps the shape and replaces the internals with attention. The encoder reads the full input with bidirectional self-attention and produces one representation per token. The decoder generates output tokens one at a time, using masked self-attention so it can only see tokens already produced, and cross-attention so each new token can look at the entire input.

This suits tasks where input and output are different sequences: translation, summarization, question answering. The encoder understands the input; the decoder produces the output conditioned on that understanding.

The three architecture families

The original model used both an encoder and a decoder, but you do not always need both. Three families emerged, each tuned to a different kind of task.

E

Encoder-only

BERT, RoBERTa

Bidirectional self-attention. Every token sees every other token. Produces representations for classification, NER, and QA.

D

Decoder-only

GPT-2, GPT-3, GPT-4

Causal (masked) self-attention. Each token sees only previous tokens. Generates text one token at a time.

ED

Encoder-decoder

T5, BART, original Transformer

Cross-attention bridges encoder and decoder. Best for translation, summarization, and sequence-to-sequence tasks.

Encoder-only (BERT)

An encoder-only model runs the input through the encoder stack and produces representations. There is no decoder and no text generation. The attention is bidirectional: every token sees every other token, before and after it, which is ideal for understanding tasks.

BERT (2018) is the canonical example. It was pre-trained with masked language modeling and next-sentence prediction, and the resulting representations fine-tune well for classification, named entity recognition, and sentiment analysis. Encoder-only models are especially useful when labeled data is scarce: pre-train on a large unlabeled corpus, then fine-tune on a small labeled set. RoBERTa, ELECTRA, and ALBERT followed the same pattern.

Decoder-only (GPT)

A decoder-only model generates an output sequence token by token, with no separate encoder. The attention is unidirectional: each token attends only to tokens before it and itself. This is autoregressive generation, predicting the next token from all previous ones.

GPT is the canonical example. It produces variable-length output one token at a time, which suits text generation, dialogue, and code completion. The key point is that you do not need a separate encoder for generation: the same stack learns to read the input and write the output.

Encoder-decoder (T5, BART)

An encoder-decoder combines both parts. The encoder turns the input into a representation, and the decoder generates output conditioned on it. This is the original Transformer, and it is the natural choice when input and output are structurally different: translation, image captioning, and summarization.

How to choose. Need to understand and classify? Encoder-only. Need to generate? Decoder-only. Need to turn one sequence into a different one? Encoder-decoder. In practice, decoder-only models have become dominant for general use, because they scale well and handle many tasks through prompting.

Attention patterns: forward, causal, and triangle

The way you constrain which tokens can attend to which changes what a model can do. Three patterns matter.

Forward attention

In forward attention, each query attends to all keys and values up to and including its own position. It can use the past and present but not the future. This is the pattern for language modeling, where the goal is to predict the next token from the preceding context.

Causal attention

Causal attention adds an explicit mask that stops each query from attending to positions after it. It is implemented as an upper-triangular mask on the attention scores before softmax, setting future positions to negative infinity so they receive zero weight. This is the standard pattern in decoder-only models like GPT: generating token t depends only on tokens 1 through t−1, which is exactly the autoregressive property text generation needs.

Triangle attention

In triangle attention, each query attends only to a subset of keys and values, forming a triangular pattern defined by a maximum distance between positions. Instead of attending to every previous token, a token at position t attends within a window. The purpose is efficiency: full self-attention is O(n²) in sequence length, which becomes expensive for long sequences, and the triangular pattern lowers that cost while still reaching further-away positions.

What came next

The Transformer did more than improve translation. It set off a new line of models.

In 2018, BERT showed that bidirectional pre-training with a Transformer encoder produced representations that led every NLP benchmark. RoBERTa improved the pre-training recipe, ELECTRA replaced masking with a discriminator, and ALBERT factorized parameters for efficiency.

GPT (2018) and GPT-2 (2019) showed that decoder-only Transformers trained autoregressively on large corpora could generate coherent text. GPT-3 (2020) scaled that to 175 billion parameters and produced few-shot learning across tasks it was never trained on directly. T5 (2020) unified NLP tasks into a text-to-text framework with the full encoder-decoder, and BART paired denoising pre-training with the same structure for generation.

Then came instruction tuning: InstructGPT, ChatGPT, LLaMA, and the models that brought LLMs into everyday use. Every one is a Transformer. Self-attention, multi-head attention, feed-forward layers, residual connections, and layer normalization have stayed the same. What changed is scale, training data, and training method.

The lasting contribution of the 2017 paper is not one technique. It is the demonstration that attention alone, without recurrence or convolution, is enough for state-of-the-art sequence modeling.

Key takeaways

  1. RNNs had three limits: slow sequential processing, a fixed reading order, and weak long-range dependencies from vanishing gradients. The Transformer addresses all three.

  2. Self-attention is the core mechanism. Each token computes scores against every other token with queries, keys, and values, which enables parallel processing and direct long-range links.

  3. Multi-head attention lets the model track several kinds of relationship at once (syntax, semantics, position) by running attention functions in parallel.

  4. Three families emerged: encoder-only (BERT, understanding), decoder-only (GPT, generation), and encoder-decoder (T5, sequence-to-sequence).

  5. Three attention patterns control what each token sees: forward (past and present), causal (masked future, used in generation), and triangle (a window, for efficiency).

  6. Every major model since 2017 (BERT, GPT, T5, LLaMA, and their descendants) is a Transformer.

Based on my CS199 Supervised Independent Study at UC Berkeley and the presentations I created in 2023.