05 Architectures

T5: one encoder, one decoder, joined by cross-attention

By Hiva Mohammadzadeh · Stanford MSCS, UC Berkeley EECS

T5 reframes every NLP task — translation, summarization, classification, question answering — as a text-to-text problem. It uses a full encoder-decoder transformer with 12 blocks, pre-trained on C4 with span corruption and fine-tuned with task-specific text prefixes. The encoder uses bidirectional self-attention, like BERT. The decoder uses causal masked self-attention, like GPT-2. Cross-attention is the bridge that lets the decoder attend to the full encoded input, and it is what separates T5 from encoder-only and decoder-only models.

Everything is text-to-text

T5 starts from a single idea. Every NLP task can be cast as taking text as input and producing text as output. Translation takes English in and returns German. Sentiment analysis takes a sentence and returns "positive" or "negative." Grammaticality checking takes a sentence and returns "acceptable" or "not acceptable."

T5 text-to-text framework with example tasks: translation, CoLA, and summarization.
Figure 1 The T5 text-to-text framework, with example tasks such as translation, CoLA, and summarization.

This unification gives T5 its flexibility. Instead of designing a separate output head for each task, a classification head for one and a regression head for another, T5 uses a single architecture and a single training objective across all of them. The task is specified as a text prefix prepended to the input. Feed in "translate English to German: That is good." and the model outputs "Das ist gut." Feed in "cola sentence: The course is jumping well." and it outputs "not acceptable." The model, weights, and decoder stay the same; only the prefix changes.

T5 is a pre-trained deep learning model built on a text-to-text transformer. It is an encoder-decoder architecture. It reads the entire input sequence at once, so it can learn context from all surrounding words, and then generates output autoregressively. The base configuration consists of 12 transformer encoder-decoder blocks, with 128 input tokens, an embedding dimension of 768, 12 attention heads, and a feed-forward filter size of 3072.

Training: span corruption and prefix conditioning

T5 trains in two phases.

Pre-training on C4

T5 is pre-trained on C4 (Colossal Clean Crawled Corpus) with a span corruption objective. Contiguous spans of tokens are selected and replaced with unique sentinel tokens. The model learns to predict those sentinel tokens and the missing text they represent. The task is fill-in-the-blanks: given a sentence with holes punched in it, reconstruct what was removed.

This differs from BERT's masked language modeling. BERT masks individual tokens and predicts them independently. T5 masks contiguous spans, multiple adjacent tokens at once, and generates the missing content sequentially. That trains the decoder to produce coherent multi-token outputs from the start.

Fine-tuning with prefix conditioning

Once pre-trained, T5 is fine-tuned on downstream tasks by converting every task into text-to-text format. Each task is specified with a text prefix prepended to the input. During fine-tuning, the model learns to generate output text for the specified task, conditioned on both the input text and the task prefix. The prefix acts as a soft routing mechanism. It tells the model which transformation to apply, with no architectural changes.

The encoder-decoder architecture

T5 has an input encoder and a task-specific decoder. That pairing distinguishes it from BERT (encoder-only) and GPT-2 (decoder-only). The encoder reads the full input bidirectionally. The decoder generates the output one token at a time, left to right. Cross-attention connects them.

Full T5 architecture: stacked encoder blocks (self-attention, add and norm, feed-forward, add and norm) and decoder blocks (masked self-attention, add and norm, encoder-decoder attention, add and norm, feed-forward, add and norm) with positional encoding, linear, and softmax layers.
Figure 2 The full T5 architecture. Encoder blocks hold self-attention and a feed-forward network; decoder blocks add encoder-decoder attention. Positional encoding, a linear layer, and softmax complete the model.

Each of the 12 encoder blocks has two sub-layers: a multi-head self-attention mechanism and a position-wise fully connected feed-forward network. Each decoder block has three sub-layers: masked self-attention, encoder-decoder cross-attention, and a feed-forward network. Every sub-layer has a residual connection followed by layer normalization.

Encoder-decoder block structure: encoder blocks feed an encoded sequence to decoder blocks with cross-attention, self-attention, and a feed-forward network.
Figure 3 Encoder blocks pass an encoded sequence to the decoder blocks, which use cross-attention, self-attention, and a feed-forward network.

I think of the encoder as the "understanding" half and the decoder as the "generating" half. The encoder builds a rich bidirectional representation of the input. The decoder uses that representation, through cross-attention, to produce the output. Neither half works alone in T5. That is the architectural difference from BERT, which only encodes, and GPT-2, which only decodes.

Encoder self-attention

The encoder uses standard multi-head self-attention with these parameters: embedding dimension Ed = 768, sequence length L = 128, number of heads h = 12, and per-head dimension d = 768/12 = 64.

The input embedding matrix X has shape (L, Ed) = (128, 768). It is projected into queries, keys, and values using weight matrices Wq, Wk, and Wv, each of shape (768, 768). The resulting Q, K, and V matrices have shape (128, 768), which are reshaped to (128, 64, 12) for parallel multi-head computation.

Encoder self-attention flow: X(L,Ed) projected to Q, K, V; then Q times K transpose, scaled by the square root of d, softmax, multiplied by V, heads concatenated, projected through Wo, residual connection, layer norm, output (128, 768).
Figure 4 The encoder self-attention flow, from the input X(L,Ed) through scaled dot-product attention to the output of shape (128, 768).

The attention computation follows the standard formula: Softmax(QK^T / sqrt(d)) * V. Each head computes its own attention scores independently, then the outputs from all 12 heads are concatenated and projected through an output weight matrix Wo of shape (768, 768). A residual connection adds the original input X back to the attention output, followed by layer normalization, producing the final output of shape (128, 768).

Encoder self-attention is fully bidirectional. Every token attends to every other token in the input, with no masking. This matches BERT's attention mechanism, and it lets the encoder build contextually rich representations of the entire input.

Decoder masked self-attention

The decoder's self-attention looks structurally identical to the encoder's, except that it uses an attention mask.

Decoder masked self-attention flow, with an attention mask applied before the softmax.
Figure 5 Decoder masked self-attention. The mask is applied before the softmax.

The mask ensures that when the decoder generates the t-th output token, it can only attend to tokens at positions 1 through t. Future tokens are masked out, set to negative infinity before the softmax, which drives their attention weights to zero. This is autoregressive generation, the same mechanism used in GPT-2.

During training, the query at each position is the current token embedding, and the keys and values come from all tokens up to and including that position. During inference, when generating one token at a time, the query has shape (1, 768) representing the single current token, while the keys and values grow with each step as the generated sequence gets longer.

The weight matrices are the same shape as in the encoder: Wq, Wk, Wv all (768, 768). The feed-forward output for a single generated token has shape (1, 768).

Cross-attention: where the encoder meets the decoder

cross-attention each decoder token reads the whole encoder output enc₁ enc₂ enc₃ enc₄ decoder token encoder outputs

Figure The cross-attention bridge: every decoder position queries all of the encoder’s outputs, which is how T5 conditions what it writes on what it read.

Cross-attention flow
Encoder output(128, 768)
→
WK, WV→ K, V
⤵
Decoder state(1, 768)
→
WQ→ Q
→
Q × KTscores
→
Softmax × V(1, 768)
→
Output+ residual

Q comes from the decoder; K and V come from the encoder. This is how the decoder reads the encoded input.

Cross-attention is the piece that sets T5 apart from both BERT and GPT-2. Also called encoder-decoder attention, it lets the decoder look at the encoder's output while generating each token.

Cross-attention computation: the query comes from the decoder input X(L,Ed) with Wq; the key and value come from the encoder hidden states with Wk and Wv.
Figure 6 Cross-attention. The query comes from the decoder; the key and value come from the encoder hidden states.

The query comes from the decoder: the current decoder hidden state is projected through Wq. The key and value come from the encoder: its output hidden states are projected through Wk and Wv. When the decoder computes attention scores (Q * K^T), it measures how relevant each encoded input token is to the current decoder state. The softmax over these scores produces a weighted combination of the encoder's value vectors, which the decoder then uses to generate the next output token.

Cross-attention. Q comes from the decoder; K and V come from the encoder. This lets the decoder look back at the full encoded input while generating each output token. It bridges understanding (the encoder) and generation (the decoder).

The dimensions are consistent with the rest of the model: Ed = 768, L = 128, h = 12, d = 64. The weight matrices Wq, Wk, Wv are each (768, 768). The attention output passes through Wo (768, 768), gets a residual connection, and then layer normalization, producing the final output of shape (128, 768).

Cross-attention is why T5 can handle tasks where the output is a complex function of the input. In translation, the decoder needs to know what the source sentence said. In summarization, it needs to attend selectively to the most important parts of the input. In question answering, it needs to focus on the passage regions relevant to the question. Without cross-attention, the decoder would generate output blind to the input, and it would be just another language model.

The feed-forward networks

Every block in both the encoder and decoder includes a position-wise feed-forward network (FFN). The structure is the same everywhere: two dense layers with a GELU activation in between.

Feed-forward network flow: y(L,Ed) through Linear(Ed,Fl), GELU, Linear(Fl,Ed), residual connection, layer norm, output (128, 768).
Figure 7 The position-wise feed-forward network, from input through two linear layers with a GELU in between to the output (128, 768).

The first linear layer expands from embedding dimension to filter size: (768, 3072). The GELU activation introduces nonlinearity. The second linear layer projects back down: (3072, 768). A residual connection adds the FFN input back to the output, followed by layer normalization.

This is the same FFN design used in GPT-2 and BERT. It is applied independently to each position in the sequence, which is why it is called "position-wise." The expansion to 3072 (4x the embedding dimension) gives the network extra capacity to learn complex transformations at each position before compressing back to the original dimension.

In the encoder, the FFN output has shape (128, 768). In the decoder, when generating one token at a time during inference, the output is (1, 768).

Weight matrix dimensions

The table below lists the key weight matrix dimensions in T5-base.

Component Matrix Dimensions
Input embedding Embedding (batch_size, 128, 768)
Per-head view Reshaped (batch_size, 128, 64, 12)
Encoder self-attn Wq, Wk, Wv (768, 768) each
Encoder self-attn Q, K, V (batch_size, 128, 768)
Decoder masked self-attn Wq, Wk, Wv (768, 768) each
Decoder self-attn (inference) Q (batch_size, 1, 768)
Decoder self-attn (inference) K, V (batch_size, seq_len, 768)
Cross-attention Wq (from decoder), Wk/Wv (from encoder) (768, 768) each
Feed-forward (all blocks) Dense 1 (768, 3072)
Feed-forward (all blocks) Dense 2 (3072, 768)
Output projection Wo (768, 768)

All attention weight matrices share the same (768, 768) shape. Encoder attention, decoder masked attention, and cross-attention differ in where the inputs come from and whether masking is applied, not in the weight shapes.

Where T5 fits in the evolution

T5 represents the encoder-decoder approach at its best. It helps to see where it sits relative to the other two architecture families I surveyed.

In my independent study, I compared the three architecture types head-to-head:

  1. Encoder-only (BERT): produces fixed-size, bidirectional representations. Best for understanding tasks such as classification, NER, and question answering, where the full input must be comprehended before output.

  2. Decoder-only (GPT-2/3/4): generates variable-length output autoregressively and reads input unidirectionally. Best for generation tasks, or when the input data distribution is open-ended.

  3. Encoder-decoder (T5): combines both. The encoder reads the full input bidirectionally, and the decoder generates output autoregressively with access to the encoder's representations through cross-attention. Best for sequence-to-sequence tasks such as translation and summarization, and any task where input and output have different lengths or meanings.

T5 showed that you do not need separate architectures for different tasks. By framing everything as text-to-text with prefix conditioning, a single encoder-decoder model handles classification, regression, translation, and summarization identically.

PaLM: scaling the decoder path

While T5 proved the power of encoder-decoder models, Google's PaLM (Pathways Language Model) took a different route with a 540-billion parameter, dense decoder-only Transformer. PaLM uses parallel layers to speed up training and multi-query attention to speed up inference. It reached state-of-the-art few-shot performance on hundreds of benchmarks, which showed that at sufficient scale, decoder-only models can match or exceed encoder-decoder performance even on understanding tasks.

This does not diminish T5's contribution. It shows that the encoder-decoder architecture gives strong performance at smaller scales, which matters for practical deployment.

Key takeaways

T5 is a synthesis. It takes the best ideas from encoder-only models (BERT) and decoder-only models (GPT-2) and combines them into a single encoder-decoder architecture.

  • From BERT, it gets bidirectional encoding. The encoder reads the full input with no masking, building rich contextual representations where every token can attend to every other token.
  • From GPT-2, it gets autoregressive decoding. The decoder generates output left to right, using masked self-attention to ensure causal generation.
  • Cross-attention is the glue. It is the mechanism that neither BERT nor GPT-2 has on its own. Cross-attention lets the decoder condition its generation on the full encoded input, which matters for tasks where the output depends on the whole input.

The text-to-text framing makes T5 practical. By converting every task to the same input/output format and using prefix conditioning, a single model and a single training pipeline handle translation, classification, summarization, and question answering. It needs no task-specific heads and no architectural modifications, only different prefixes.

When I work with T5, I find that the three types of attention are the key to the whole model. Encoder self-attention gives you context. Decoder masked self-attention gives you generation. Cross-attention connects the two. The rest, the FFNs, the residual connections, and the layer norms, is shared infrastructure that both stacks reuse.

Based on my CS199 Supervised Independent Study at UC Berkeley and the presentations I created in 2023.