A ten-part series, written from first principles
Deep Learning
from the Inside Out
I take modern language models apart — the transformer, BERT, GPT-2, T5, and LLaMA — down to their weight matrices, tensor shapes, and FLOP counts. Every claim here is one you can check with arithmetic.
Written from the presentations and the CS199 independent study I did at UC Berkeley EECS in 2023, surveying eleven papers from Attention Is All You Need through GPT-4.
Contents
Foundations
fourSelf-attention and multi-head attention from first principles, and the three model families every LLM descends from. Attention Is All You Need.
The nonlinearities that let a network learn curves, from sigmoid and tanh to ReLU, GeLU, and Swish. Derivatives and smoothness.
Why activations drift during training, and how each normalizer fixes it. Batch, Layer, Group, RMS.
The benchmarks that defined the field, task by task, with real examples and metrics. 17 tasks.
Model architectures
fourThe full data path with exact tensor shapes and a FLOP count per block. 12 blocks, 768-dim, 1.86B FLOPs.
The causal mask, the computation diagram, and how its FLOPs compare to BERT. Autoregressive.
Encoder self-attention, decoder masked attention, and the cross-attention bridge. Text-to-text.
RMSNorm, SwiGLU, and rotary embeddings, traced through the forward pass. 7B to 65B.
Techniques
twoSoft prompts, prefix tuning, and CoT, ending with MedPrompt at 90.2% on medical QA. Parameter-efficient.
What changes from millions to hundreds of billions of parameters: few-shot, multimodality, risk. GPT-3, PaLM, GPT-4.