LLaMA: smaller models, more data, three sharp ideas

By Hiva Mohammadzadeh · Stanford MSCS, UC Berkeley EECS

The LLaMA philosophy: open data and efficient inference

LLaMA starts from a claim that runs against the trend of the time. The best-performing language models are not the largest ones; they are smaller models trained on more data, better data, and for longer. This changes the scaling mindset that led the field for years. LLaMA asks a different question. What happens if you spend more compute to train a reasonably sized model, instead of adding parameters?

LLaMA is a collection of decoder-only foundation language models, from 7B to 65B parameters. It works like every autoregressive model. It takes a sequence of words as input and predicts the next word, and it generates text one token at a time. The training method and the architectural refinements are what make it efficient.

Two commitments define LLaMA. First, the authors train the model only on publicly available data, such as CommonCrawl, C4, GitHub, and Wikipedia, and they use no proprietary datasets. This makes LLaMA fully compatible with open-sourcing, and it became the backbone of the open-source LLM ecosystem. Second, the authors want models that are cheaper at inference. A 13B model that matches GPT-3's performance at a fraction of the serving cost is more useful in practice than a 175B model that needs a cluster to run.

Training: pre-training and fine-tuning

LLaMA training uses the standard two-phase approach, and the pre-training phase is where the details matter.

Pre-training is unsupervised. The model learns to predict the next token in a sequence from all past and present tokens. The training data covers the 20 languages with the most speakers, and it focuses on languages that use Latin and Cyrillic alphabets. The scale and diversity of this corpus let smaller models perform above their parameter count.

Fine-tuning adapts the pre-trained model to specific downstream tasks. It adds a task-specific output layer. The expensive pre-training happens once, and fine-tuning is comparatively cheap and fast.

Key innovations

LLaMA does not invent a new architecture. It takes the original transformer and makes targeted modifications that improve training stability, convergence speed, and computational efficiency. Three changes stand out.

LLaMA's three modifications to the standard Transformer architecture.

RMSNorm: pre-normalization for stability

The original transformer applies Layer Normalization after each sub-layer (post-norm). LLaMA switches to pre-normalization and normalizes the input of each sub-layer before the computation, not the output. It also replaces standard Layer Norm with RMSNorm.

RMSNorm extends Layer Norm and drops the re-centering step. It does not compute both mean and variance. It computes only the root mean square of the activations across all feature dimensions, which produces a single scalar per example. That scalar normalizes the activations, and a learnable scale parameter is applied afterward:

RMSNorm is more effective than Layer Norm when the data has high variance, which is common in large-scale language modeling. It gives the model re-scaling invariance and implicit learning-rate adaptation. It needs only one pass through the activations, with no separate mean computation, so it is simpler and faster to compute. At the scale LLaMA operates, that saving matters.

SwiGLU: a gated activation function

LLaMA replaces the standard ReLU (or GeLU) activation in the feed-forward network with SwiGLU, which combines the Swish activation function and the Gated Linear Unit (GLU).

These are its building blocks:

SwiGLU is smoother than ReLU, which gives better performance and faster convergence. The gating mechanism lets it capture complex nonlinear relationships that a simple point-wise activation cannot. As a result, the feed-forward network now has three weight matrices instead of two. This changes the dimension calculations, which the next sections cover.

Rotary Positional Embeddings (RoPE)

LLaMA removes absolute positional embeddings entirely and replaces them with Rotary Positional Embeddings (RoPE) applied at every layer of the network.

RoPE encodes absolute position with a rotation matrix, and it builds explicit relative-position dependency into the self-attention formulation. Instead of adding a position vector to the embeddings once at the input, it rotates the query and key vectors by an angle proportional to their position in the sequence:

The rotation means that the dot product between any two position-encoded vectors depends only on their relative distance, not their absolute positions. This gives the model a sense of distance between tokens, and it avoids the rigid absolute position encodings that limit generalization to longer sequences.

The architecture: a modified transformer

LLaMA is a decoder-only transformer, like GPT-2, with the three modifications above included. The optimizer is AdamW. The authors also apply several efficiency techniques: optimized multi-head attention to reduce memory usage and runtime, activation checkpointing (it saves activations during the forward pass so they do not need to be recomputed during the backward pass), and model and sequence parallelism to reduce memory consumption across devices.

Each decoder block follows one pattern. RMS Norm runs before every sub-layer (masked self-attention and feed-forward), the SwiGLU activation sits inside the feed-forward network, and RoPE applies to the attention queries and keys. Residual connections wrap each sub-layer as usual.

The SwiGLU feed-forward network in detail

The feed-forward network in LLaMA is not a standard two-matrix FFN. SwiGLU needs a gating path, so it uses three weight matrices instead of two.

The base configuration uses sequence length L = 128 and embedding dimension Ed = 4096.

The hidden dimension follows this calculation:

The factor of 2/3 exists because SwiGLU adds a third weight matrix. Standard transformers use a hidden dimension of 4d, where d is the model dimension. The SwiGLU gating mechanism adds roughly 50% more parameters in the FFN, so the hidden dimension is scaled down by 2/3 to keep the total parameter count comparable. The rounding to a multiple of 256 is a hardware optimization, because it aligns matrix dimensions for efficient GPU computation.

The feed-forward block works like this:

1. The input y (shape L × Ed = 128 × 4096) goes through two parallel linear projections, both mapping from Ed to the hidden dimension Fl (4096 → 11008), using column parallelism.
2. One path applies the SwiGLU activation. The other path passes through unchanged.
3. The two paths are element-wise multiplied together, which is the gating operation.
4. The result passes through a final linear projection from Fl back to Ed (11008 → 4096), producing the output (128 × 4096).

Weight matrix dimensions

For the LLaMA-7B configuration with sequence length L = 128, embedding dimension Ed = 4096, 32 attention heads (h = 32), head dimension d = 128, and FFN hidden dimension Fl = 11008:

Input Embedding:

• Shape: (batch_size, 128, 4096)
• Per attention head: (batch_size, 128, 4096/32) = (batch_size, 128, 128, 32)

Self-Attention Weight Matrices (Wq, Wk, Wv, Wo):

• All four: (4096, 4096)

Query, Key, Value Matrices:

• Full: (batch_size, 128, 4096)
• Per head: (batch_size, 128, 128, 32)

Feed-Forward Network (three weight matrices):

The three weight matrices in the FFN, instead of the standard two, come directly from the SwiGLU gating mechanism. The first and third matrices create the two parallel paths, and the second matrix projects back to the model dimension.

Full computation flow

Each decoder block processes the sequence with the following flow (L = 128, Ed = 4096, h = 32, d = 128, Fl = 11008):

1. Input X (L, Ed) passes through RMS Norm.
2. Masked multi-head self-attention with RoPE on queries and keys. A causal mask prevents attending to future tokens. The attention is computed with optimized memory usage (column-parallel projections for Q, K, V).
3. Residual connection adds the attention output back to the input.
4. The result passes through another RMS Norm.
5. The SwiGLU feed-forward network: two column-parallel linear projections (4096 → 11008), SwiGLU gating, then a row-parallel projection (11008 → 4096).
6. Residual connection adds the FFN output back.

Key takeaways

These are the points I take away from the LLaMA architecture:

• Data quality over model size. LLaMA showed that state-of-the-art performance is achievable with publicly available data and smaller models. The 13B model competes with GPT-3 (175B) on most benchmarks, which is a 13x reduction in parameters.
• Pre-normalization with RMSNorm is now the standard for training stability in large language models. It is simpler and faster than post-norm Layer Norm, and it trains more stably. For any new transformer, use pre-norm RMSNorm.
• SwiGLU costs a third weight matrix and gives smoother gradients and better convergence. The 2/3 scaling of the hidden dimension (11008 instead of 16384) keeps the parameter count in check while it accommodates the gating mechanism.
• RoPE improves on absolute positional embeddings. It captures relative position and generalizes better to sequence lengths not seen during training.
• Efficiency is a design priority. Activation checkpointing, optimized attention, and model and sequence parallelism are part of the architecture from the start. LLaMA is designed to be cheap to serve and cheap to train.
• Open data matters. By training only on public datasets, LLaMA enabled the open-source LLM ecosystem that followed, including Alpaca, Vicuna, and hundreds of derivatives. The architectural choices matter, and the open-data commitment is what gave LLaMA its reach.

This post is based on my presentation "Details of the LLaMA Model (Large Language Model Meta AI)" at Berkeley EECS. The original slides, including all architecture diagrams and computation flows, are available in Details_of_the_LLaMA_Model.pdf.