Scaling: from GPT-3 to GPT-4

By Hiva Mohammadzadeh · Stanford MSCS, UC Berkeley EECS

The scaling hypothesis

One pattern has held in deep learning for several years. Make the model bigger, give it more data, and new capabilities appear. The models gain behaviors that smaller models could not do at all, and the change is qualitative.

This is the scaling hypothesis, and the models in this post are its strongest evidence. GPT-2 had 1.5 billion parameters and could generate coherent paragraphs. GPT-3 scaled to 175 billion and could translate languages, answer trivia, and write code from a few prompt examples. PaLM reached 540 billion and solved multi-step reasoning problems. GPT-4 added vision and matched human professionals on standardized exams.

At each scale threshold, qualitatively new capabilities emerge that were absent in smaller models.

GPT-3: few-shot learning

GPT-3 gave the scaling hypothesis its clearest evidence. At 175 billion parameters, 10x more than any previous non-sparse language model, it showed that you could get useful task performance without any task-specific training data.

The mechanism is few-shot learning. Instead of fine-tuning on thousands of labeled examples, you provide a handful of input-output demonstrations directly in the prompt. The model generalizes from these examples and produces correct outputs on new inputs. GPT-3 did this across language translation, question answering, arithmetic, and basic programming tasks.

GPT-3 was pre-trained on a large corpus of books, articles, and web pages to predict the next word in a text sequence. The training objective matches GPT-2, and the difference is scale. That scale gave three modes of operation:

• Zero-shot: Give the model a task description and an input, with no examples. GPT-3 generates useful output on tasks it was never explicitly trained on.
• Few-shot: Provide a few input-output examples in the prompt, with no gradient updates. The model picks up the pattern and applies it to new inputs.
• Fine-tuning: Traditional adaptation with task-specific training data and gradient updates to the model's parameters.

What stood out was how far zero-shot and few-shot could go. GPT-3 reached state-of-the-art on SuperGLUE, ahead of previous models by a wide margin. It produced open-ended generation, coherent and contextually appropriate text that was not tied to a specific task format.

This changed the economics of NLP. Instead of collecting labeled data and training a model for every task, you could use a single large model and steer it with prompts. The previous post on prompt tuning covers the techniques that grew from this.

PaLM: 540 billion parameters and the Pathways system

GPT-3 showed scale matters; PaLM pushed it further. PaLM (Pathways Language Model) is a 540-billion-parameter, dense decoder-only Transformer that extended what a single language model can do.

PaLM stands out for how it was trained and the infrastructure that made it possible.

The Pathways system

PaLM was trained with the Pathways system, which trains a single model efficiently across multiple TPU v4 Pods. This was the first large-scale use of Pathways. It scaled to 6,144 TPU chips, the largest TPU-based system configuration at the time. The system uses a hierarchical design with data parallelism at the Pod level across two Cloud TPU v4 Pods, combined with standard data and model parallelism within each Pod.

PaLM is an infrastructure result as much as a model result. Training a 540-billion-parameter model needs a system that can coordinate thousands of accelerators, handle communication bottlenecks, and recover from hardware failures. Pathways did this at very large scale.

Architectural choices

PaLM uses parallel layers to speed up training; it computes the attention and feed-forward components at the same time rather than one after the other. It uses multi-query attention to speed up inference. It also introduces the MLM-Mix pre-training objective, which jointly pre-trains autoencoding and autoregressive language modeling on a large unlabeled corpus and improves out-of-domain performance.

Results

PaLM 540B improved on tasks that previous models handled poorly:

• Multi-step reasoning: PaLM outperformed fine-tuned state-of-the-art models on reasoning tasks, using few-shot prompting alone.
• BIG-bench: PaLM outperformed average human performance on BIG-bench, a diverse suite of over 200 tasks designed to probe model capabilities.
• Efficiency: although it is 3x larger than GPT-3, PaLM reached these results more efficiently, thanks to Pathways and the architectural optimizations.

The reasoning result matters most. Many people considered reasoning beyond the reach of next-token prediction. PaLM showed that scale, with the right training recipe, could produce compositional reasoning behavior.

GPT-4: multimodal and human-level

GPT-4 is the next step in the trajectory. It is a large-scale, multimodal model that accepts image and text inputs and produces text outputs. The multimodal input expands what a language model can do.

GPT-4 is a Transformer model pre-trained to predict the next token, on publicly available data and data licensed from third parties. Its training objective matches GPT-2 and GPT-3, but its capabilities are much wider.

What changed

Multimodal input. GPT-4 accepts images alongside text and reasons about their content, beyond simple captioning. In one published example, it identifies that chicken nuggets are arranged to look like a world map and explains why that is funny. That requires reading the image content, recognizing the resemblance, and connecting it to the joke in the accompanying text.

Human-level professional performance. GPT-4 passes a simulated bar exam around the median of human test-takers. It reaches human-level performance across a range of professional and academic benchmarks, and the results hold across them.

Longer context. GPT-4 handles up to 25,000 words of input, which covers entire documents, long conversations, and multi-part instructions.

Better programming. GPT-4 handles programming instructions better than GPT-3, including more complex code generation, debugging, and explanation.

Steerability. Users can instruct GPT-4 to adopt specific response styles, personas, or constraints, which makes it more useful as a general-purpose tool.

Safety alignment. Post-training alignment improves factuality and adherence to instructions. The model was trained to limit harmful responses as a deliberate engineering effort, though it is not perfect.

Foundation models: opportunities

The term "foundation model" names what these large-scale models have become: a base layer that whole applications and industries build on. The opportunities span several domains.

Healthcare. Foundation models can process medical literature, assist with diagnosis, and support clinical decision-making. As I covered in the prompt tuning post, GPT-4 with MedPrompt reached 90.2% on medical exam questions, ahead of fine-tuned specialist models. The potential for research, drug discovery, and clinical documentation is large.

Education. Personalized tutoring, automated grading, and adaptive learning systems become more feasible at this model level. A model that can explain calculus, grade essays, and adapt to individual students could widen access to good education.

Finance. Sentiment analysis, report generation, risk assessment, and fraud detection all benefit from models that reason over large volumes of text. Financial applications need high reliability, which pushes the field toward better calibration and uncertainty estimation.

Teams already deploy these use cases.

Foundation models: risks

Foundation models present large opportunities alongside concrete risks.

The same capabilities that make foundation models useful make them dangerous if deployed without care. The risks are concrete and well-documented.

Bias

Foundation models learn from their training data. That data contains biases, because it reflects the internet and published text, and the model reproduces and sometimes amplifies them. Models have produced biased outputs across gender, race, religion, and other categories. In high-stakes domains such as healthcare and criminal justice, biased outputs cause real harm.

Concentration of power

Training a 540-billion-parameter model requires infrastructure that only a few organizations have. This concentrates the ability to build and control foundation models in a small number of large tech companies. Smaller organizations, academic researchers, and developers in less-resourced countries depend more and more on APIs that these companies control.

Employment impact

Foundation models can automate tasks that once required human expertise, such as writing, translation, coding, and analysis. The productivity gains come with worker displacement. The economic impact will not be evenly distributed.

Accountability gaps

When a foundation model produces harmful output, who is responsible? The model developer? The application builder? The end user? Current legal and regulatory frameworks do not have clear answers, and the technology is moving faster than policy.

What the scaling trajectory shows

The four models line up in a clear progression.

Three patterns stand out.

First, emergent capabilities appear at scale thresholds. GPT-2 did not do few-shot learning; it emerged at GPT-3 scale. Multi-step reasoning was weak in GPT-3 and became strong in PaLM. Multimodal understanding did not exist in text-only models, and GPT-4 added it. Each scale jump produces capabilities that the previous scale did not have.

Second, the gap between few-shot and fine-tuned performance is closing. GPT-3 showed few-shot could compete with fine-tuning on some tasks. PaLM showed it could beat fine-tuned models on reasoning. The distinction between "general-purpose" and "task-specific" models is dissolving.

Third, infrastructure is becoming the bottleneck. PaLM required 6,144 TPU chips. Training runs at this scale cost millions of dollars and take weeks. The limiting factor is no longer algorithmic; it is who can afford the compute. That shapes who can take part in frontier research.

The open question is whether this trajectory continues. The optimistic view is that more scaling will produce more capabilities. The cautious view is that we are near data limits (the internet is finite), compute limits (chips are expensive), and the limits of next-token prediction as an objective. Both views have evidence.

The models we have today can already change industries, displace workers, and concentrate power. The time to build governance, safety, and accountability frameworks is before the next capability jump.

Key takeaways

1. Scale produces qualitatively new capabilities. GPT-3 at 175B parameters showed few-shot learning that GPT-2 at 1.5B could not do. PaLM at 540B handled multi-step reasoning. These capabilities emerge at scale rather than improve steadily.

2. Few-shot learning changes the economics of NLP. Instead of collecting labeled data and fine-tuning for every task, a single large model can be steered with prompts. This collapses the cost of building task-specific systems.

3. Infrastructure determines who can participate. Training PaLM required 6,144 TPU chips and the Pathways system. This level of compute is available to a handful of organizations. The scaling trajectory is concentrating frontier research in fewer and fewer hands.

4. Multimodal is the next frontier. GPT-4 accepting image inputs signals that the next generation of foundation models will process text, images, audio, and video together. Applications built on text-only models will need to adapt.

5. Foundation models carry benefit and risk together. Healthcare, education, and finance stand to gain. Bias amplification, power concentration, employment disruption, and accountability gaps are risks that need deliberate intervention: policy, governance, and institutional design as well as technical fixes.

6. The gap between research and deployment is shrinking. GPT-3 was a research demonstration; GPT-4 is a product. The time between "interesting paper" and "deployed system affecting millions" is measured in months, not years.

7. Alignment and safety are engineering problems. GPT-4's post-training alignment improves factuality, reduces harmful outputs, and increases steerability, and it matters as much as the pre-training.

The models in this series, from activation functions through BERT, GPT-2, T5, LLaMA, benchmarks, prompt tuning, and now scaling, trace a clear arc. The work went from building individual components to assembling architectures to scaling those architectures until new capabilities emerged. We can build larger models; the open question is what we do with them.

Based on my CS199 Supervised Independent Study at UC Berkeley and the presentations I created in 2023.