Prompt tuning: from fine-tuning to chain-of-thought
The problem with fine-tuning
Fine-tuning is the most direct way to adapt a pre-trained language model to a new task. You take your base LLM, feed it a smaller labeled dataset specific to your domain, and update all of the model's parameters through backpropagation. The result is a fine-tuned model that performs well on your target task.
The problem is cost and scalability. Every new task requires its own fine-tuning run. If you have ten tasks, you need ten separate fine-tuned models. For a model with 11 billion parameters, that means storing and serving ten copies of 11 billion parameters. The compute cost of the fine-tuning runs themselves adds up fast. And you need a labeled dataset for each task, which in many domains is hard to come by.
One tension drives everything that follows. How do you get task-specific behavior from a general-purpose model without retraining the whole thing?
Hard prompts: prompt engineering
The simplest answer is prompt engineering, sometimes called "hard prompting." Instead of changing the model, you change the input. You write a natural language prompt (a task description, maybe some examples, and an output indicator) and prepend it to your input at inference time. The model stays frozen and needs no training.
The prompt is made of discrete input tokens, the actual words and sentences a human writes. That is what makes them "hard" prompts. You can read them, edit them, and reason about them.
Hard prompts have real limits. First, a good prompt is hard to write. Small wording changes can produce very different outputs, and no systematic method tells you which phrasing works best. Second, the search space is combinatorial. You are looking for the right sequence of discrete tokens from a vocabulary of tens of thousands, and gradient descent cannot help because the search space is not differentiable. Third, you often cannot know the downstream impact of a prompt before you run it. You guess and check.
Natural language tokens set a ceiling. However careful the prompt engineering, you are limited by what those tokens can express.
Soft prompts: learning the prompt
Soft prompts move past that ceiling. Instead of discrete tokens that a human writes, soft prompts are continuous vectors in the model's embedding space. Backpropagation optimizes them directly. They are embeddings, not words, and they encode patterns the model can use for a specific task.
The model itself stays frozen, and only the soft prompt vectors are trainable. You can adapt a massive model to a new task by learning a tiny set of parameters and leaving the billions of model weights untouched.
Soft prompts can be high-level, capturing general task patterns, or task-specific, tuned for a narrow domain. They are consistently more effective than hard prompts because they are not limited to the discrete token vocabulary. They can occupy any point in the continuous embedding space, which gives them far more expressiveness.
The tradeoff is interpretability. You cannot read a soft prompt, because it is a matrix of floating point numbers. You cannot inspect it and see what the model "learned" the way you can read a hard prompt and understand the instruction. That loss of interpretability is the price of moving from discrete to continuous optimization.
Prefix tuning vs. prompt tuning
The terminology here is easy to confuse, because "prefix tuning" and "prompt tuning" are two distinct methods with similar names. Both freeze the model and learn task-specific continuous vectors. They differ in where and how those vectors are inserted.
Prefix tuning
Prefix tuning, introduced by Li and Liang (2021), adds task-specific token vectors to the key and value matrices at every layer of the transformer. These prefix parameters are inserted across all layers and are optimized through a separate feed-forward network during training.
The practical benefits are real. Because tasks differ only in their prefix parameters, you can batch requests from different tasks through the same model and swap the prefix per request. Prefix tuning also tends to improve out-of-domain generalization compared with full fine-tuning, likely because it does not overfit the whole model to one task's distribution.
The performance data is clear. Prefix tuning matches or approaches fine-tuning across standard metrics, and the gap is smallest as training data decreases, which is when parameter-efficient methods matter most.
Prompt tuning
Prompt tuning, introduced by Lester et al. (2021), is a simpler and more parameter-efficient variant. Instead of adding learned vectors at every layer, prompt tuning adds soft prompt embeddings only at the model's input embedding layer. These embeddings are then tuned through standard backpropagation.
The efficiency gains are large. A full 11-billion-parameter model needs 11 billion parameters per task when fine-tuned. Prompt tuning needs roughly 20,000 parameters per task, the soft prompt vectors, and shares the single frozen model across all tasks. That cuts task-specific parameters by more than five orders of magnitude.
A key finding from Lester et al. is that prompt tuning's performance scales with model size. At smaller scales, a noticeable gap separates prompt tuning from full model tuning. As models grow toward 10 billion parameters, prompt tuning closes the gap almost entirely.
For the largest models, the ones where fine-tuning is most expensive, prompt tuning is both the cheapest and the most effective alternative.
Summary of pros and cons for prompt tuning:
- You can tune a massive model with a tiny number of parameters.
- You do not need large labeled datasets.
- It is faster and more efficient than fine-tuning while achieving comparable accuracy.
- It is universally effective across model scales and NLU tasks.
- It supports multi-task and continual learning natively.
- The main downside is that soft prompts remain uninterpretable.
Prompt ensembling: boosting and bagging for prompts
Single prompts, even well-tuned ones, can be unstable. Small changes in the input or the prompt can flip the output. Prompt ensembling uses several prompts together to get more reliable predictions.
The idea is borrowed directly from classical ensemble methods in machine learning. A set of few-shot prompts together form a "boosted prompt ensemble," and their combined predictions outperform any single prompt.
There are two main strategies, mirroring the classical ensemble literature:
- Boosting: Prompts are combined sequentially. Each successive prompt focuses on correcting the errors made by previous prompts, reducing bias over iterations.
- Bagging: Prompts are run in parallel on different subsets of the training data. The diversity across prompts reduces variance, and predictions are aggregated (typically by majority vote).
Prompt ensembling tackles two of the biggest problems with LLMs: hallucination and output instability. When several diverse prompts agree on an answer, you can trust it more. When they disagree, the disagreement is a useful signal that the model is uncertain and the output needs caution.
Use prompt ensembling when you need to guarantee quality of data and output. It costs more inference compute (multiple forward passes), but for high-stakes applications, the reliability gains are worth it.
Chain-of-thought prompting
Chain-of-thought (CoT) prompting is a different kind of intervention. Instead of changing what the model is asked to do, it changes how the model is asked to think. You prompt the model to produce intermediate reasoning steps before it reaches a final answer.
The example above shows this. On an arithmetic problem, a standard prompt leads the model to output 27, which is wrong. The same problem, prompted with a chain-of-thought example that shows intermediate steps, leads the model to output 9 correctly by working through the logic step by step.
This works because LLMs are next-token predictors. When you show them a reasoning chain in the prompt, they learn to produce similar chains, and the act of generating intermediate steps forces the model to "think" sequentially rather than jumping to a potentially wrong conclusion.
The performance data shows two patterns. First, CoT prompting consistently improves reasoning tasks in commonsense, symbolic, and arithmetic settings. Second, the gains depend on scale. CoT helps smaller models little but becomes very effective as model size grows. Smaller models may lack the capacity to follow complex reasoning chains, while larger models can use the intermediate steps.
CoT brings two clear benefits: better reasoning performance and better interpretability, since you can inspect the reasoning chain and see how the model reached its answer. CoT prompting is now a standard technique in the practitioner's toolkit.
Case study: MedPrompt
MedPrompt (Nori et al., 2023) brings these techniques together in a high-stakes application. It is a prompting strategy that raises GPT-4's performance on medical exam questions, specifically the MedQA benchmark.
The question was whether careful prompting can close the gap with fine-tuned specialist models. It can.
MedPrompt combines three techniques.
Dynamic few-shot selection
Instead of using random or hand-picked few-shot examples, MedPrompt uses K-nearest-neighbor clustering in the model's embedding space to find the five most relevant examples for each test question. Every question gets its own tailored set of few-shot demonstrations. It uses training data the way fine-tuning does, but without modifying any model parameters.
Self-generated chain of thought
Rather than manually writing chain-of-thought examples, MedPrompt asks GPT-4 itself to generate the reasoning chains. This automates what would otherwise be the most labor-intensive part of CoT prompting, and the model's own reasoning chains turn out to be highly effective as demonstrations.
Choice shuffling ensemble
LLMs have known position biases and may favor answer choice "A" because it comes first. MedPrompt shuffles the order of answer choices across multiple reasoning paths. For each question, it makes five API calls, each with a different ordering of the answer choices, then selects the final answer by majority vote across the five runs. That removes position bias and adds the reliability benefits of ensembling.
The results
Accuracy rises with each added strategy:
| Strategy | MedQA Accuracy |
|---|---|
| Zero-shot | 81.7% |
| Random few-shot | 83.9% |
| kNN few-shot CoT | 87.3% |
| Random few-shot CoT | 88.4% |
| Ensemble w/ choice shuffle | 90.2% |
From a zero-shot baseline of 81.7%, each technique adds an increment. Dynamic few-shot selection adds 2.2 points. Chain of thought adds another 3.4 to 4.5 points. The choice shuffling ensemble pushes the final score to 90.2%, which beats fine-tuned specialist models like Med-PaLM 2 that needed extensive domain-specific training.
A general-purpose model, GPT-4 with no medical fine-tuning but the right prompting strategies, outperforms models trained specifically on medical data. Few-shot selection, chain of thought, and ensembling work outside papers. Together they beat fine-tuned specialists in a domain where accuracy affects patient outcomes.
The bigger picture
These techniques are a starting point. The field is growing in several directions:
- Zero-shot CoT: Prompting with just "Let's think step by step," with no hand-crafted examples.
- Automatic CoT: Automatically generating diverse reasoning chains as demonstrations.
- Self-consistency with CoT: Sampling multiple reasoning paths and taking the majority vote.
- Visual in-context learning: Applying prompting techniques to vision tasks.
- Multi-modal in-context learning: Extending prompting across text, image, and other modalities.
- Speech in-context learning: Adapting prompting strategies to speech models.
- Graph classification prompt tuning: Applying soft prompt methods to graph neural networks.
The common thread stays the same. Instead of retraining the model, find better ways to communicate with it.
Key takeaways
Fine-tuning works but does not scale. Updating all parameters for each task is expensive, requires labeled data, and means storing a full model copy per task.
Hard prompts are cheap but limited. Prompt engineering requires no training, but you are constrained to discrete tokens and cannot systematically optimize.
Soft prompts balance cost and quality. They are continuous, learnable vectors optimized through backpropagation while the model stays frozen. Prefix tuning adds them at every layer; prompt tuning adds them only at the embedding layer and is simpler and more parameter-efficient.
Prompt tuning scales with model size. At 10 billion+ parameters, prompt tuning matches full fine-tuning performance with roughly 20,000 trainable parameters per task instead of 11 billion.
Ensembling reduces instability. Running diverse prompts and aggregating their predictions tackles hallucination and output variance, which matters for production systems.
Chain of thought improves reasoning. Asking the model to show its work improves performance on arithmetic, symbolic, and commonsense tasks, especially at scale.
Combining techniques gives the largest gains. MedPrompt stacks dynamic few-shot selection, self-generated CoT, and choice shuffling ensembles to push a general-purpose model past fine-tuned specialists.
The direction is steady. We are moving from expensive model modification toward cheap, modular, composable prompting strategies that keep the model fixed.
Based on the presentations I created in 2023.